Data generation system, learning device, data generation device, data generation method, and data generation program

Abstract

The present invention provides a technique for reducing the cost of collecting a plurality of different samples. A data generation system of one aspect of the present invention generates a first pseudo sample including a first feature of a category corresponding to an input value using a first generator that has completed training, generates a second pseudo sample including a second feature of the category corresponding to the input value using a second generator that has completed training, and synthesizes the generated first and second pseudo samples, thereby generating a new sample including the first and second features.

Description

TECHNICAL FIELD

[0001] The present application relates to a data generation system, a learning device, a data generation device, a data generation method, and a data generation program. BACKGROUND

[0002] Conventionally, in a situation where products are manufactured on a production line or the like, a technique has been used in which a product being manufactured is imaged by an imaging device, and the quality of the product is inspected based on the obtained image data. For example, Patent Literature 1 proposes an appearance inspection device that moves a light source by a joint robot, simultaneously images an appearance of an object to be inspected to which inspection light from the light source is irradiated, and analyzes the obtained image, thereby performing appearance inspection of the object to be inspected. Further, for example, Patent Literature 2 proposes an inspection device that determines whether an object to be inspected appearing in an image is normal or abnormal based on a first neural network whose learning is completed, and in a case where the object to be inspected is determined to be abnormal, classifies a kind of the abnormality based on a second neural network whose learning is completed.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2019-045330

[0006] Patent Literature 2: Japanese Patent Application Publication No. 2012-026982

[0007] Patent Literature 3: Japanese Patent Application Publication No. 2018-175343

[0008] Patent Literature 4: Japanese Patent Application Publication No. 2019-071050

[0009] Patent Literature 5: Japanese Patent Application Publication No. 2018-024340

[0010] Patent Literature 6: Japanese Patent Application Publication No. 2018-189522 SUMMARY

[0011] According to the existing method, it is possible to perform appearance inspection of a product based on an image obtained by imaging. Further, according to a method using a machine learning model such as a neural network as in Patent Literature 2, even if the image processing on the obtained image is not specified in detail (for example, even if the values of detection parameters for detecting defects are not set), it is possible to perform appearance inspection of a product based on the output of the machine learning model whose training is completed. Thus, it is possible to simplify the information processing of appearance inspection, and reduce the work of making an inspection program. However, the present inventors and others have found that the following problems exist in the existing inspection method as described above.

[0012] In a case where appearance inspection is performed based on a captured image, an image in which a product in which a defect is present is captured is collected in advance as learning data. In a case where a defect is detected by image processing, the collected learning data is used to set a value of a detection parameter used for detecting a defect. In addition, in a case where a machine learning model is used to detect a defect, the collected learning data is used for machine learning for causing the machine learning model to acquire an ability to detect a defect present on a provided image. The detection of a defect can be identification of a category of a defect. Basically, the more learning data is collected, i.e., the more categories of defects present in the learning data, the more likely it is that the accuracy of appearance inspection is improved.

[0013] However, the more categories of defects and categories of backgrounds including the appearance of a product, the more combinations of categories of defects and categories of backgrounds increase, and the more the number (and kinds) of samples desired to be collected as learning data increase. Therefore, there is a problem that cost is consumed to collect samples of each combination of categories of defects and categories of backgrounds.

[0014] In a case where there is a combination of a category of a defect and a category of a background for which a sample is not obtained, the following problem can occur. That is, in a case where a defect is detected by image processing, it is difficult to set a value of a detection parameter for a combination of a category of a defect and a category of a background for which a sample is not obtained, and this can cause a decrease in the detection accuracy of a defect represented by the combination. For the same reason, even in a case where a machine learning model is used to detect a defect, a decrease in the accuracy of an inference related to detection of a defect performed by a trained machine learning model for the combination can occur due to the inability to reflect the combination of a category of a defect and a category of a background for which a sample is not obtained to machine learning.

[0015] Also, in a case where a machine learning model is used, the following problem can occur. That is, in a case where there is a bias in the combination of a category of a defect and a category of a background present in a sample, particularly in a case where a category of a defect and a category of a background are present in the sample with the same distribution, the bias of the sample can adversely affect the ability acquired by the machine learning model by machine learning. As an example of an adverse effect, since there is a bias in the combination of a category of a defect and a category of a background present in a sample, machine learning using the sample can not acquire an ability to detect a defect according to a category by the machine learning model, but can acquire an ability to identify a category of a background.

[0016] As a specific example, assume that two kinds of products, the first product and the second product, generate two kinds of defects, the first defect and the second defect. Also, in this example, assume that the first product is prone to generate the first defect and the second product is prone to generate the second defect. Also, assume that the obtained samples are biased toward images of the first product including the first defect and images of the second product including the second defect. That is, assume that images of the first product including the second defect and images of the second product including the first defect are not obtained as samples for machine learning. Assume that machine learning of the machine learning model is performed in order to acquire the ability to detect defects according to categories using the samples obtained under this assumption. In this case, by the bias in the obtained samples, the machine learning model can acquire, through the machine learning, not the ability to detect the first defect according to the category of the first product, but the ability to recognize the category of the background. That is, by biasing the samples, the machine learning model can acquire an ability different from the desired ability. In the case where the ability to recognize the category of the background is acquired, when an image of the first product including the second defect is presented, the machine learning model trained can erroneously detect that the first product generates the first defect.

[0017] Therefore, in any case, in the presence of a combination of samples not obtained, there is a problem that the accuracy of appearance inspection for the combination decreases (in the worst case, even the inspection cannot be performed). Thus, in order to improve the accuracy of appearance inspection, it is preferable to collect samples comprehensively in the combination of the category of defects and the category of the background. However, as described above, there is a problem that collecting samples as above comprehensively incurs a cost. In particular, the more the category of defects and the category of the background, the more there can be a specific combination of the category of defects and the category of the background that generates a defect with a very low probability, and thus collecting samples comprehensively incurs a cost.

[0018] Note that this problem is not unique to the situation of performing appearance inspection of products based on captured images. It can occur in all situations in which a plurality of different samples of a prescribed kind of data that can exhibit at least two or more features are collected. The prescribed kind of data refers to image data as described above, and also to measured data obtained by sensors, such as sound data, numerical data, text data, and the like. The situation in which samples of a prescribed kind of data that can exhibit a plurality of features are collected refers to the situation of performing appearance inspection of products based on captured images as described above, and also to situations such as inferring the state of a patient based on medical images of an organ, inferring attributes related to a moving body appearing in a captured image, inferring the driving state of a vehicle based on measured data obtained by sensors provided in the vehicle, and inferring the state of a machine based on measured data obtained by sensors that observe the state of the machine. The moving body is, for example, a pedestrian, a vehicle, or the like. The attributes related to the moving body are, for example, the movement state (e.g., speed, path, and the like) of the moving body, and the degree of concentration (e.g., the degree of congestion, and the like) of the moving body.

[0019] As a specific example, Patent Literature 3 proposes detecting a lesion region from a medical image by two inferrers. Patent Literature 4 proposes inferring the distribution of the density and the distribution of the movement vector of a person appearing in a captured image from the captured image using a neural network that has been trained. Patent Literature 5 proposes determining the driving tendency of a driver based on measured data of acceleration measured during the period in which the driver drives a vehicle. Patent Literature 6 proposes a method of judging a precursor to a failure of a device using measured data of at least one of sound and vibration.

[0020] Even in these situations, in the case where there is a combination of classes of features for which samples have not been obtained, there is a problem in that the accuracy of the respective inferences decreases. Thus, in order to improve the accuracy of the inferences, it is preferable to collect samples comprehensively for each combination of classes of features. However, this gives rise to the problem of incurring costs for collecting samples comprehensively.

[0021] One aspect of the present application has been made in view of such circumstances, and aims to provide a technology for reducing the costs incurred in collecting a plurality of different samples of a prescribed kind of data that can exhibit at least two or more features.

[0022] The present application has been made in order to solve the above-described problems, and adopts the following configuration.

[0023] That is, a data generation system of one aspect of the present application includes: a first acquisition unit that acquires a plurality of first learning data sets each composed of a combination of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data; a second acquisition unit that acquires a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data; a first training unit that performs machine learning of a first learning model including a first generator using the acquired plurality of first learning data sets, the machine learning of the first learning model including training the first generator for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample; a second training unit that performs machine learning of a second learning model including a second generator using the acquired plurality of second learning data sets, the machine learning of the second learning model including training the second generator for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample; a first generation unit that generates a first pseudo sample including the first feature by providing a first input value corresponding to the first label to the trained first generator; a second generation unit that generates a second pseudo sample including the second feature by providing a second input value corresponding to the second label to the trained second generator; and a data synthesis unit that generates a new sample of the prescribed kind of data by synthesizing the generated first pseudo sample and the generated second pseudo sample.

[0024] In the data generation system of this structure, the first generator is trained to generate a pseudo sample matching the first sample including the first feature from the first label by machine learning using the plurality of first learning data sets. The second generator is trained to generate a pseudo sample matching the second sample including the second feature from the second label by machine learning using the plurality of second learning data sets. After each of the machine learning, the first generator acquires an ability to generate a pseudo sample including the first feature of a category corresponding to a provided input value, and the second generator acquires an ability to generate a pseudo sample including the second feature of a category corresponding to a provided input value. The data generation system of this structure generates a first pseudo sample and a second pseudo sample using the first generator and the second generator, and synthesizes the generated first pseudo sample and the generated second pseudo sample, thereby generating a new sample of the prescribed kind of data.

[0025] Thus, in the data generation system of this configuration, the respective classes of the first feature and the second feature contained in the generated new sample can be controlled based on the input provided to the first generator and the second generator that have completed learning (hereinafter also referred to as "completed training"), respectively. That is, by using the first generator and the second generator that have completed learning, respectively, a new sample including a combination of arbitrary classes of the first feature and the second feature can be generated. In the case where there is a combination of classes of the first feature and the second feature for which no sample or a small number of samples are available, a new sample including the combination of classes of the first feature and the second feature can be freely generated. Furthermore, by using the first generator and the second generator that have completed learning, respectively, the process of generating a new sample including the first feature and the second feature can be automated to a large extent. Thus, according to this configuration, a plurality of different combinations of samples can be freely and automatically generated, and thus, the cost of collecting a plurality of different samples for a prescribed category that can exhibit at least two or more features can be reduced.

[0026] Note that the first feature and the second feature can be appropriately selected according to the embodiment, respectively. For example, the first feature and the second feature can be related to components (elements) that can appear directly or indirectly in data. Direct appearance refers to appearance in data itself, such as image data. Indirect appearance refers to derivation from data inferred from image data, etc. At least one of the first feature and the second feature can be related to a component that can be a target of some kind of inference processing. Inference can also be referred to as "inference". Inference can include prediction. Inference can be, for example, any one of derivation of a discrete value (e.g., a classification corresponding to a specific feature) by grouping (classification, recognition), and derivation of a continuous value (e.g., a probability of appearance of a specific feature) by regression. Inference can include some kind of authentication such as detection, determination, etc. based on the result of the grouping or regression. The prescribed category of data can be constituted by a plurality of kinds of data, as in dynamic image data including sound data. In this case, the first feature and the second feature can be related to at least any one of the plurality of kinds of data, respectively. For example, the first feature can be related to image data, and the second feature can be related to sound data, etc., as long as they can be synthesized, and the kinds of data related to the first feature and the second feature can also be different, respectively.

[0027] The first generator and the second generator are each constituted by a machine learning model. The machine learning model of each generator is configured to generate a fake sample (pseudo sample) of each sample of the simulation data, according to input of an input value corresponding to each label. The "pseudo sample" refers to a sample generated by each generator. The kind of the machine learning model of each generator is not particularly limited, and can be appropriately selected according to the embodiment. The machine learning model of each generator can use, for example, a neural network. The kind of the neural network is not particularly limited, and can be appropriately selected according to the embodiment. The input value of each generator and the parameter of the synthesis can each be determined manually, or can be determined automatically by a method such as randomization. The first sample and the second sample used for training the first generator and the second generator can be obtained from the real world, or can be artificially generated by simulation, manual operation, or the like.

[0028] In the data generation system according to the above aspect, the data of the prescribed kind can include a first component and a second component different from the first component, and the second component can be a prescribed object of estimation. The first feature can be related to the first component, and the second feature can be related to the second component. According to this configuration, by reflecting a plurality of different samples collected at low cost in the implementation of the prescribed estimation, the accuracy of the estimation can be improved.

[0029] In the data generation system according to the above aspect, the first learning model can further include a first discriminator. The training of the first generator can be constituted by alternately repeating the following steps: training the first discriminator to identify whether an input sample input to the first discriminator is the first sample obtained from any one of the plurality of first learning data sets or a pseudo sample generated by the first generator; and training the first generator to generate a pseudo sample that reduces the performance of the identification of the first discriminator.

[0030] In this configuration, by alternately repeating the training of the first discriminator and the training of the first generator, it is expected that the performance of the first generator that generates a pseudo sample matching the first sample will be improved according to the improvement in the identification performance of the first discriminator. Therefore, according to this configuration, it is possible to construct a learning-completed first generator that can generate a suitable pseudo sample including the first feature, and thus it is possible to appropriately reduce the cost of collecting a plurality of different samples.

[0031] In the data generation system of the above aspect, the second learning model can further include a second discriminator. Training the second generator can be constituted by alternately repeating the following steps: training the second discriminator to identify whether an input sample input to the second discriminator is the second sample obtained from any one of the plurality of second learning data sets or a pseudo sample generated by the second generator; and training the second generator to generate a pseudo sample that reduces the performance of the identification by the second discriminator.

[0032] In this configuration, by alternately repeating the training of the second discriminator and the training of the second generator, it is possible to improve the performance of the second generator that generates a pseudo sample matching the second sample in accordance with the improvement in the identification performance of the second discriminator. Therefore, according to this configuration, it is possible to construct a learning-completed second generator that can generate a suitable pseudo sample including the second feature, and thus it is possible to appropriately reduce the cost of collecting a plurality of different samples.

[0033] The data generation system of the above aspect can further include a third acquisition unit that acquires a plurality of third learning data sets each composed of a sample value of an attribute of the second feature, a reflection level indicating a degree of reflection of the second feature, and the second label; and a third training unit that performs machine learning of a third learning model including a third generator using the acquired plurality of third learning data sets, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value. The synthesizing can include generating a pseudo value of the attribute of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level to the third generator completed with the training, and converting the second pseudo sample so that a value of the attribute of the second feature included in the second pseudo sample matches the generated pseudo value, and synthesizing the second pseudo sample to the first pseudo sample.

[0034] In this configuration, the third generator is trained by machine learning using a plurality of third learning sets to generate pseudo values of the attributes of the second features that match the sample values from the reflection level and the second label. After this machine learning, the third generator acquires the ability to generate pseudo values of the attributes of the second features that correspond to the input provided. Therefore, it is possible to control the values of the attributes of the second features contained in the new samples generated based on the input provided to the third generator whose learning is completed. That is, by further utilizing the third generator whose learning is completed, it is possible to freely generate new samples including the second features having arbitrary attribute values. Therefore, according to this configuration, it is possible to generate more different samples with respect to the prescribed kind of data that can exhibit at least two or more features. Note that the "attribute" relates to the degree or classification of the properties that the features such as size, position (temporal or regional), brightness, amplitude, frequency (temporal or regional), and the like have.

[0035] The data generation system of the above-described one solution can further include a third acquisition unit that acquires a plurality of third learning data sets each composed of a sample value of an attribute of a second feature, a reflection level indicating a degree of reflection of the second feature, and a second label; and a third training unit that performs machine learning of a third learning model including a third generator using the plurality of third learning data sets acquired, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value. The prescribed kind of data can include a first component and a second component different from the first component, the second component being an object of a prescribed speculation. The first feature can be related to the first component, and the second feature can be related to the second component. The prescribed speculation can be detection of the second component. The synthesizing can include generating a pseudo value of the attribute of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level, i.e., a third input value provided according to the limit of the detection, to the third generator whose training is completed, and converting the second pseudo sample so that a value of the attribute of the second feature contained in the second pseudo sample matches the pseudo value generated, and synthesizing the second pseudo sample to the first pseudo sample. According to this configuration, with respect to the prescribed kind of data that can exhibit at least two or more features, it is possible to generate a plurality of different samples corresponding to the limit case of the detection. In addition, by reflecting the plurality of different samples generated in the implementation of the prescribed speculation, it is possible to improve the accuracy of the speculation.

[0036] In the data generation system of the above aspect, the third learning model can further include a third discriminator. The third generator is trained by alternately repeating the following steps: training the third discriminator to identify whether an input value input to the third discriminator is the sample value obtained from any one of the plurality of third learning data sets or the pseudo value generated by the third generator; and training the third generator to generate the pseudo value of the second feature attribute such that the performance of the identification by the third discriminator is reduced.

[0037] In this configuration, by alternately repeating the training of the third discriminator and the training of the third generator, it is possible to improve the performance of the third generator that generates the pseudo value matching the sample value of the second feature attribute in accordance with the improvement of the identification performance of the third discriminator. Therefore, according to this configuration, it is possible to construct the third generator of which the learning is completed, which can generate a suitable pseudo value of the second feature attribute, and thus, it is possible to appropriately reduce the cost required for collecting a plurality of different samples.

[0038] In the data generation system of the above aspect, each of the third learning data sets can further have the first label. The machine learning that implements the third learning model can include training the third generator such that the pseudo value of the second feature attribute generated by the third generator from the reflection level, the second label, and the first label matches the sample value.

[0039] In this configuration, the third generator is trained to further generate the pseudo value of the second feature attribute in correspondence with the class of the first feature indicated by the first label. Therefore, according to the third generator of which the learning is completed, it is possible to control the value of the second feature attribute in accordance with the class of the first feature. That is, by using the third generator of which the learning is completed, it is possible to freely generate a new sample including the second feature having an attribute value suitable for the class of the first feature. Therefore, according to this configuration, it is possible to generate a greater variety of suitable samples with respect to data of which at least two or more kinds of features can be expressed.

[0040] In the data generation system of the above aspect, the reflection level can be constituted by a continuous value. According to this configuration, it is possible to control the attribute value of the second feature by a continuous value, and thus, it is possible to generate a greater variety of different samples with respect to data of which at least two or more kinds of features can be expressed. Note that the constitution of the reflection level can not be limited to this example. The reflection level can also be constituted by a discrete value.

[0041] In the data generation system according to the above aspect, the acquiring the plurality of second learning data sets includes: acquiring a plurality of third samples each including the first feature and the second feature, the third samples being third samples of the prescribed kind of data; inferring, by providing each of the third samples to an inferrer trained to infer an input provided to the first generator when a pseudo sample generated by the first generator is generated, an inferred value of the input provided to the first generator for generating a pseudo sample corresponding to each of the third samples; generating, for each of the third samples, a third pseudo sample by providing the inferred value to the first generator; and generating the second sample of each of the second learning data sets by subtracting the third pseudo sample from each of the third samples. Basically, the first generator of which the learning is completed is configured to generate a pseudo sample including the first feature and not including the second feature. Therefore, according to this configuration, the second sample not including the first feature and including the second feature can be appropriately generated. In addition, according to this configuration, the second sample can be automatically generated from the third sample, and thus, the cost of acquiring the second sample can be reduced.

[0042] The data generation system according to the above aspect can further include: a fourth acquiring section that acquires a plurality of fourth learning data sets each composed of a combination of a sample value of an input provided to the first generator of which the learning is completed and a fourth pseudo sample generated by providing the sample value to the first generator of which the learning is completed; and a fourth training section that constructs the inferrer of which the learning is completed by performing machine learning of the inferrer using the acquired plurality of fourth learning data sets, the machine learning of the inferrer including training the inferrer for each of the fourth learning data sets so that an inferred value obtained by inferring, by the inferrer, the input provided to the first generator of which the learning is completed from the fourth pseudo sample matches the sample value. According to this configuration, by performing machine learning using the plurality of fourth learning data sets, the inferrer of which the learning is completed can be generated, which can appropriately infer an input value provided to the first generator of which the learning is completed for generating a pseudo sample corresponding to the third sample, including the first feature and not including the second feature. Thus, by using the generated inferrer of which the learning is completed, the second sample not including the first feature and including the second feature can be appropriately generated from the third sample.

[0043] In the data generation system according to the above aspect, the prescribed kind of data can be image data including a background and a foreground. The first feature can be related to the background, and the second feature can be related to the foreground. According to this configuration, the cost of collecting samples of various combinations of the background and the foreground of different categories for the image data can be reduced.

[0044] In the data generation system of the above-described aspect, the data of the prescribed category can be image data of a product to be inspected. The first feature can be related to a background including the product, and the second feature can be related to a defect of the product. According to this configuration, it is possible to reduce the cost of collecting samples of various combinations of products and defects. In addition, by reflecting the generated various samples in the implementation of the appearance inspection, it is possible to improve the accuracy of the appearance inspection.

[0045] The forms of the present application are not limited to the above-described data generation system. By extracting at least a part of the configuration of the above-described data generation system, a new form of device (or system) can be configured. For example, the learning device of an aspect of the present application can be configured by extracting a part for generating at least any one of the generators by machine learning from each of the above-described forms of the data generation system. In addition, for example, the data generation device of an aspect of the present application can also be configured by extracting a part for generating new samples using the generators completed by learning from each of the above-described forms of the data generation system.

[0046] As an example, the learning device of an aspect of the present application includes: a first acquisition unit that acquires a plurality of first learning data sets each composed of a combination of a first sample of a prescribed category of data including a first feature and a first label indicating a category of the first feature; a second acquisition unit that acquires a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature; a first training unit that performs machine learning of a first learning model including a first generator using the acquired plurality of first learning data sets, the machine learning of the first learning model including training the first generator for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample; and a second training unit that performs machine learning of a second learning model including a second generator using the acquired plurality of second learning data sets, the machine learning of the second learning model including training the second generator for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample.

[0047] Note that the "learning device" can also be referred to as a "model generation device" or simply a "generation device" for generating a machine learning model (generator) completed with training. The learning method can also be referred to as a model generation method or simply a generation method for generating a machine learning model completed with training. That is, the learning method corresponds to a method of generating (producing) a machine learning model completed with training.

[0048] As other examples, a data generation device of an aspect of the present technology includes: a first generation unit that has a first generator trained by machine learning using a plurality of first learning data sets each composed of a combination of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data, the first generator being trained by the machine learning for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, the first generation unit generating a first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first generator trained; a second generation unit that has a second generator trained by machine learning using a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, the second generator being trained by the machine learning for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample, the second generation unit generating a second pseudo sample including the second feature by providing a second input value corresponding to the second label to the second generator trained; and a data synthesis unit that generates a new sample of the prescribed kind of data by synthesizing the generated first pseudo sample with the generated second pseudo sample.

[0049] In addition, as other aspects of each of the data generation system, the learning device, and the data generation device of each of the above-described aspects, an aspect of the present technology can be an information processing method that implements all or a part of each of the above-described aspects, a program, or a storage medium readable by a computer or the like, a machine or the like, in which the program is stored. Here, the storage medium readable by a computer or the like refers to a medium that stores a program or the like by electricity, magnetism, optics, machine, or chemistry.

[0050] For example, a data generation method of one aspect of the present application is an information processing method in which a computer executes the steps of: acquiring a plurality of first learning data sets each composed of a combination of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data; acquiring a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data; implementing machine learning of a first learning model including a first generator using the acquired plurality of first learning data sets, the machine learning of the first learning model including training the first generator for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample; implementing machine learning of a second learning model including a second generator using the acquired plurality of second learning data sets, the machine learning of the second learning model including training the second generator for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample; generating a first pseudo sample including the first feature by providing a first input value corresponding to the first label to the trained first generator; generating a second pseudo sample including the second feature by providing a second input value corresponding to the second label to the trained second generator; and generating a new sample of the prescribed kind of data by synthesizing the generated first pseudo sample and the generated second pseudo sample.

[0051] Further, for example, a data generation method according to an aspect of the present disclosure is an information processing method in which a computer executes the steps of: generating, by a first trained generator, a first pseudo sample including a first feature, the first trained generator being constructed by machine learning using a plurality of first learning data sets each composed of a combination of a first sample including the first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data, in the machine learning, the first generator being trained for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, the computer generating the first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first trained generator; generating, by a second trained generator, a second pseudo sample including a second feature, the second trained generator being constructed by machine learning using a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, in the machine learning, the second generator being trained for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample, the computer generating the second pseudo sample including the second feature by providing a second input value corresponding to the second label to the second trained generator; and generating a new sample of the prescribed kind of data by synthesizing the generated first pseudo sample and the generated second pseudo sample.

[0052] Further, for example, a data generation program of one aspect of the present application is a data generation program for causing a computer to execute the steps of: generating a first pseudo sample including a first feature, using a first generator that is completed training, the first generator being constructed by machine learning using a plurality of first learning data sets each composed of a combination of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data, in the machine learning, the first generator being trained for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, the computer generating the first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first generator that is completed training; generating a second pseudo sample including a second feature, using a second generator that is completed training, the second generator being constructed by machine learning using a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, in the machine learning, the second generator being trained for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample, the computer generating the second pseudo sample including the second feature by providing a second input value corresponding to the second label to the second generator that is completed training; and generating a new sample of the prescribed kind of data by synthesizing the generated first pseudo sample and the generated second pseudo sample.

[0053] According to the present application, it is possible to reduce the cost of collecting a plurality of different samples for a prescribed kind of data that can exhibit at least two or more features. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 An example of a situation in which the present application is applied schematically.

[0055] Figure 2A An example of a sample including a first feature and a second feature of an illustrative example embodiment.

[0056] Figure 2B An example of a sample including a first feature and a second feature of an illustrative example embodiment.

[0057] Figure 3 An example of a hardware configuration of a learning device of an illustrative example embodiment.

[0058] Figure 4An example of a hardware configuration of the data generation apparatus of the illustrative example embodiment.

[0059] Figure 5 An example of a hardware configuration of the inference apparatus of the illustrative example embodiment.

[0060] Figure 6 An example of a software configuration of the learning apparatus of the illustrative example embodiment.

[0061] Figure 7A A process of machine learning of the first learning model by the learning apparatus of the illustrative example embodiment.

[0062] Figure 7B A process of machine learning of the second learning model by the learning apparatus of the illustrative example embodiment.

[0063] Figure 7C A process of machine learning of the third learning model by the learning apparatus of the illustrative example embodiment.

[0064] Figure 7D A process of machine learning of the inferrer by the learning apparatus of the illustrative example embodiment.

[0065] Figure 7E A process of generation of the second sample by the learning apparatus of the illustrative example embodiment.

[0066] Figure 8 An example of a software configuration of the data generation apparatus of the illustrative example embodiment.

[0067] Figure 9 An example of a software configuration of the inference apparatus of the illustrative example embodiment.

[0068] Figure 10A An example of a processing flow of machine learning of the first learning model by the learning apparatus of the example embodiment.

[0069] Figure 10B An example of a processing flow of a subroutine related to machine learning of the first learning model by the learning apparatus of the example embodiment.

[0070] Figure 11A An example of a processing flow of machine learning of the second learning model by the learning apparatus of the example embodiment.

[0071] Figure 11B An example of a processing flow of a subroutine related to machine learning of the second learning model by the learning apparatus of the example embodiment.

[0072] Figure 12AAn example of a processing flow of machine learning of a third learning model by the learning device of the embodiment.

[0073] Figure 12B An example of a processing flow of a subprogram related to machine learning of a third learning model by the learning device of the embodiment.

[0074] Figure 13 An example of a processing flow of machine learning of a predictor by the learning device of the embodiment.

[0075] Figure 14 An example of a processing flow related to generation of a second sample by the learning device of the embodiment.

[0076] Figure 15 An example of a processing flow of a data generation device of the embodiment.

[0077] Figure 16 An example of an input screen of a data generation device of the illustrative embodiment, the input screen being for accepting input of each input value.

[0078] Figure 17A An example of a processing flow of a prediction device of the embodiment.

[0079] Figure 17B An example of a process of a prediction process related to the second feature of the embodiment.

[0080] Figure 18A An example of an output screen of a data group of the embodiment.

[0081] Figure 18B An example of an output screen of a data group of the embodiment.

[0082] Figure 19 An example of other contexts in which the present invention is applied.

[0083] Figure 20A An example of a hardware configuration of an inspection device in other ways.

[0084] Figure 20B An example of a software configuration of an inspection device in other ways.

[0085] Figure 21 An example of other contexts in which the present invention is applied.

[0086] Figure 22 An example of other contexts in which the present invention is applied.

[0087] Figure 23 An example of other contexts in which the present invention is applied.

[0088] Figure 24A An example of a hardware configuration of the monitoring device of another aspect.

[0089] Figure 24B An example of a software configuration of the monitoring device of another aspect.

[0090] Figure 25 An example of another aspect in which the present application is applied.

[0091] Figure 26A An example of a hardware configuration of the abnormality detection device of another aspect.

[0092] Figure 26B An example of a software configuration of the abnormality detection device of another aspect.

[0093] Figure 27 An example of a configuration of the generator of another aspect.

[0094] Figure 28 An example of another aspect in which the present application is applied.

[0095] Figure 29 An example of a hardware configuration of the other learning device of another aspect.

[0096] Figure 30 An example of a software configuration of the other learning device of another aspect.

[0097] Figure 31 An example of a processing flow of the other learning device of another aspect.

[0098] Figure 32A An example of a hardware configuration of the inference device of another aspect.

[0099] Figure 32B An example of a software configuration of the inference device of another aspect. DETAILED DESCRIPTION

[0100] Hereinafter, an embodiment of a scheme of the present application (hereinafter, also referred to as "the present embodiment") will be described based on the drawings. However, the present embodiment described below is merely an example of the present application in any respect. Of course, various modifications and alterations can be made without departing from the scope of the present application. That is, when the present application is implemented, a specific configuration corresponding to the embodiment can be appropriately adopted. Note that, although data appearing in the present embodiment is described by natural language, more specifically, it is specified by a virtual language, a command, a parameter, a machine language, and the like which are recognized by a computer.

[0101] §1 Application Examples

[0102] Figure 1 An illustrative example of a scenario in which the invention is applied. For example... Figure 1 As shown, the inference system 100 of this embodiment includes a learning device 1, a data generation device 2, and an inference device 3. The learning device 1 and the data generation device 2 generate a first generator 411 and a second generator 421 that have completed learning, and use the generated first generator 411 and second generator 421 to construct a data generation system for generating new samples 65 of a specified type of data.

[0103] The learning device 1 in this embodiment is a computer configured to generate a first generator 411 and a second generator 421 that have completed learning by respectively implementing machine learning. Specifically, the learning device 1 in this embodiment acquires multiple first learning datasets 51, each consisting of a combination of a first sample 511 and a first label 512 of data of a specified type. Additionally, the learning device 1 in this embodiment acquires multiple second learning datasets 52, each consisting of a combination of a second sample 521 and a second label 522 of data of a specified type.

[0104] For each sample (511, 521), if the data of a specified type is data that can represent a certain characteristic, then the type of data is not particularly limited and can be appropriately selected according to the implementation method. The specified type of data can be, for example, image data, sound data, numerical data, text data, and sensor data obtained through various sensors. The specified type of data can be, for example, sensor data obtained by observing an object through a sensor. Sensors can be, for example, cameras, microphones, encoders, LiDAR sensors, medical examination devices, life sensors, human sensory sensors, environmental sensors, etc. Cameras can be, for example, ordinary digital cameras configured to acquire RGB images, depth cameras configured to acquire depth images, infrared cameras configured to image infrared quantities, etc. Medical examination devices can be, for example, X-ray devices, CT devices, MRI devices, etc. Life sensors can be, for example, thermometers, blood pressure monitors, heart rate monitors, etc. Environmental sensors can be, for example, light intensity meters, thermometers, hygrometers, etc.

[0105] The first sample 511 includes a first feature that appears in the data of the prescribed kind, and the first label 512 indicates the category of the first feature. On the other hand, the second sample 521 includes a second feature that is different from the first feature, and the second label 522 indicates the category of the second feature. The first feature and the second feature can each be appropriately selected according to the embodiment. For example, the first feature and the second feature can each be related to a component (element) that can directly or indirectly appear in the data of the prescribed kind. Direct appearance refers to appearance in the data itself, such as appearance in image data. Indirect appearance refers to derivation from data that is inferred from image data, and the like. At least one of the first feature and the second feature can be an object of a certain inference process. Note that the object sample includes an object feature that corresponds to the object feature appearing in the object sample.

[0106] In the present embodiment, the data of the prescribed kind can include a first component and a second component that is different from the first component. The second component becomes an object of a prescribed inference. The first feature included in each of the first samples 511 is related to the first component, and the second feature included in each of the second samples 521 is related to the second component. The inference can include prediction. The inference can be, for example, any one of derivation of a discrete value (for example, a classification corresponding to a certain feature) by grouping (classification, identification), and derivation of a continuous value (for example, a probability of appearance of a certain feature) by regression. The inference can include detection, determination, or the like, of a certain authentication based on the result of the grouping or the regression. The data of the prescribed kind can be constituted by a plurality of kinds of data, such as dynamic image data including sound data. In this case, the first feature and the second feature can each be related to at least any one of the plurality of kinds of data. For example, the first feature can be related to image data, and the second feature can be related to sound data, and the like, and the kinds of data to which the first feature and the second feature are related can also be different as long as they can be synthesized.

[0107] Next, the learning device 1 of the present embodiment implements machine learning of the first learning model 41 including the first generator 411 using the plurality of acquired first learning data sets 51. Implementing machine learning of the first learning model 41 includes training the first generator 411 with respect to each of the first learning data sets 51 so that the pseudo sample generated by the first generator 411 from the first label 512 matches the first sample 511. In addition, the learning device 1 of the present embodiment implements machine learning of the second learning model 42 including the second generator 421 using the plurality of acquired second learning data sets 52. Implementing machine learning of the second learning model 42 includes training the second generator 421 with respect to each of the second learning data sets 52 so that the pseudo sample generated by the second generator 421 from the second label 522 matches the second sample 521. After each of the machine learning, the first generator 411 and the second generator 421 that have completed learning are generated.

[0108] Each of the generators (411, 421) is constituted by a machine learning model. The machine learning model of each of the generators (411, 421) is configured to generate a fake sample (pseudo sample) of each of the samples (511, 521) of the simulation data, according to an input of an input value corresponding to each of the labels (512, 522). The pseudo sample refers to a sample generated by each of the generators (411, 421). The kind of the machine learning model of each of the generators (411, 421) can not be particularly limited, and can be appropriately selected according to the embodiment. In the embodiment, a neural network is used in the machine learning model of each of the generators (411, 421). Details will be described later. Note that the learning device 1 of the embodiment is also referred to as a "model generation device" or simply a "generation device" for generating a machine learning model that has completed training.

[0109] The data generation device 2 of the embodiment is a computer configured to generate a new sample 65 of data of a prescribed kind, using the first generator 411 and the second generator 421 each of which has completed learning. Specifically, the data generation device 2 of the embodiment generates a first pseudo sample 61 including a first feature, by providing a first input value 221 corresponding to the first label 512 to the first generator 411 that has completed training. In addition, the data generation device 2 of the embodiment generates a second pseudo sample 62 including a second feature, by providing a second input value 223 corresponding to the second label 522 to the second generator 421 that has completed training. Furthermore, the data generation device 2 of the embodiment generates a new sample 65 of data of a prescribed kind, by synthesizing the generated first pseudo sample 61 and the generated second pseudo sample 62.

[0110] The inference device 3 of the embodiment is a computer configured to perform a prescribed inference process with respect to the second feature. In the embodiment, the prescribed inference refers to detection of the second component (in other words, detection of the presence of the second feature). The inference device 3 of the embodiment acquires an object sample 325, and performs a process for detecting the second component on the acquired object sample 325. The parameters in the detection process can be determined based on a data group 320. The data group 320 is constituted by a plurality of samples 321 of data of a prescribed kind. The plurality of samples 321 can include the sample 65 generated by the data generation device 2.

[0111] In summary, in this embodiment, the categories of the first and second features contained in the newly generated sample 65 can be controlled based on the inputs provided to the first generator 411 and the second generator 421 after learning, respectively. That is, the data generation apparatus 2 of this embodiment can generate new samples 65 including combinations of first and second features of arbitrary categories using the first generator 411 and the second generator 421 after learning, respectively. In cases where there are no samples or a small number of samples, new samples 65 including combinations of first and second features of such categories can be freely generated. Furthermore, by utilizing the first generator 411 and the second generator 421 after learning, most of the processing for generating new samples 65 can be automated. Therefore, according to this embodiment, samples with various combinations can be generated freely and automatically, thus reducing the cost of collecting various samples for a specified type of data exhibiting at least two or more features.

[0112] Furthermore, in this embodiment, by adding new samples 65 to the data set 320, it is possible to diversify the samples 321 contained in the data set 320. That is, the data set 320 can be enriched with samples for combinations that are not obtained or are few in number. As a result, the accuracy of the inference for such combinations can be improved. Therefore, according to this embodiment, by reflecting the new samples 65 generated as described above in the implementation of the inference processing in the inference device 3, it is possible to improve the accuracy of the inference processing.

[0113] It should be noted that, as Figure 1 As shown, the learning device 1, the data generation device 2, and the estimation device 3 can be interconnected via a network. The type of network can be appropriately selected from, for example, the Internet, wireless communication networks, mobile communication networks, telephone networks, dedicated networks, etc. However, the method for exchanging data between the learning device 1, the data generation device 2, and the estimation device 3 is not limited to this example and can be appropriately selected according to the implementation method. For example, data can be exchanged between the learning device 1, the data generation device 2, and the estimation device 3 using a storage medium.

[0114] In addition, Figure 1 In the example, the learning device 1, the data generation device 2, and the inference device 3 are each a separate computer. However, the configuration of the system in this embodiment is not limited to such an example and can be appropriately determined according to the embodiment. For example, at least any pair of the learning device 1, the data generation device 2, and the inference device 3 can be an integrated computer. In addition, for example, at least one of the learning device 1, the data generation device 2, and the inference device 3 can be composed of multiple computers.

[0115] Here, further use Figure 2Aand Figure 2B A specific example of the prescribed kind of data that can be processed in the above-described inference system 100 will be described. Figure 2A An example of a sample P10 including the first feature and the second feature obtained in the context of appearance inspection of a schematic example product P1. Figure 2B An example of a sample P20 including the first feature and the second feature obtained in the context of machine state inspection based on machine operation sound.

[0116] Figure 2A In the example, the image data in which the product P1 is presented is an example of the prescribed kind of data. In this example, the first feature is related to a background P11 including the product P1, and the second feature is related to a defect P12 that can occur on the product P1. That is, the image of the background P11 including the product P1 in the sample P10 is an example of the first component, and the image of the defect P12 is an example of the second component. In this case, in each of the first learning data sets 51, the first sample 511 can be a sample of the image data in which the product P1 is presented, and the first label 512 can indicate the category of the product P1. The second sample 521 can be a sample of the image data in which the defect P12 is presented, and the second label 522 can indicate the category of the defect P12.

[0117] The kind of the product P1 and the defect P12 can be arbitrarily selected. The product P1 can be, for example, a product conveyed on a production line, such as an electronic device, an electronic component, an automobile component, a medicine, a food, or the like. The electronic component can be, for example, a substrate, a chip capacitor, a liquid crystal, a winding of a relay, or the like. The automobile component can be, for example, a connecting rod, a shaft, an engine block, an electric window switch, a panel, or the like. The medicine can be, for example, a packaged tablet, an unpackaged tablet, or the like. The product can be any one of a final product generated after a manufacturing process, an intermediate product generated during the manufacturing process, and an initial product prepared before the manufacturing process. The defect P12 can be, for example, a scratch, a stain, a crack, a dent, a burr, uneven color, foreign matter contamination, or the like.

[0118] In this example, the learning device 1 of the present embodiment can construct the first generator 411 by performing machine learning using the plurality of first learning data sets 51 described above, the first generator 411 being trained to generate a pseudo sample in which the background including the product P1 of the class corresponding to the input is presented. In addition, the learning device 1 of the present embodiment can also construct the second generator 421 by performing machine learning using the plurality of second learning data sets 52 described above, the second generator 421 being trained to generate a pseudo sample in which the foreground including the defect P12 of the class corresponding to the input is presented. In the data generation device 2 of the present embodiment, by respectively utilizing the first generator 411 and the second generator 421 for which the learning is completed, it is possible to automate most of the process of causing a new sample 65 in which the product P1 and the defect P12 of an arbitrary combination of classes are presented to be generated. Therefore, it is possible to reduce the cost of collecting a plurality of different samples in which the product P1 and the defect P12 are presented. Also, in the estimation device 3 of the present embodiment, by adding the generated new sample 65 to diversify the samples 321 contained in the data group 320, it is possible to improve the accuracy of the appearance inspection of the product P1.

[0119] On the other hand, Figure 2B In this example, the sound data including the operating sound of the machine is an example of the prescribed kind of data. In this example, the first feature is related to an operating sound P21 generated when the machine is normally operating, an environmental sound such as noise around the machine, and the like, and the second feature is related to an abnormal sound P22 generated due to an abnormality such as aging, malfunction, or the like of the machine. That is, the component of the environmental sound P21 in the sample P20 is an example of the first component, and the component of the abnormal sound P22 is an example of the second component. In this case, in each of the first learning data sets 51, the first sample 511 can be a sample of sound data including the environmental sound P21, and the first label 512 can indicate the class of the machine and the surrounding environment. In each of the second learning data sets 52, the second sample 521 can be a sample of sound data including the abnormal sound P22, and the second label 522 can indicate the class of the abnormality.

[0120] The kind of the machine and the abnormality can be arbitrarily selected. The machine can be, for example, a device constituting a production line such as a conveyance device or an industrial robot. The machine can be the entire device, or a part of the device such as a motor. The surrounding environment of the machine can be distinguished, for example, according to the driving product of the machine, the time, or the like. The abnormality can be, for example, a malfunction, a foreign matter mixed in, a dirt attached, or a component wear.

[0121] In this example, the learning device 1 of this embodiment can construct a first generator 411 by performing machine learning using the aforementioned multiple first learning datasets 51. The first generator 411 is trained to generate pseudo-samples including machine and ambient sound P21 of the surrounding environment, corresponding to the input category. Furthermore, the learning device 1 of this embodiment can construct a second generator 421 by performing machine learning using the aforementioned multiple second learning datasets 52. The second generator 421 is trained to generate pseudo-samples including abnormal sound P22 of the corresponding input category. In the data generation device 2 of this embodiment, by utilizing the learned first generator 411 and second generator 421 respectively, most of the processing for generating new samples 65 of ambient sound P21 and abnormal sound P22, including combinations of arbitrary categories, can be automated. Therefore, the cost of collecting various different samples including ambient sound P21 and abnormal sound P22 can be reduced. Furthermore, in the inference device 3 of this embodiment, by adding the generated new samples 65 to the samples 321 contained in the data set 320, the accuracy of detecting machine anomalies is expected to be improved.

[0122] It should be noted that, in Figure 2A and Figure 2B In the example, features with high occurrence rates that appear in both normal and abnormal situations are selected as the first feature, while features with low occurrence rates that appear only in abnormal situations are selected as the second feature. Thus, features with high occurrence rates that appear individually can be selected as the first feature, and features with low occurrence rates that appear together with the first feature can be selected as the second feature. In this case, detecting the second component is equivalent to detecting low-occurrence events such as anomalies.

[0123] §2 Examples of Composition

[0124] [Hardware Configuration]

[0125] <Learning Device>

[0126] Figure 3 This is an illustrative example of the hardware configuration of the learning device 1 in this embodiment. For example... Figure 3 As shown, the learning device 1 in this embodiment is a computer electrically connected to a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a driver 17. It should be noted that... Figure 3 In this context, the communication interface and the external interface are referred to as "Communication I / F" and "External I / F".

[0127] The control section 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as a hardware processor configured to perform information processing based on programs and various data. The storage section 12 is an example of a storage device, which is configured by, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage section 12 stores various information such as a learning program 81, first to fourth learning data sets 51 to 54, first to fourth learning result data 121 to 124, and the like.

[0128] The learning program 81 is a program for causing the learning device 1 to perform information processing described later, i.e., generation of each completed learning model by implementing machine learning. The learning program 81 includes a series of instructions for the information processing. The learning program 81 can be referred to as a "model generation program" or simply a "generation program". The first to fourth learning data sets 51 to 54 are respectively used for machine learning of each model. The first to fourth learning result data 121 to 124 respectively represent information related to each completed learning model constructed by machine learning. In the present embodiment, the first to fourth learning result data 121 to 124 are respectively generated as a result of execution of the learning program 81. Details will be described later. Figure 10A to Figure 14 ) of each completed learning model. In the present embodiment, the first to fourth learning result data 121 to 124 are respectively generated as a result of execution of the learning program 81. Details will be described later.

[0129] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, which is an interface for wired or wireless communication via a network. The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, which is an interface for connection with an external device. The kind and number of the external interface 14 can be arbitrarily selected. The learning device 1 can be connected with a sensor for obtaining samples such as the first sample 511 and the second sample 521 via the external interface 14.

[0130] The input device 15 is, for example, a mouse, a keyboard, or the like, which is a device for input. In addition, the output device 16 is, for example, a display, a speaker, or the like, which is a device for output. An operator such as a user can operate the learning device 1 by using the input device 15 and the output device 16.

[0131] The drive 17 is, for example, a CD drive, a DVD drive, or the like, which is a drive device for reading various information such as programs stored in a storage medium 91. The kind of the drive 17 can be appropriately selected according to the kind of the storage medium 91. At least one set of the above-described learning program 81 and the first to fourth learning data sets 51 to 54 can be stored in the storage medium 91.

[0132] The storage medium 91 is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action so that a device such as a computer, a machine, or the like can read various information stored therein. The learning device 1 can acquire at least one of the learning program 81 and the first to fourth learning data sets 51 to 54 from the storage medium 91.

[0133] Here, Figure 3 In this embodiment, as an example of the storage medium 91, a disc-type storage medium such as a CD or a DVD is illustrated. However, the type of the storage medium 91 is not limited to the disc-type, and can be in a form other than the disc-type. As the storage medium other than the disc-type, a semiconductor memory such as a flash memory can be cited. The type of the drive 17 can be appropriately selected in accordance with the type of the storage medium 91.

[0134] Note that, regarding the specific hardware configuration of the learning device 1, components can be appropriately omitted, replaced, and added in accordance with the embodiment. For example, the control section 11 can include a plurality of hardware processors. The hardware processor can be constituted by a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage section 12 can be constituted by a RAM and a ROM included in the control section 11. At least any one of the communication interface 13, the external interface 14, the input device 15, the output device 16, and the drive 17 can be omitted. The learning device 1 can be constituted by a plurality of computers. In this case, the hardware configuration of each computer can be identical or different. In addition, the learning device 1 can be an information processing device designed specifically for a service provided, and a general-purpose server device, a PC (Personal Computer), or the like.

[0135] <DATA GENERATION DEVICE>

[0136] Figure 4 An example of the hardware configuration of the data generation device 2 of the present embodiment is schematically illustrated. As illustrated in FIG. 2, the data generation device 2 of the present embodiment is a computer to which a control section 21, a storage section 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a drive 27 are electrically connected. Figure 4

[0137] ​The control unit 21, driver 27, and storage medium 92 of the data generation device 2 are configured identically to the control unit 11, driver 17, and storage medium 91 of the learning device 1. The control unit 21 includes a hardware processor (CPU), RAM, ROM, etc., and is configured to perform various information processing based on programs and data. The storage unit 22 is configured as, for example, a hard disk drive, a solid-state drive, etc. The storage unit 22 stores various information such as the data generation program 82 and the first to third learning result data 121 to 123. The data generation program 82 is used to cause the data generation device 2 to perform information processing related to the generation of the new sample 65 (described later). Figure 15 The data generation program 82 includes a series of instructions for information processing. The data generation program 82 and at least one of the first to third learning result data 121 to 123 can be stored in the storage medium 92. Additionally, the data generation device 2 can retrieve at least one of the aforementioned data generation program 82 and the first to third learning result data 121 to 123 from the storage medium 92.

[0138] It should be noted that the specific hardware configuration of the data generation device 2 can be appropriately omitted, replaced, or added depending on the implementation method. For example, the control unit 21 may include multiple hardware processors. The hardware processors may be composed of microprocessors, FPGAs, DSPs, etc. The storage unit 22 may be composed of RAM and ROM contained in the control unit 21. At least one of the communication interface 23, external interface 24, input device 25, output device 26, and driver 27 may be omitted. The data generation device 2 may be composed of multiple computers. In this case, the hardware configuration of each computer may be the same or different. In addition, the learning device 2 may also be an information processing device specifically designed for the service provided, as well as a general-purpose server device, PC (Personal Computer), etc.

[0139] <Speculation Device>

[0140] Figure 5 This is an illustrative example of the hardware configuration of the speculative device 3 in this embodiment. For example... Figure 5 As shown, the speculative device 3 in this embodiment is a computer electrically connected to a control unit 31, a storage unit 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a driver 37.

[0141] The control section 31 to the driver 37 and the storage medium 93 of the estimation device 3 can be configured identically to the control section 11 to the driver 17 and the storage medium 91 of the learning device 1 described above. The control section 31 includes a hardware processor CPU, a RAM, a ROM, and the like, and is configured to perform various information processing based on a program and data. The storage section 32 is configured by, for example, a hard disk drive, a solid state drive, and the like. The storage section 32 stores various information such as the estimation program 83, a data group 320, the first / fourth learning result data (121, 124), and the like. The estimation program 83 is a program for causing the estimation device 3 to perform information processing (described later) related to a prescribed estimation for the second feature. Figure 17A The data group 320 can be used to determine the value of a parameter in the prescribed estimation (detection of the second component in the present embodiment) processing. At least any one of the estimation program 83, the data group 320, and the first / fourth learning result data (121, 124) can be stored in the storage medium 93. In addition, the estimation device 3 can load any one of the estimation program 83, the data group 320, and the first / fourth learning result data (121, 124) from the storage medium 93.

[0142] Note that, regarding the specific hardware configuration of the estimation device 3, components can be omitted, replaced, and added as appropriate according to the embodiment. For example, the control section 31 can include a plurality of hardware processors. The hardware processors can be configured by a microprocessor, an FPGA, a DSP, and the like. The storage section 32 can be configured by a RAM and a ROM included in the control section 31. At least any one of the communication interface 33, the external interface 34, the input device 35, the output device 36, and the driver 37 can be omitted. The estimation device 3 can be configured by a plurality of computers. In this case, the hardware configurations of the respective computers can be identical or different. In addition, the estimation device 3 can also be an information processing device designed specifically for a service provided, and a general-purpose server device, a PC, a PLC (programmable logic controller), and the like.

[0143] [Software Configuration]

[0144] [Learning Device]

[0145] Figure 6 An example of the software configuration of the learning device 1 of the present embodiment is schematically shown. The control section 11 of the learning device 1 expands the learning program 81 stored in the storage section 12 to a RAM. Next, the control section 11 interprets and executes the instructions included in the learning program 81 expanded to the RAM by a CPU to control the components. Thus, as shown in FIG. 2, the learning device 1 performs the learning processing (described later) related to the prescribed learning for the first feature. Figure 6As illustrated, the learning device 1 of the present embodiment functions as a computer that includes the first acquisition unit 111, the second acquisition unit 112, the third acquisition unit 113, the fourth acquisition unit 114, the first training unit 115, the second training unit 116, the third training unit 117, the fourth training unit 118, and the saving processing unit 119 as software modules. That is, in the present embodiment, each software module of the learning device 1 is implemented by the control unit 11 (CPU).

[0146] The first acquisition unit 111 acquires a plurality of first learning data sets 51 each composed of a combination of a first sample 511 including a first feature and a first label 512 representing a category of the first feature, where the first sample 511 is a first sample 511 of data of a prescribed kind. The first training unit 115 implements machine learning of a first learning model 41 including the first generator 411 using the plurality of first learning data sets 51 acquired. In the machine learning of the first learning model 41, the first training unit 115 trains the first generator 411 with respect to each first learning data set 51 so that a pseudo sample generated by the first generator 411 from the first label 512 matches the first sample 511.

[0147] The second acquisition unit 112 acquires a plurality of second learning data sets 52 each composed of a combination of a second sample 521 including a second feature different from the first feature and a second label 522 representing a category of the second feature, where the second sample 521 is a second sample 521 of data of a prescribed kind. The second training unit 116 implements machine learning of a second learning model 42 including the second generator 421 using the plurality of second learning data sets 52 acquired. In the machine learning of the second learning model 42, the second training unit 116 trains the second generator 421 with respect to each second learning data set 52 so that a pseudo sample generated by the second generator 421 from the second label 522 matches the second sample 521.

[0148] The third acquisition unit 113 acquires a plurality of third learning data sets 53 each composed of a sample value 531 of an attribute of the second feature, a reflection level 532 indicating a degree of reflection of the second feature, and the second label 522. The sample value 531 is also referred to as an attribute sample value. The attribute is related to a degree or a classification of a property possessed by the second feature, such as a size, a (temporal or regional) position, a brightness, an amplitude, a frequency (temporal or regional), and the like. The third training unit 117 performs machine learning of the third learning model 43 including the third generator 431 using the acquired plurality of third learning data sets 53. In the machine learning of the third learning model 43, the third training unit 117 trains the third generator 431 for each of the third learning data sets 53 so that a pseudo value of the attribute of the second feature generated by the third generator 431 from the reflection level 532 and the second label 522 matches the sample value 531. Note that each of the third learning data sets 53 can further have the first label 512. In this case, in the machine learning of the third learning model 43, the third training unit 117 can train the third generator 431 so that a pseudo value of the attribute of the second feature generated by the third generator 431 from the reflection level 532, the second label 522, and the first label 512 matches the sample value 531.

[0149] The fourth acquisition unit 114 acquires a plurality of fourth learning data sets 54 each composed of a sample value 541 of an input provided to the first generator 411 on which the training is completed and a pseudo sample 542 generated by providing the sample value 541 to the first generator 411 on which the training is completed. The pseudo sample 542 is an example of the "fourth pseudo sample" of the present application. The fourth training unit 118 performs machine learning of the predictor 44 using the acquired plurality of fourth learning data sets 54. In the machine learning of the predictor 44, the fourth training unit 118 trains the predictor 44 for each of the fourth learning data sets 54 so that a predicted value obtained by predicting the input provided to the first generator 411 on which the training is completed using the predictor 44 from the pseudo sample 542 matches the sample value 541.

[0150] The saving processing unit 119 generates information related to the result of each of the machine learning and saves the generated information in a prescribed storage area. Specifically, the saving processing unit 119 generates information related to each of the generators (411, 421, 431) and the predictor 44 on which the training is completed, respectively, as first to fourth learning result data 121 to 124. Next, the saving processing unit 119 saves the generated first to fourth learning result data 121 to 124 in a prescribed storage area. The prescribed storage area can not be particularly limited and can be appropriately selected depending on the embodiment. The prescribed storage area can be, for example, a RAM within the control unit 11, the storage unit 12, the storage medium 91, an external storage device, or a combination of these.

[0151] (machine learning of the first learning model)

[0152] Figure 7A An example of a process of machine learning of the first learning model 41 of the illustrative example embodiment. As the configuration of the first learning model 41, as long as the first generator 411 can be trained to generate a pseudo sample including a first feature of a category corresponding to an input, the first learning model 41 can not be particularly limited and can be appropriately determined according to the embodiment. In the present embodiment, the first learning model 41 can further include the first discriminator 412. The first generator 411 and the first discriminator 412 can be configured by any machine learning model capable of implementing machine learning.

[0153] In the present embodiment, the first generator 411 and the first discriminator 412 are each configured by a neural network of a multi-layer structure for deep learning. The first generator 411 and the first discriminator 412 each have an input layer (4111, 4121), an intermediate (hidden) layer (4112, 4122), and an output layer (4113, 4123). However, the structure of each of the first generator 411 and the first discriminator 412 can not be limited to such an example and can be appropriately determined according to the embodiment. For example, the number of intermediate layers (4112, 4122) can not be limited to one and can be two or more. Alternatively, the intermediate layer (4112, 4122) can be omitted. The number of each layer configuring the neural network can not be particularly limited and can be arbitrarily selected. In addition, the structures of the first generator 411 and the first discriminator 412 can be at least partially identical or can not be identical.

[0154] Each layer (4111 to 4113, 4121 to 4123) has one or more neurons (nodes). The number of neurons (nodes) included in each layer (4111 to 4113, 4121 to 4123) can not be particularly limited and can be appropriately determined according to the embodiment. Neurons of adjacent layers are appropriately combined with each other. In the example of FIG. 4, each neuron is combined with all neurons of the adjacent layer. However, the combination relationship of each neuron can not be limited to such an example and can be appropriately set according to the embodiment. Figure 7A

[0155] ​A weight value (binding load) is provided for each binding. A threshold value is provided for each neuron, and the output of each neuron is basically determined by whether the sum of the products of each input and each weight value exceeds the threshold value. The threshold value can be expressed by an activation function. In this case, the sum of the products of each input and each weight value is input to the activation function, and the operation of the activation function is performed, thereby determining the output of each neuron. The kind of the activation function can not be particularly limited, and can be appropriately selected depending on the embodiment. The weight value of the binding between each neuron included in each layer (4111 to 4113, 4121 to 4123) and the threshold value of each neuron are examples of the operation parameters of the first generator 411 and the first discriminator 412, respectively.

[0156] In the present embodiment, the first generator 411 is trained by machine learning by alternately repeating a first training step of training the first discriminator 412 and a second training step of training the first generator 411. In the first training step, the first training unit 115 trains the first discriminator 412 to recognize whether an input sample input to the first discriminator 412 is a first sample 511 obtained from any one of the plurality of first learning data sets 51 or a pseudo sample 515 generated by the first generator 411. In other words, the first discriminator 412 is trained to recognize whether the input sample is from the plurality of first learning data sets 51 (learning data) or from the first generator 411. Note that, in the present embodiment, the first discriminator 412 is trained to recognize whether the input sample is from the plurality of first learning data sets 51 or from the first generator 411. However, the present embodiment is not limited to this example. For example, the first discriminator 412 can be trained to recognize whether the input sample is from the first generator 411 or from the plurality of first learning data sets 51. Figure 7A In the example of the present embodiment, the origin from the learning data is expressed as "true", and the origin from the first generator 411 is expressed as "pseudo". "True" can be replaced with "1", and "pseudo" can be replaced with "0". However, the method of expressing each origin can not be limited to this example, and can be appropriately selected depending on the embodiment. On the other hand, in the second training step, the first training unit 115 trains the first generator 411 to generate a pseudo sample 515 such as to reduce the performance of recognition by the first discriminator 412. That is, the machine learning of the first learning model 41 is constituted by the adversarial learning between the first generator 411 and the first discriminator 412.

[0157] As an example of the processing of the machine learning, in the first training step, first, the first training unit 115 extracts noise 510 (latent variable) from a prescribed probability distribution with respect to each first learning data set 51. The kind of the prescribed probability distribution can be arbitrarily selected. The prescribed probability distribution can be, for example, a known probability distribution such as a Gaussian distribution. Next, the first training unit 115 supplies the extracted noise 510 and the first label 512 to the first generator 411, and thereby generates a pseudo sample 515 from the extracted noise 510 and the first label 512. Specifically, the first training unit 115 inputs the extracted noise 510 and the first label 512 to the input layer 4111 of the first generator 411, and executes the operation processing of the first generator 411. Thereby, the first training unit 115 acquires, from the output layer 4113 of the first generator 411, an output (i.e., the pseudo sample 515) corresponding to the result of generating the pseudo sample from the noise 510 and the first label 512.

[0158] Next, the first training unit 115 inputs the generated pseudo sample 515 and the corresponding first label 512 to the input layer 4121 of the first recognizer 412, and executes the operation processing of the first recognizer 412. Thereby, the first training unit 115 acquires, from the output layer 4123 of the first recognizer 412, an output value corresponding to the recognition result of the input input sample being from the learning data or from the first generator 411. In this context, since the input input sample is the pseudo sample 515, the first recognizer 412 recognizes "fake" as the correct answer. The first training unit 115 calculates, with respect to each pseudo sample 515 generated by the first generator 411, an error of the output value obtained from the output layer 4123 from the correct answer.

[0159] In addition, the first training unit 115 inputs each first learning data set 51 (the first sample 511 and the first label 512) to the input layer 4121 of the first recognizer 412, and executes the operation processing of the first recognizer 412. Thereby, the first training unit 115 acquires, from the output layer 4123 of the first recognizer 412, an output value corresponding to the recognition result of the input input sample being from the learning data or from the first generator 411. In this context, since the input input sample is the first sample 511, the first recognizer 412 recognizes "true" as the correct answer. The first training unit 115 calculates, with respect to each first learning data set 51, an error of the output value obtained from the output layer 4123 from the correct answer.

[0160] And, after fixing the values of the operation parameters of the first generator 411, the first training section 115 adjusts the values of the operation parameters of the first recognizer 412 so as to reduce the sum of the calculated errors. Thus, the first training section 115 can train the first recognizer 412 to acquire the ability to recognize whether the input sample input to the first recognizer 412 is the first sample 511 obtained from any one of the plurality of first learning data sets 51 or the pseudo sample 515 generated by the first generator 411.

[0161] On the other hand, in the second training step, first, the first training section 115 extracts a noise 510 (latent variable) from a prescribed probability distribution for each of the first learning data sets 51. Next, the first training section 115 outputs the extracted noise 510 and the first label 512 to the input layer 4111 of the first generator 411 and executes the operation processing of the first generator 411. Thus, the first training section 115 acquires, from the output layer 4113 of the first generator 411, an output corresponding to the result of generating a pseudo sample from the noise 510 and the first label 512 (i.e., the pseudo sample 515). Note that the processing of generating the pseudo sample 515 can be commonly executed in the first training step and the second training step. That is, the pseudo sample 515 generated in the first training step can also be used in the second training step.

[0162] Next, the first training section 115 inputs the generated pseudo sample 515 and the corresponding first label 512 to the input layer 4121 of the first recognizer 412 and executes the operation processing of the first recognizer 412. Thus, the first training section 115 acquires, from the output layer 4123 of the first recognizer 412, an output value corresponding to the recognition result of the input input sample, whether it is from the learning data or from the first generator 411. In the training of the first generator 411, it is the correct answer that the recognition performance of the first recognizer 412 is reduced (i.e., the recognition result is wrong). That is, it is the correct answer that the output value obtained from the output layer 4123 corresponds to "true". The first training section 115 calculates, for each pseudo sample 515, an error of the output value obtained from the output layer 4123 from this correct answer.

[0163] And, after fixing the values of the operation parameters of the first recognizer 412, the first training section 115 adjusts the values of the operation parameters of the first generator 411 so as to reduce the sum of the calculated errors. Thus, the first training section 115 trains the first generator 411 so that it acquires the ability to generate a pseudo sample 515 that reduces the recognition performance of the first recognizer 412.

[0164] The first training unit 115 can gradually improve the performance of the first recognizer 412 and the first generator 411 by alternately repeating the first training step and the second training step described above. Thus, according to the improvement of the recognition performance of the first recognizer 412, it is expected that the performance of the first generator 411 that generates the pseudo sample 515 matching the first sample 511 can be improved. Therefore, in the present embodiment, by the machine learning described above, it is possible to construct the first generator 411 of the completed training that can generate a suitable pseudo sample including the first feature. Note that the processing order of the first training step and the second training step can be arbitrary.

[0165] After the machine learning ends, the saving processing unit 119 generates information indicating the structure and the operation parameter of the constructed first generator 411 of the completed training as the first learning result data 121. Next, the saving processing unit 119 saves the generated first learning result data 121 in a prescribed storage area. Note that as the content of the first learning result data 121, as long as the first generator 411 of the completed training can be reproduced, it can not be limited to such an example. For example, in a case where the structure of the first generator 411 is made the same among the devices, the information indicating the structure of the first generator 411 can be omitted in the first learning result data 121. In addition, the first learning result data 121 can further include information indicating the structure and the operation parameter of the constructed first recognizer 412 of the completed training.

[0166] (Machine learning of second learning model)

[0167] Figure 7B An example of the process of the machine learning of the second learning model 42 of the present embodiment is schematically shown. As the configuration of the second learning model 42, as long as the second generator 421 can be trained to generate a pseudo sample including a second feature of a category corresponding to an input, it can not be particularly limited and can be appropriately determined according to the embodiment. In the present embodiment, like the first learning model 41, the second learning model 42 further includes the second recognizer 422. The second generator 421 and the second recognizer 422 can be configured by any machine learning model that can perform machine learning.

[0168] In this embodiment, similar to the first generator 411 described above, the second generator 421 and the second recognizer 422 are composed of a multi-layered neural network for deep learning. The second generator 421 and the second recognizer 422 each have an input layer (4211, 4221), intermediate (hidden) layers (4212, 4222), and an output layer (4213, 4223). However, similar to the first generator 411 described above, the structures of the second generator 421 and the second recognizer 422 are not limited to this example and can be appropriately determined according to the implementation. The number of layers constituting the neural network can be arbitrarily chosen. The structures of the second generator 421 and the second recognizer 422 can be at least partially identical, or they can be different.

[0169] Each layer (4211-4213, 4221-4223) has one or more neurons (nodes). The number of neurons (nodes) in each layer (4211-4213, 4221-4223) is not particularly limited and can be appropriately determined according to the implementation method. Neurons in adjacent layers are appropriately combined with each other. Figure 7B In the example, each neuron combines with all neurons in the adjacent layer. However, the combination relationships of each neuron are not limited to this example and can be appropriately set according to the implementation method. A weighted value is set for each combination. A threshold is set for each neuron. The threshold can be expressed by an activation function. The type of activation function is not particularly limited and can be appropriately selected according to the implementation method. The weighted value of the combination between each neuron contained in each layer (4211-4213, 4221-4223) and the threshold of each neuron are examples of the operation parameters of the second generator 421 and the second recognizer 422, respectively.

[0170] In this embodiment, the second generator 421 is trained by machine learning by alternately and repeatedly executing the third training step of training the second recognizer 422 and the fourth training step of training the second generator 421. In the third training step, the second training unit 116 trains the second recognizer 422 to identify whether the input sample to the second recognizer 422 is a second sample 521 obtained from any one of the multiple second learning datasets 52 or a pseudo sample 525 generated by the second generator 421. In other words, the second recognizer 422 is trained to identify whether the input sample comes from the multiple second learning datasets 52 (learning data) or from the second generator 421. It should be noted that in Figure 7B In the example above, each source, through [the aforementioned] Figure 7AThe same method is used for expression, but the method of expression of each source can not be limited to such examples, and can be appropriately selected according to the embodiment. On the other hand, in the fourth training step, the second training unit 116 trains the second generator 421 to generate the pseudo sample 525 like reducing the performance of the recognition of the second discriminator 422. That is, the machine learning of the second learning model 42 is constituted by the adversarial learning between the second generator 421 and the second discriminator 422.

[0171] The processing of the machine learning of the second learning model 42 can be the same as the machine learning of the first learning model 41 described above except for the data to be processed. That is, in the third training step, first, the second training unit 116 extracts the noise 520 (latent variable) from a prescribed probability distribution (for example, Gaussian distribution or the like) for each second learning data set 52. Next, the second training unit 116 inputs the extracted noise 520 and the second label 522 to the input layer 4211 of the second generator 421 and executes the operation processing of the second generator 421. Thereby, the second training unit 116 acquires the output (that is, the pseudo sample 525) corresponding to the result of generating the pseudo sample from the noise 520 and the second label 522 from the output layer 4213 of the second generator 421.

[0172] Next, the second training unit 116 inputs the generated pseudo sample 525 and the corresponding second label 522 to the input layer 4221 of the second discriminator 422 and executes the operation processing of the second discriminator 422. Thereby, the second training unit 116 acquires the output value corresponding to the result of the recognition of the input sample from the output layer 4223 of the second discriminator 422. In this context, since the input sample is the pseudo sample 525, it is the correct answer that the second discriminator 422 recognizes as "fake". The second training unit 116 calculates the error of the output value from the output layer 4223 and the correct answer for each pseudo sample 525 generated by the second generator 421.

[0173] In addition, the second training unit 116 inputs each second learning data set 52 (the second sample 521 and the second label 522) to the input layer 4221 of the second discriminator 422 and executes the operation processing of the second discriminator 422. Thereby, the second training unit 116 acquires the output value corresponding to the result of the recognition of the input sample from the output layer 4223 of the second discriminator 422. In this context, since the input sample is the second sample 521, it is the correct answer that the second discriminator 422 recognizes as "true". The second training unit 116 calculates the error of the output value from the output layer 4223 and the correct answer for each second learning data set 52.

[0174] And, after the second training unit 116 fixes the values of the operation parameters of the second generator 421, the second training unit 116 adjusts the values of the operation parameters of the second discriminator 422 so as to reduce the sum of the errors calculated. Thus, the second training unit 116 is able to train the second discriminator 422 so as to obtain the ability to identify whether the input sample input to the second discriminator 422 is the second sample 521 obtained from any one of the plurality of second learning data sets 52 or the pseudo sample 525 generated by the second generator 421.

[0175] On the other hand, in the fourth training step, first, the second training unit 116 extracts a noise 520 (a latent variable) from a prescribed probability distribution for each of the second learning data sets 52. Next, the second training unit 116 inputs the extracted noise 520 and the second label 522 to the input layer 4211 of the second generator 421 and executes the operation processing of the second generator 421. Thus, the second training unit 116 acquires, from the output layer 4213 of the second generator 421, an output corresponding to the result of generating a pseudo sample from the noise 520 and the second label 522 (i.e., the pseudo sample 525). Note that the processing of generating the pseudo sample 525 can be executed commonly in the third training step and the fourth training step. That is, the pseudo sample 525 generated in the third training step can also be used in the fourth training step.

[0176] Next, the second training unit 116 inputs the generated pseudo sample 525 and the corresponding second label 522 to the input layer 4221 of the second discriminator 422 and executes the operation processing of the second discriminator 422. Thus, the second training unit 116 acquires, from the output layer 4223 of the second discriminator 422, an output value corresponding to the result of the identification of the input sample. In the training of the second generator 421, it is the correct answer that the identification performance of the second discriminator 422 is reduced (i.e., the identification result is wrong). That is, it is the correct answer that the output value obtained from the output layer 4223 corresponds to "true". The second training unit 116 calculates, for each pseudo sample 525, an error of the output value obtained from the output layer 4223 from this correct answer.

[0177] And, after the second training unit 116 fixes the values of the operation parameters of the second discriminator 422, the second training unit 116 adjusts the values of the operation parameters of the second generator 421 so as to reduce the sum of the errors calculated. Thus, the second training unit 116 is able to train the second generator 421 so as to obtain the ability to generate a pseudo sample 525 that reduces the identification performance of the second discriminator 422.

[0178] The second training unit 116 can gradually improve the performance of the second recognizer 422 and the second generator 421 by alternately repeating the third training step and the fourth training step described above. Thus, according to the improvement of the recognition performance of the second recognizer 422, it is expected that the performance of the second generator 421 that generates the pseudo sample 525 matching the second sample 521 can be improved. Therefore, in the present embodiment, by the machine learning described above, it is possible to construct the second generator 421 of the completed training that can generate a suitable pseudo sample including the second feature. Note that the processing order of the third training step and the fourth training step can be arbitrary.

[0179] After the machine learning ends, the saving processing unit 119 generates information indicating the structure and the operation parameter of the constructed second generator 421 of the completed training as the second learning result data 122. Next, the saving processing unit 119 saves the generated second learning result data 122 in a prescribed storage area. Note that as the content of the second learning result data 122, as long as the second generator 421 of the completed training can be reproduced, it can not be limited to such an example. For example, in a case where the structure of the second generator 421 is made the same among the devices, information indicating the structure of the second generator 421 can be omitted in the second learning result data 122. In addition, the second learning result data 122 can further include information indicating the structure and the operation parameter of the constructed second recognizer 422 of the completed training.

[0180] (Machine learning of third learning model)

[0181] Figure 7C An example of the process of the machine learning of the third learning model 43 of the present embodiment is schematically illustrated. As the configuration of the third learning model 43, as long as the third generator 431 can be trained to generate a pseudo value (pseudo value) of an attribute of a second feature including a category and a degree (reflection level) corresponding to an input, it can not be particularly limited and can be appropriately determined according to the embodiment. In the present embodiment, the third learning model 43 also includes the third recognizer 432, like the first learning model 41 and the like. The third generator 431 and the third recognizer 432 can be configured of any machine learning model that can implement machine learning.

[0182] In the present embodiment, the third generator 431 and the third discriminator 432 are configured by a neural network of a multi-layer structure for deep learning, like the first generator 411 and the like. The third generator 431 and the third discriminator 432 each have an input layer (4311, 4321), an intermediate (hidden) layer (4312, 4322), and an output layer (4313, 4323). However, like the first generator 411 and the like, the configuration of the third generator 431 and the third discriminator 432 can not be limited to this example, and can be appropriately determined according to the embodiment. The number of each layer configuring the neural network can be arbitrarily selected. The configurations of the third generator 431 and the third discriminator 432 can be identical at least in part, or can be different.

[0183] Each layer (4311 to 4313, 4321 to 4323) has one or more neurons (nodes). The number of neurons (nodes) included in each layer (4311 to 4313, 4321 to 4323) can not be particularly limited, and can be appropriately determined according to the embodiment. Neurons of adjacent layers are appropriately coupled to each other. In the example of FIG. 4, each neuron is coupled to all neurons of the adjacent layer. However, the coupling relationship of each neuron can not be limited to this example, and can be appropriately set according to the embodiment. A weighting value is set for each coupling. A threshold value is set for each neuron. The threshold value can be expressed by an activation function. The type of the activation function can not be particularly limited, and can be appropriately selected according to the embodiment. The weighting value of the coupling between each neuron included in each layer (4311 to 4313, 4321 to 4323) and the threshold value of each neuron are examples of the operation parameters of the third generator 431 and the third discriminator 432, respectively. Figure 7C

[0184] In the present embodiment, the third generator 431 is trained by machine learning by alternately repeating a fifth training step of training the third discriminator 432 and a sixth training step of training the third generator 431. In the fifth training step, the third training unit 117 trains the third discriminator 432 to discriminate whether the input value input to the third discriminator 432 is a sample value 531 obtained from any one of the plurality of third learning data sets 53 or a pseudo value 535 generated by the third generator 431. In other words, the third discriminator 432 is trained to discriminate whether the input value is from the plurality of third learning data sets 53 (learning data) or from the third generator 431. Note that, in the example of FIG. 4, the third discriminator 432 is trained to discriminate whether the input value is from the plurality of third learning data sets 53 or from the third generator 431. However, the third discriminator 432 can be trained to discriminate whether the input value is from the plurality of third learning data sets 53 or from the third generator 431, and the third discriminator 432 can be trained to discriminate whether the input value is from the plurality of third learning data sets 53 or from the second generator 421. Figure 7C Figure 7A ​​The same method as described above is applied to the expression of the second feature, but the method of expression of each source can not be limited to such an example, and can be appropriately selected according to the embodiment. On the other hand, in the sixth training step, the third training unit 117 trains the third generator 431 to generate a pseudo value 535 of the attribute of the second feature such as to reduce the performance of the recognition of the third discriminator 432. That is, the machine learning of the third learning model 43 is constituted by the adversarial learning between the third generator 431 and the third discriminator 432.

[0185] The processing of the machine learning of the third learning model 43 can be the same as the machine learning of the first learning model 41 and the like described above except for the difference in the data to be processed. That is, in the fifth training step, first, the third training unit 117 extracts a noise 530 (latent variable) from a prescribed probability distribution (for example, a Gaussian distribution and the like) for each third learning data set 53. Next, the third training unit 117 inputs the extracted noise 530, the corresponding reflection level 532, and the corresponding second label 522 to the input layer 4311 of the third generator 431, and executes the operation processing of the third generator 431. Thereby, the third training unit 117 acquires an output (that is, the pseudo value 535) corresponding to the result of generating a pseudo value of the attribute of the second feature from the noise 530, the reflection level 532, and the second label 522 from the output layer 4313 of the third generator 431. In a case where each third learning data set 53 also has a first label 512, the third training unit 117 can further input the corresponding first label 512 to the input layer 4311 of the third generator 431, and execute the above processing.

[0186] Next, the third training unit 117 inputs the generated pseudo value 535, the corresponding reflection level 532, and the corresponding second label 522 to the input layer 4321 of the third discriminator 432, and executes the operation processing of the third discriminator 432. Thereby, the third training unit 117 acquires an output value corresponding to the result of the recognition of the input value from the output layer 4323 of the third discriminator 432. In this context, since the input value is the pseudo value 535, the third discriminator 432 recognizes that "fake" is the correct answer. The third training unit 117 calculates the error of the output value obtained from the output layer 4323 from this correct answer for each pseudo value 535 generated by the third generator 431.

[0187] In addition, the third training unit 117 inputs each of the third learning data sets 53 (the sample value 531, the reflection level 532, and the second label 522) to the input layer 4321 of the third discriminator 432, and executes the operation processing of the third discriminator 432. Thereby, the third training unit 117 acquires, from the output layer 4323 of the third discriminator 432, an output value corresponding to the result of the recognition of the input value. In this case, since the input value is the sample value 531, the third discriminator 432 recognizes that "true" is the correct answer. The third training unit 117 calculates, for each of the third learning data sets 53, an error of the output value obtained from the output layer 4323 from the correct answer. In a case where each of the third learning data sets 53 also has the first label 512, the third training unit 117 can further input the corresponding first label 512 to the input layer 4321 of the third discriminator 432, and execute the above processing.

[0188] Further, after the third training unit 117 fixes the values of the operation parameters of the third generator 431, the third training unit 117 adjusts the values of the operation parameters of the third discriminator 432 so as to reduce the sum of the calculated errors. Thereby, the third training unit 117 can train the third discriminator 432 to obtain the ability to recognize whether the input value input to the third discriminator 432 is the sample value 531 obtained from any one of the plurality of third learning data sets 53 or the pseudo value 535 generated by the third generator 431.

[0189] On the other hand, in the sixth training step, first, the third training unit 117 extracts the noise 530 (the latent variable) from the prescribed probability distribution for each of the third learning data sets 53. Next, the third training unit 117 inputs the extracted noise 530, the corresponding reflection level 532, and the corresponding second label 522 to the input layer 4311 of the third generator 431, and executes the operation processing of the third generator 431. Thereby, the third training unit 117 acquires, from the output layer 4313 of the third generator 431, an output (i.e., the pseudo value 535) corresponding to the result of generating the pseudo value of the attribute of the second feature from the noise 530, the reflection level 532, and the second label 522. In a case where each of the third learning data sets 53 also has the first label 512, the third training unit 117 can further input the corresponding first label 512 to the input layer 4311 of the third generator 431, and execute the above processing. Note that the processing of generating the pseudo value 535 can be executed commonly in the fifth training step and the sixth training step. That is, the pseudo value 535 generated in the fifth training step can also be used in the sixth training step.

[0190] Next, the third training unit 117 inputs the generated pseudo value 535, the corresponding reflection level 532, and the corresponding second label 522 into the input layer 4321 of the third discriminator 432, and performs the operation processing of the third discriminator 432. Thereby, the third training unit 117 acquires an output value corresponding to the result of the recognition of the input value from the output layer 4323 of the third discriminator 432. In the training of the third generator 431, it is the correct answer that the recognition performance of the third discriminator 432 is reduced (i.e., the recognition result is wrong). That is, it is the correct answer that the output value obtained from the output layer 4323 corresponds to "true". The third training unit 117 calculates the error of the output value obtained from the output layer 4323 from the correct answer for each pseudo value 535. In a case where each third learning data set 53 further has the first label 512, the third training unit 117 can further input the corresponding first label 512 into the input layer 4321 of the third discriminator 432, and perform the above processing.

[0191] Further, after the third training unit 117 fixes the values of the operation parameters of the third discriminator 432, the third training unit 117 adjusts the values of the operation parameters of the third generator 431 so as to reduce the sum of the calculated errors. Thereby, the third training unit 117 can train the third generator 431 so as to acquire the ability to generate a pseudo value 535 that reduces the performance of the recognition of the third discriminator 432.

[0192] The third training unit 117 can gradually and alternately improve the performance of the third discriminator 432 and the third generator 431 by alternately repeating the above-described fifth training step and the sixth training step. Thereby, according to the improvement of the recognition performance of the third discriminator 432, it is expected that the performance of the third generator 431 that generates a pseudo value 535 matching the sample value 531 can be improved. Therefore, in the present embodiment, by the above-described machine learning, a third generator 431 that is completely trained to be able to generate a suitable pseudo value of the attribute of the second feature can be constructed from the input value and the noise corresponding to the reflection level 532 and the second label 522, respectively. In a case where the first label 512 is further input in each of the above-described training steps, a third generator 431 that is completely trained to be able to generate a suitable pseudo value of the attribute of the second feature can be constructed from the input value and the noise corresponding to the reflection level 532, the second label 522, and the first label 512, respectively. Note that the processing order of the fifth training step and the sixth training step is arbitrary.

[0193] After the machine learning process is completed, the storage processing unit 119 generates information representing the structure and operational parameters of the constructed and trained third generator 431 as third learning result data 123. Then, the storage processing unit 119 saves the generated third learning result data 123 in a designated storage area. It should be noted that the content of the third learning result data 123 is not limited to such examples, as long as the trained third generator 431 can be regenerated. For example, if the structure of the third generator 431 is the same across devices, the information representing the structure of the third generator 431 can be omitted from the third learning result data 123. Furthermore, the third learning result data 123 may also include information representing the structure and operational parameters of the constructed and trained third recognizer 432.

[0194] (Machine learning for inferring)

[0195] Figure 7D This is an illustrative example of the machine learning process of the specifier 44 in this embodiment. The specifier 44 can be constructed from any machine learning model capable of implementing machine learning. In this embodiment, similar to the first generator 411 described above, the specifier 44 is constructed from a multi-layered neural network for deep learning. The specifier 44 includes an input layer 441, intermediate (hidden) layers 442, and an output layer 443. However, similar to the first generator 411 described above, the structure of the specifier 44 is not limited to this example and can be appropriately determined according to the embodiment. The number of layers in the neural network constituting the specifier 44 can be arbitrarily chosen.

[0196] Each layer 441-443 contains one or more neurons (nodes). The number of neurons (nodes) in each layer 441-443 is not particularly limited and can be appropriately determined according to the implementation method. Neurons in adjacent layers are appropriately combined with each other. Figure 7D In the example, each neuron binds to all neurons in the adjacent layer. However, the binding relationships of neurons are not limited to this example and can be appropriately set according to the implementation method. A weighted value is set for each binding. A threshold is set for each neuron. The threshold can be expressed by an activation function. The type of activation function is not particularly limited and can be appropriately selected according to the implementation method. The weighted values ​​of the bindings between neurons in each layer 441-443 and the threshold of each neuron are examples of the operational parameters of the predictor 44.

[0197] The inferencer 44 is trained by machine learning to infer the input provided to the first trained generator 411 when generating the pseudo sample therefrom. Each fourth learning data set 54 used for the machine learning can be appropriately generated by utilizing the first trained generator 411. As an example, a sample value 541 of the input provided to the first trained generator 411 is appropriately determined. The sample value 541 can be determined manually by an operator's input, or can also be determined automatically by a random method or the like. The sample value 541 corresponds to a first label 512 representing a class of the first feature. From a prescribed probability distribution (e.g., a Gaussian distribution or the like), a noise 540 (latent variable) is extracted corresponding to the sample value 541. The order of the process of determining the sample value 541 and the process of extracting the noise 540 can be arbitrary. The extracted noise 540 and sample value 541 are input to the input layer 4111 of the first trained generator 411, and the operation process of the first trained generator 411 is executed. Thereby, an output corresponding to the result of generating the pseudo sample from the noise 540 and sample value 541 (i.e., a pseudo sample 542) can be acquired from the output layer 4113 of the first trained generator 411. The generated pseudo sample 542 is associated with the corresponding noise 540 and sample value 541. Thereby, each fourth learning data set 54 can be generated.

[0198] In the present embodiment, the fourth training section 118 performs machine learning of the inferencer 44 configured of the neural network described above using the plurality of fourth learning data sets 54. In the machine learning of the inferencer 44, the fourth training section 118 uses the pseudo sample 542 in each fourth learning data set 54 as training data (input data), and uses the noise 540 and sample value 541 as correct answer data (teacher signal).

[0199] Specifically, the fourth training section 118 inputs the pseudo sample 542 to the input layer 441 of the inferencer 44, and executes the operation process of the inferencer 44. Thereby, the fourth training section 118 acquires an output value corresponding to the result of inferring the input provided to the first trained generator 411 when generating the pseudo sample 542 from the pseudo sample 542 from the output layer 443 of the inferencer 44. The obtained output value includes a first inferred value corresponding to the noise provided to the first trained generator 411 and a second inferred value corresponding to the sample value. The fourth training section 118 calculates the error of each inferred value from the noise 540 and sample value 541 for each fourth learning data set 54.

[0200] Further, the fourth training section 118 adjusts values of the operation parameters of the predictor 44 so as to reduce the sum of the calculated errors. Thus, the fourth training section 118 can construct the predictor 44 of which the training is completed, which has the ability to obtain the input (the noise 540 and the sample value 541) provided to the first generator 411 of which the training is completed at the time when the pseudo sample 542 is generated by the first generator 411 of which the training is completed, by using the pseudo sample 542.

[0201] After the machine learning ends, the saving processing section 119 generates information indicating the structure and the operation parameters of the predictor 44 of which the training is completed, which is constructed, as the fourth learning result data 124. Then, the saving processing section 119 saves the generated fourth learning result data 124 in a prescribed storage area. Note that, as the contents of the fourth learning result data 124, it is not limited to such an example as long as the predictor 44 of which the training is completed can be reproduced. For example, in a case where the structure of the predictor 44 is made the same among the devices, the information indicating the structure of the predictor 44 can be omitted in the fourth learning result data 124.

[0202] (Generation of second samples)

[0203] Figure 7E An example of the process of generating the second samples 521 of the present embodiment will be described. In the present embodiment, the second samples 521 of each second learning data set 52 can be generated by using the first generator 411 of which the training is completed and the predictor 44 generated by the machine learning described above.

[0204] Specifically, the second acquisition section 112 acquires a plurality of third samples 551 each including a first feature and a second feature, where the third samples 551 are a plurality of third samples 551 of data of a prescribed kind. Then, the second acquisition section 112 estimates each estimated value of the input provided to the first generator 411 of which the training is completed in order to generate a pseudo sample corresponding to each third sample 551, by providing each third sample 551 to the predictor 44 of which the training is completed. If a pseudo sample corresponding to each third sample 551 is generated by the first generator 411 of which the training is completed, the obtained estimated values (first estimated values and second estimated values) correspond to the input (noise and input value) that will be provided to the first generator 411 of which the training is completed.

[0205] Then, the second acquisition section 112 generates a pseudo sample 555 for each third sample 551 by providing the estimated estimated values (first estimated values and second estimated values) estimated to the first generator 411 of which the training is completed. The pseudo sample 555 is an example of the "third pseudo sample" of the present application. Further, the second acquisition section 112 generates the second sample 521 of each second learning data set 52 by calculating the difference between each third sample 551 and the pseudo sample 555.

[0206] The first generator 411 completed with training, although it has acquired the ability to generate pseudo samples including the first feature through the above-described machine learning, does not acquire the ability to generate pseudo samples including the second feature. Therefore, substantially, in the resultant pseudo samples 555, the first feature is reproduced, but the second feature is not reproduced. By this, through the process of taking the difference between the above-described third samples 551 and the pseudo samples 555, a sample that does not include the first feature and includes the second feature, which can be used as the second samples 521, can be appropriately generated.

[0207] <DATA GENERATION APPARATUS>

[0208] Figure 8 An example of a software configuration of the data generation apparatus 2 of the present embodiment is schematically shown. The control section 21 of the data generation apparatus 2 expands the data generation program 82 stored in the storage section 22 to the RAM. Next, the control section 21 controls each component by interpreting and executing the instructions contained in the data generation program 82 expanded to the RAM by the CPU. By this, as shown in Figure 8 the data generation apparatus 2 of the present embodiment functions as a computer that has the acceptance section 211, the first generation section 212, the second generation section 213, and the data synthesis section 214 as software modules. That is, in the present embodiment, like the learning apparatus 1, each software module of the data generation apparatus 2 is also realized by the control section 11 (CPU).

[0209] The acceptance section 211 accepts the designation of the first input value 221 and the second input value 223. The first input value 221 corresponds to the first label 512 that represents the category of the first feature, and the second input value 223 corresponds to the second label 522 that represents the category of the second feature.

[0210] The first generation section 212 has the first generator 411 completed with training by retaining the first learning result data 121. The first generation section 212 extracts a noise 220 (latent variable) from a prescribed probability distribution (for example, a Gaussian distribution or the like). The first generation section 212 generates the first pseudo sample 61 including the first feature by providing the extracted noise 220 and the first input value 221 to the first generator 411 completed with training.

[0211] The second generation section 213 has the second generator 421 completed with training by retaining the second learning result data 122. The second generation section 213 extracts a noise 222 (latent variable) from a prescribed probability distribution (for example, a Gaussian distribution or the like). The second generation section 213 generates the second pseudo sample 62 including the second feature by providing the extracted noise 222 and the second input value 223 to the second generator 421 completed with training.

[0212] The data synthesizing section 214 generates a new sample 65 of data of a prescribed kind by synthesizing the generated first pseudo sample 61 and the generated second pseudo sample 62. The method of synthesizing the first pseudo sample 61 and the second pseudo sample 62 can not be particularly limited and can be appropriately determined in accordance with the embodiment. In the present embodiment, the data synthesizing section 214 is provided with a third generator 431 for generating an attribute value of the second feature by completing training in accordance with the degree of reflection of the second feature by retaining the third learning result data 123. Thus, it is possible to synthesize the first pseudo sample 61 and the second pseudo sample 62 while adjusting the degree of reflection of the second feature.

[0213] Specifically, the accepting section 211 accepts designation of a third input value 225 corresponding to the reflection level 532 indicating the degree of reflection of the second feature. The third input value 225 can be provided in accordance with the limit of detection of the second component. The data synthesizing section 214 extracts noise 224 from a prescribed probability distribution (e.g., a Gaussian distribution or the like). The data synthesizing section 214 generates a pseudo value 63 of the attribute of the second feature by providing the extracted noise 224, the second input value 223, and the third input value 225 to the third generator 431 of which the training is completed. Note that in a case where the first label 512 is further used in the machine learning of the above-described third learning model 43, the third generator 431 is configured so as to be able to further input an input value corresponding to the first label 512. In this case, the data synthesizing section 214 generates the pseudo value 63 of the attribute of the second feature by providing the extracted noise 224, the first input value 221, the second input value 223, and the third input value 225 to the third generator 431 of which the training is completed.

[0214] Also, the data synthesizing section 214 converts the second pseudo sample 62 so that the value of the attribute of the second feature contained in the second pseudo sample 62 matches the generated pseudo value 63, and synthesizes the second pseudo sample 62 to the first pseudo sample 61. The specific processing contents of the conversion and the synthesis can be appropriately determined in accordance with the kind of data, respectively. Thus, the data synthesizing section 214 of the present embodiment generates a new sample 65 of data of a prescribed kind.

[0215] <Speculation device>

[0216] Figure 9 An example of a software configuration of the speculation device 3 of the present embodiment is schematically shown. The control section 31 of the speculation device 3 expands the speculation program 83 stored in the storage section 32 to the RAM. Next, the control section 31 controls each component by interpreting and executing the instructions contained in the speculation program 83 expanded to the RAM by the CPU. Thus, as shown in FIG. 6, the speculation device 3 of the present embodiment includes the data acquisition section 21, the data synthesizing section 22, the data output section 23, and the data storage section 24. Figure 9As illustrated, the estimation device 3 of the present embodiment functions as a computer that includes the acquisition unit 311, the estimation unit 312, the output unit 313, and the setting unit 314 as software modules. That is, in the present embodiment, like the learning device 1 and the like, each software module of the estimation device 3 is implemented by the control unit 11 (CPU).

[0217] The acquisition unit 311 acquires the object sample 325 as an object of performing a prescribed estimation process with respect to the second feature. The estimation unit 312 performs the prescribed estimation process on the acquired object sample 325. In the present embodiment, the estimation unit 312 includes the first generator 411 and the estimator 44 that have completed training by retaining the first / fourth learning result data (121, 124). The estimation unit 312 performs processing of detecting the second component (in other words, determining whether or not the second feature exists) with respect to the object sample 325 using the first generator 411 and the estimator 44 that have completed training.

[0218] Specifically, the estimation unit 312 estimates each estimation value of the input provided to the first generator 411 that has completed training in order to generate a pseudo sample corresponding to the object sample 325 by providing the object sample 325 to the estimator 44 that has completed training. If a pseudo sample corresponding to the object sample 325 is generated by the first generator 411 that has completed training, the obtained estimation values (first estimation value and second estimation value) correspond to the input (noise and input value) that will be provided to the first generator 411 that has completed training. The estimation unit 312 generates the pseudo sample 326 by providing the estimated each estimation value to the first generator 411 that has completed training. Also, the estimation unit 312 generates the difference sample 327 by taking the difference between the object sample 325 and the pseudo sample 326. The estimation unit 312 detects the second component using the difference sample 327.

[0219] As described above, the first generator 411 that has completed training has acquired the ability to generate a pseudo sample including the first feature, but has not acquired the ability to generate a pseudo sample including the second feature. Therefore, even in a case where the object sample 325 exhibits both the first feature and the second feature, basically, in the obtained pseudo sample 326, the first feature is reproduced, and the second feature is not reproduced. Thus, in a case where the object sample 325 includes the second component (that is, exhibits the second feature), the second component is also included in the difference sample 327. On the other hand, in a case where the object sample 325 does not include the second component (that is, does not exhibit the second feature), the second component is also not included in the difference sample 327. Therefore, the estimation unit 312 can detect the second component based on the degree to which the second component is included in the difference sample 327.

[0220] The output section 313 outputs information related to the result of the speculation by the speculation section 312 (in the present embodiment, the detection result of the above-described second component). The setting section 314 sets the value of the parameter in the above-described speculation processing based on the samples 321 included in the data group 320. The samples that exhibit the first characteristic and the second characteristic can be used as the samples 321 of the data group 320. At least a part of the samples 321 included in the data group 320 can be the samples 65 generated by the above-described data generation apparatus 2.

[0221] [Others]

[0222] As for each software module of the learning apparatus 1, the data generation apparatus 2, and the speculation apparatus 3, detailed description will be given in the action example described later. Note that, in the present embodiment, description is given of the example in which each software module of the learning apparatus 1, the data generation apparatus 2, and the speculation apparatus 3 is implemented by a general-purpose CPU. However, part or all of the above-described software modules can also be implemented by one or a plurality of dedicated processors. In addition, as for the software configuration of each of the learning apparatus 1, the data generation apparatus 2, and the speculation apparatus 3, the software module can be omitted, replaced, and added as appropriate in accordance with the embodiment.

[0223] §3 Action Example

[0224] [Learning Apparatus]

[0225] (A) Machine Learning of First Learning Model

[0226] Figure 10A is a flowchart showing an example of the processing flow related to the machine learning of the first learning model 41 by the learning apparatus 1 of the present embodiment. Note that the processing flow described below is merely an example, and each step can be changed as appropriate as long as the change is permitted. Also, as for each processing flow described below, a step can be omitted, replaced, and added as appropriate in accordance with the embodiment.

[0227] (Step S101)

[0228] In step S101, the control section 11 functions as the first acquisition section 111 to acquire a plurality of first learning data sets 51. Each first learning data set 51 is constituted by a combination of a first sample 511 including a first characteristic and a first label 512 indicating the category of the first characteristic.

[0229] The method of generating each first learning data set 51 can not be particularly limited and can be appropriately selected depending on the embodiment. For example, a sample exhibiting the first feature is acquired in a real space or a virtual space. The method of acquiring the sample can be appropriately determined depending on the first feature and the kind of data. As an example, in a case where the prescribed kind of data is sensor data, an object is observed by a sensor in a state where the second feature is not exhibited and the first feature is exhibited, whereby a sample not exhibiting the second feature and exhibiting the first feature can be obtained. In the example of the above-described Figure 2A , a sample in which the background P11 in which the product P1 is included is exhibited can be obtained by capturing the product P1 having no defect by using a camera. In the example of the above-described Figure 2B , a sample including the environmental sound P21 can be acquired by recording the operation sound of a machine having no abnormality by using a microphone. Alternatively, a sample exhibiting the first feature can be acquired by simulating or data processing such an environment on a virtual space. Next, the class of the first feature exhibited in the acquired sample is recognized, and information indicating the recognized result is associated with the sample as a label. The recognition of the class of the first feature can be performed by an operator such as a user. Alternatively, a recognizer is used in the recognition of the class of the first feature. The recognizer can be configured to recognize the class of the first feature by a known information processing such as image analysis, sound analysis, or can be constituted by a machine learning model trained to recognize the class of the first feature from a sample of an object. Next, information indicating the recognized result of the class of the first feature is associated with the sample as a label. Thereby, each first learning data set 51 can be generated.

[0230] Each first learning data set 51 can be automatically generated by the operation of a computer, or can be manually generated by at least partially including the operation of an operator. In addition, the generation of each first learning data set 51 can be performed by the learning device 1, or can be performed by another computer other than the learning device 1. In a case where the learning device 1 generates each first learning data set 51, the control section 11 automatically performs the above-described series of generation processes, or manually performs the above-described series of generation processes by the operation of an operator using the input device 15, whereby the plurality of first learning data sets 51 are obtained. On the other hand, in a case where each first learning data set 51 is generated by another computer, the control section 11 acquires the plurality of first learning data sets 51 generated by the other computer via, for example, a network, a storage medium 91, or the like. Some of the first learning data sets 51 can be generated by the learning device 1, and the other first learning data sets 51 can be generated by one or more other computers.

[0231] The number of pieces of the acquired first learning data set 51 can not be particularly limited and can be appropriately selected depending on the embodiment. After the plurality of first learning data sets 51 are acquired, the control section 11 proceeds to the next step S102 and performs the process.

[0232] (Step S102)

[0233] In step S102, the control section 11 functions as a first training section 115 and performs machine learning of the first learning model 41 including the first generator 411 using the acquired plurality of first learning data sets 51. In the machine learning of the first learning model 41, the control section 11 trains the first generator 411 with respect to each first learning data set 51 to acquire an ability to generate a pseudo sample matching the first sample 511 from the first label 512. As long as the training can be performed as such, the configuration of the first learning model 41 and the method of the machine learning can each be not particularly limited and can be appropriately selected according to the embodiment. In the present embodiment, the first learning model 41 further includes the first discriminator 412, and thus the machine learning of the first learning model 41 can be performed by the following method.

[0234] Figure 10B FIG. 6 is a flowchart illustrating an example of a processing flow of a subprogram related to the machine learning of step S102 of the first learning model 41 of the present embodiment. The processing of step S102 of the present embodiment includes the processing of steps S1021 to S1023 below. However, the processing flow described below is merely an example, and each process can be changed as long as it is permitted. In addition, with respect to the processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment.

[0235] Note that a machine learning model constituting the first generator 411 and the first discriminator 412 that are processing targets of the machine learning can be appropriately prepared. The structure of each of the prepared first generator 411 and the first discriminator 412 (for example, the number of layers, the number of neurons included in each layer, the binding relationship of the neurons of adjacent layers to each other, and the like), the initial value of the weight of the binding between each neuron, and the initial value of the threshold of each neuron can be provided from a template or can be provided by input of an operator. In addition, in a case where relearning is performed, the control section 11 can prepare the first generator 411 and the first discriminator 412, respectively, on the basis of learning result data obtained by performing past machine learning.

[0236] (Step S1021)

[0237] In step S1021, the control section 11 trains the first discriminator 412 using the first learning data set 51 and the pseudo sample 515 generated by the first generator 411 to recognize the origin of an input sample. In the training processing of this machine learning, an approximate gradient descent method, a small batch gradient descent method, or the like can be used. The same applies to the processing of other machine learning such as step S1022 and the like described below.

[0238] As an example, first, the control section 11 supplies the noise 510 extracted from the prescribed probability distribution and the first label 512 to the first generator 411 with respect to each first learning data set 51, and executes the operation processing of the first generator 411. That is, the control section 11 inputs the noise 510 and the first label 512 to the input layer 4111 of the first generator 411, and sequentially performs the firing determination (i.e., performs the forward propagation calculation) on each neuron contained in each layer 4111 to 4113 from the input side. Through this calculation processing, the control section 11 acquires the output value corresponding to the pseudo sample 515 generated from the noise 510 and the first label 512 from the output layer 4113 with respect to each first learning data set 51.

[0239] Next, the control section 11 inputs the generated pseudo sample 515 and the corresponding first label 512 to the input layer 4121 of the first recognizer 412, and sequentially performs the firing determination on each neuron contained in each layer 4121 to 4123 from the input side. Thereby, the control section 11 acquires the output value corresponding to the recognition result of the origin of the input sample from the output layer 4123 of the first recognizer 412. In this context, since the input input sample is the pseudo sample 515, it is correct that the first recognizer 412 is recognized as "fake". The control section 11 calculates the error of the output value obtained from the output layer 4123 with respect to this correct answer with respect to each pseudo sample 515.

[0240] In addition, the control section 11 inputs each first learning data set 51 (the first sample 511 and the first label 512) to the input layer 4121 of the first recognizer 412, and sequentially performs the firing determination on each neuron contained in each layer 4121 to 4123 from the input side. Thereby, the control section 11 acquires the output value corresponding to the recognition result of the origin of the input sample from the output layer 4123 of the first recognizer 412. In this context, since the input input sample is the first sample 511, it is correct that the first recognizer 412 is recognized as "true". The control section 11 calculates the error of the output value obtained from the output layer 4123 with respect to this correct answer with respect to each pseudo sample 515.

[0241] A loss function can be used when calculating each error (loss). The loss function is a function that evaluates the difference (i.e., the degree of difference) between the output of the machine learning model and the correct answer, and the larger the difference value between the output value obtained from the output layer 4123 and the correct answer, the larger the value of the error calculated by the loss function. The kind of loss function used for the calculation of the error can not be particularly limited, and can be appropriately selected according to the embodiment.

[0242] The control section 11 calculates errors in the values of the respective operation parameters (the weighted values of the connections between the respective neurons, the threshold values of the respective neurons, and the like) of the first recognizer 412 using the gradients of the errors in the calculated output values by the error back propagation method. The control section 11 updates the values of the respective operation parameters of the first recognizer 412 on the basis of the calculated errors. The degree of updating the values of the respective operation parameters can be adjusted by a learning rate. The learning rate can be provided by operator designation or as a set value in the program.

[0243] After the control section 11 fixes the values of the respective operation parameters of the first generator 411, it adjusts the values of the respective operation parameters of the first recognizer 412 by the above series of update processes to reduce the sum of the calculated errors. For example, the control section 11 can repeatedly adjust the values of the respective operation parameters of the first recognizer 412 by the above series of processes until a prescribed condition, such as the performance of a prescribed number of times, the sum of the calculated errors being below a threshold value, or the like, is satisfied. Thus, the control section 11 can construct the first recognizer 412 trained to recognize whether an input input sample is a first sample 511 from any one of the plurality of first learning data sets 51 or a pseudo sample 515 generated by the first generator 411. After the training process of this first recognizer 412 is completed, the control section 11 proceeds to the next step S1022 to perform processing.

[0244] (Step S1022)

[0245] In step S1022, the control section 11 trains the first generator 411 to generate a pseudo sample 515 such as to degrade the recognition performance of the first recognizer 412. In other words, the control section 11 trains the first generator 411 to generate a pseudo sample 515 such as to be recognized as "true" (i.e., as coming from the learning data) by the first recognizer 412.

[0246] As an example, first, the control section 11 generates a pseudo sample 515 by providing the first generator 411 with a noise 510 extracted from a prescribed probability distribution and a first label 512 for each first learning data set 51, similarly to step S1021 described above. In the case where the pseudo sample 515 generated in step S1021 described above is also used in this step S1022, the control section 11 can omit the series of processes of generating a pseudo sample 515 in this step S1022.

[0247] Next, the control section 11 inputs the generated pseudo sample 515 and the corresponding first label 512 to the input layer 4121 of the first recognizer 412, and sequentially performs discharge determination of each neuron contained in each layer 4121 to 4123 from the input side. Thereby, the control section 11 acquires an output value corresponding to the recognition result of the origin of the input sample from the output layer 4123 of the first recognizer 412. In this context, the first recognizer 412 erroneously recognizes the input sample as "true" (i.e., from the learning data) as the correct answer. The control section 11 calculates the error of the output value obtained from the output layer 4123 from this correct answer for each pseudo sample 515. As described above, the calculation of this error can use a loss function.

[0248] The control section 11 calculates the gradient of the error of the output value by the error backpropagation method, and reversely propagates it to each operation parameter of the first generator 411 via the first recognizer 412, and calculates the error of the value of each operation parameter of the first generator 411. The control section 11 updates the value of each operation parameter of the first generator 411 based on the calculated error. As described above, the degree of updating the value of each operation parameter can be adjusted by the learning rate.

[0249] After the control section 11 fixes the value of each operation parameter of the first recognizer 412, it adjusts the value of each operation parameter of the first generator 411 by the above series of update processes to reduce the sum of the calculated errors. As with the training of the first recognizer 412 described above, the control section 11 can repeatedly adjust the value of each operation parameter of the first generator 411 by the above series of processes until a prescribed condition is satisfied. Thereby, the control section 11 can construct the first generator 411 trained to generate a pseudo sample 515 like reducing the recognition performance of the first recognizer 412. After the training process of this first generator 411 is completed, the control section 11 proceeds to the next step S1023 for processing.

[0250] (Step S1023)

[0251] In step S1023, the control section 11 determines whether the processing of steps S1021 and S1022 is repeated. The criterion for repeating the processing can be appropriately determined according to the embodiment. For example, a prescribed number of times of repeating the processing can be provided. The prescribed number of times can be provided by, for example, operator designation or as a setting value within the program. In this case, the control section 11 determines whether the number of times of executing the processing of steps S1021 and S1022 has reached the prescribed number of times. In the case where it is determined that the number of times of execution has not reached the prescribed number of times, the control section 11 returns to the processing of step S1021 and executes the processing of steps S1021 and S1022 therefrom. On the other hand, in the case where it is determined that the number of times of execution has reached the prescribed number of times, the control section 11 completes the processing of machine learning of the first learning model 41 and proceeds to the next step S103.

[0252] Thus, the control section 11 alternately repeats the training step of the first discriminator 412 and the training step of the first generator 411. Thus, it is possible to alternately and gradually improve the performance of the first discriminator 412 and the first generator 411. As a result, it is possible to construct the first generator 411 that has completed training with respect to each first learning dataset 51, which has acquired the ability to generate a pseudo sample that matches the first sample 511 from the first label 512 (i.e., a pseudo sample including the first feature of the class corresponding to the input value).

[0253] (Step S103)

[0254] Return Figure 10A In step S103, the control section 11 functions as a saving processing section 119 and generates information related to the first generator 411 that has completed training constructed by machine learning as first learning result data 121. Next, the control section 11 saves the generated first learning result data 121 to a prescribed storage area.

[0255] The prescribed storage area can be, for example, a RAM within the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof. The storage medium can be, for example, a CD, a DVD, or the like, and the control section 11 can store the first learning result data 121 in the storage medium via the drive 17. The external storage device can be, for example, a data server such as a NAS (Network Attached Storage) or the like. In this case, the control section 11 can store the first learning result data 121 in the data server via a network using the communication interface 13. In addition, the external storage device can be an external storage device connected to the learning device 1 via, for example, the external interface 14.

[0256] Thus, after the first learning result data 121 is saved, the control section 11 ends the series of processes relating to the machine learning of the first learning model 41.

[0257] Note that the generated first learning result data 121 can be provided to the data generation device 2 and the estimation device 3 at any point in time. For example, the control section 11 can forward the first learning result data 121 to the data generation device 2 and the estimation device 3 as the process of step S103 or separately from the process of step S103. The data generation device 2 and the estimation device 3 can each acquire the first learning result data 121 by receiving the forwarding. Alternatively, for example, the data generation device 2 and the estimation device 3 can each acquire the first learning result data 121 by accessing the learning device 1 or a data server via a network using a communication interface (23, 33). Alternatively, for example, the data generation device 2 and the estimation device 3 can each acquire the first learning result data 121 via a storage medium (92, 93). Alternatively, for example, the first learning result data 121 can be incorporated in the data generation device 2 and the estimation device 3 in advance, respectively.

[0258] Also, the control section 11 can update or regenerate the first learning result data 121 by repeatedly performing the processes of steps S101 to S103 periodically or aperiodically. When repeated, at least a part of the plurality of first learning data sets 51 is appropriately subjected to change, modification, addition, deletion, or the like. Also, the control section 11 can update the first learning result data 121 held by the data generation device 2 and the estimation device 3, respectively, by providing the updated or regenerated first learning result data 121 to the data generation device 2 and the estimation device 3, respectively, using any method.

[0259] (B) Machine Learning of Second Learning Model

[0260] Figure 11A is a flowchart showing an example of a process flow relating to the machine learning of the second learning model 42 by the learning device 1 of the present embodiment. Note that the process flow described below is merely an example, and each step can be changed as appropriate, as long as the change is permitted. Also, with respect to each process flow described below, steps can be omitted, replaced, and added as appropriate, according to the embodiment.

[0261] (Step S111)

[0262] In step S111, the control section 11 functions as a second acquisition section 112 and acquires a plurality of second learning data sets 52. Each second learning data set 52 is constituted by a combination of a second sample 521 including a second feature different from the first feature and a second label 522 indicating a category of the second feature.

[0263] The generation method and the acquisition method of each second learning dataset 52 can be the same as those of the first learning dataset 51 described above. However, in a case where a feature that is difficult to occur alone is selected as a second feature, as in the example described above, it can be difficult to obtain a sample including only the second feature. In this case, a sample including only the second feature can be generated by including a sample including the first feature and the second feature and appropriately processing the obtained sample. In the present embodiment, a second sample 521 can be generated by using the first generator 411 for which the training is completed and the predictor 44. The details of the information processing will be described later. Figure 2A Figure 2B In a case where a feature that is difficult to occur alone is selected as a second feature, as in the example described above, it can be difficult to obtain a sample including only the second feature. In this case, a sample including only the second feature can be generated by including a sample including the first feature and the second feature and appropriately processing the obtained sample. In the present embodiment, a second sample 521 can be generated by using the first generator 411 for which the training is completed and the predictor 44. The details of the information processing will be described later.

[0264] The number of pieces of the acquired second learning dataset 52 can not be particularly limited and can be appropriately selected according to the embodiment. After the plurality of second learning datasets 52 are acquired, the control unit 11 proceeds to the next step S112 to perform the processing.

[0265] (Step S112)

[0266] In step S112, the control unit 11 functions as a second training unit 116 and performs machine learning of a second learning model 42 including a second generator 421 using the plurality of acquired second learning datasets 52. In the machine learning of the second learning model 42, the control unit 11 trains the second generator 421 for each second learning dataset 52 to acquire the ability to generate a pseudo sample matching the second sample 521 from the second label 522. As long as the training can be performed as such, the configuration of the second learning model 42 and the method of the machine learning can not be particularly limited and can be appropriately selected according to the embodiment. In the present embodiment, the second learning model 42 further includes a second recognizer 422, and thus the machine learning of the second learning model 42 can be performed by the following method.

[0267] Figure 11B is a flowchart showing an example of a processing flow of a subprogram related to the machine learning of step S112 in the second learning model 42 of the present embodiment. The processing of step S112 of the present embodiment includes the processing of steps S1121 to S1123 described below. However, the processing flow described below is only an example, and each process can be changed as long as it is allowed. In addition, with respect to the processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment. Note that, as with the first learning model 41 described above, a machine learning model constituting the second generator 421 and the second recognizer 422 that are the processing targets of the machine learning can be appropriately prepared.

[0268] (Step S1121)

[0269] ​In step S1121, the control unit 11 trains the second discriminator 422 using the pseudo samples 525 generated by the second generator 421 and the second learning data sets 52 so as to recognize the origin of an input sample.

[0270] The processing of step S1121 can be the same as the above-described step S1021 except for the point of replacing the learning data. That is, the control unit 11 generates the pseudo samples 525 by providing the noise 520 extracted from a prescribed probability distribution and the second label 522 to the second generator 421 for each of the second learning data sets 52. The control unit 11 provides the generated pseudo samples 525 and the corresponding second labels 522 to the second discriminator 422, acquires output values corresponding to the recognition results of the origin of an input sample, and calculates errors of the acquired output values from correct answers. In addition, the control unit 11 provides each of the second learning data sets 52 (the second samples 521 and the second labels 522) to the second discriminator 422, acquires output values corresponding to the recognition results of the origin of an input sample, and calculates errors of the acquired output values from correct answers. The control unit 11 calculates errors of the values of each of the operation parameters of the second discriminator 422 using gradients of the calculated errors of the output values by the error backpropagation method. The control unit 11 updates the values of each of the operation parameters of the second discriminator 422 based on the calculated errors. The control unit 11 adjusts the values of each of the operation parameters of the second discriminator 422 by the above-described series of update processing to reduce the sum of the calculated errors after fixing the values of each of the operation parameters of the second generator 421. The control unit 11 can repeatedly adjust the values of each of the operation parameters of the second discriminator 422 by the above-described series of processing until a prescribed condition is satisfied, similarly to the above-described step S1021 and the like.

[0271] Thus, the control unit 11 can construct the second discriminator 422 trained to recognize whether an input input sample is a second sample 521 obtained from any one of the plurality of second learning data sets 52 or a pseudo sample 525 generated by the second generator 421. After the training processing of this second discriminator 422 is completed, the control unit 11 proceeds to the next step S1122.

[0272] (Step S1122)

[0273] In step S1122, the control unit 11 trains the second generator 421 so as to generate pseudo samples 525 such as to reduce the recognition performance of the second discriminator 422. In other words, the control unit 11 trains the second generator 421 so as to generate pseudo samples 525 such as to be recognized as “true” (i.e., recognized as coming from learning data) by the second discriminator 422.

[0274] The process of step S1122 can be the same as the above-described step S1022 except for the point of replacing the learning data. That is, the control unit 11 supplies the generated pseudo sample 525 and the corresponding second label 522 to the second recognizer 422, acquires output values corresponding to the recognition result of the origin of the input sample, and calculates errors of the acquired output values and the correct answer (recognition as coming from the learning data). The control unit 11 calculates errors of the values of the respective operation parameters of the second generator 421 by back-propagating the gradients of the calculated errors of the output values to the respective operation parameters of the second generator 421 via the second recognizer 422 by the error back-propagation method. The control unit 11 updates the values of the respective operation parameters of the second generator 421 on the basis of the calculated errors. After the control unit 11 fixes the values of the respective operation parameters of the second recognizer 422, the control unit 11 adjusts the values of the respective operation parameters of the second generator 421 by the above-described series of update processes to reduce the sum of the calculated errors. The control unit 11 repeatedly adjusts the values of the respective operation parameters of the second generator 421 by the above-described series of processes until a prescribed condition is satisfied, similarly to the above-described step S1021 and the like.

[0275] Thus, the control unit 11 can construct the second generator 421 trained to generate the pseudo sample 525 like reducing the recognition performance of the second recognizer 422. After the training process of this second generator 421 is completed, the control unit 11 proceeds to the next step S1123.

[0276] (Step S1123)

[0277] In step S1123, the control unit 11 determines whether to repeatedly perform the processes of step S1121 and step S1122. The criterion for repeatedly performing the processes can be appropriately determined according to the embodiment, similarly to the above-described step S1023. In the case where it is determined to repeatedly perform the processes, the control unit 11 repeatedly performs step S1121, and executes the processes of step S1121 and step S1122 again. On the other hand, in the case where it is determined not to repeatedly perform the processes, the control unit 11 completes the process of machine learning of the second learning model 42, and proceeds to the next step S113.

[0278] Thus, the control unit 11 alternately and repeatedly executes the training step of the second recognizer 422 and the training step of the second generator 421. Thus, it is possible to gradually and alternately improve the performance of the second recognizer 422 and the second generator 421. As a result, it is possible to configure the second generator 421 trained to completion with respect to each second learning data set 52, which has acquired the ability to generate a pseudo sample (i.e., a pseudo sample including a second feature of a category corresponding to an input value) matching the second sample 521 from the second label 522.

[0279] (Step S113)

[0280] Returning to Figure 11A In step S113, the control section 11 functions as a saving processing section 119 and generates information related to the second generator 421 for which the training is completed by the machine learning as second learning result data 122. Next, the control section 11 saves the generated second learning result data 122 in a prescribed storage area. The prescribed storage area can be, for example, a RAM within the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof.

[0281] Thus, after the second learning result data 122 is saved, the control section 11 ends the series of processes related to the machine learning of the second learning model 42. Note that, as with the first learning result data 121 described above, the generated second learning result data 122 can be provided to the data generation device 2 at an arbitrary point in time. In addition, the processes of steps S111 to S113 described above can be repeatedly performed periodically or aperiodically. Furthermore, by providing the updated or newly generated second learning result data 122 to the data generation device 2 using an arbitrary method, the second learning result data 122 held by the data generation device 2 can be updated.

[0282] (C) Machine Learning of Third Learning Model 43

[0283] Figure 12A is a flowchart illustrating an example of a processing flow related to the machine learning of the third learning model 43 by the learning device 1 of the present embodiment. Note that the processing flow described below is merely an example, and each step can be changed as appropriate. Furthermore, with respect to each processing flow described below, steps can be omitted, replaced, and added as appropriate in accordance with the embodiment.

[0284] (Step S121)

[0285] In step S121, the control section 11 functions as a third acquisition section 113 and acquires a plurality of third learning data sets 53. Each third learning data set 53 is constituted by a combination of a sample value 531 of an attribute of a second feature, a reflection level 532 indicating a degree of reflection of the second feature, and a second label 522. The first label 512 can also be included in each third learning data set 53.

[0286] The generation method and the acquisition method of each third learning data set 53 can be the same as those of the first learning data set 51 described above. The second label 522 can be the same as the second learning data set 52, or can be newly generated separately from the second learning data set 52. The same applies to the first label 512. Note that in a case where a sample including only the second feature is generated from a sample including the first feature and the second feature, information indicating the class of the first feature included in the original sample can be included in the third learning data set 53 as the first label 512 together with the corresponding second label 522.

[0287] The sample value 531 and the reflection level 532 can be appropriately provided. The sample value 531 and the reflection level 532 can be determined in accordance with the sample including the second feature, or can be specified by an operator. The reflection level 532 can be determined in accordance with the sample value 531 of the attribute in accordance with an index that evaluates the attribute of the second feature. The index can be appropriately provided. As a specific example, in the example described above in Figure 2A , in a case where the sample value 531 of the attribute is provided in accordance with the size of the defect, the index can be configured to determine the reflection level 532 in accordance with the size of the defect. Alternatively, the reflection level 532 can be set in association with an event that exhibits the second feature. As a specific example, in the example described above in Figure 2A , the reflection level 532 can be set in association with the degree of the defect, such as the severity of the defect, the quality level of the product, and the like. Further, in the example described above in Figure 2B , the reflection level 532 can be set in association with the degree of the abnormality, such as the severity of the abnormality, the normality of the machine, and the like. Further, the reflection level 532 can be constituted by any one of a discrete value and a continuous value. However, from the viewpoint of richly expressing the degree of reflecting the second feature, the reflection level 532 is preferably constituted by a continuous value.

[0288] The number of pieces of the acquired third learning data set 53 can not be particularly limited, and can be appropriately selected in accordance with the embodiment. After a plurality of third learning data sets 53 are acquired, the control section 11 proceeds to the next step S122 to perform processing.

[0289] (Step S122)

[0290] In step S122, the control section 11 functions as a third training section 117 and performs machine learning of the third learning model 43 including the third generator 431 using the plurality of third learning data sets 53 acquired. In the machine learning of the third learning model 43, the control section 11 trains the third generator 431 for each third learning data set 53 to acquire the ability to generate the pseudo value of the attribute of the second feature matching the sample value 531 from the reflection level 532 and the second label 522. As long as the training can be performed as such, the configuration of the third learning model 43 and the method of the machine learning can each be not particularly limited and can be appropriately selected according to the embodiment. In the embodiment, the third learning model 43 can further include the third discriminator 432, and thus the machine learning of the third learning model 43 can be performed by the following method.

[0291] Figure 12B FIG. 18 is a flowchart illustrating an example of a processing flow of a subprogram related to the machine learning of step S122 in the third learning model 43 of the embodiment. The processing of step S122 of the embodiment includes the processing of steps S1221 to S1223 below. However, the processing flow described below is merely an example, and each process can be changed as long as it is allowed. In addition, regarding the processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment. Note that, similarly to the first learning model 41 and the like described above, a machine learning model configuring the third generator 431 and the third discriminator 432 that are the processing targets of the machine learning can be appropriately prepared.

[0292] (Step S1221)

[0293] In step S1221, the control section 11 trains the third discriminator 432 using the third learning data set 53 and the pseudo value 535 generated by the third generator 431 to recognize the origin of the input value.

[0294] The processing of step S1221 can be the same as the above-described step S1021 and so on except for the point of replacing the learning data. That is, the control section 11 provides the noise 530 extracted from the prescribed probability distribution, the reflection level 532, and the second label 522 to the third generator 431 for each third learning data set 53, thereby generating the pseudo value 535 of the attribute of the second feature. The control section 11 provides the generated pseudo value 535, the corresponding reflection level 532, and the corresponding second label 522 to the third discriminator 432, acquires the output value corresponding to the recognition result of the origin of the input value, and calculates the error of the acquired output value from the correct answer. In addition, the control section 11 provides each third learning data set 53 (the sample value 531, the reflection level 532, and the second label 522) to the third discriminator 432, acquires the output value corresponding to the recognition result of the origin of the input value, and calculates the error of the acquired output value from the correct answer. The control section 11 calculates the error of the value of each operation parameter of the third discriminator 432 using the gradient of the calculated error of the output value by the error back propagation method. The control section 11 updates the value of each operation parameter of the third discriminator 432 based on the calculated error. After fixing the value of each operation parameter of the third generator 431, the control section 11 adjusts the value of each operation parameter of the third discriminator 432 by the above-described series of update processing to reduce the sum of the calculated errors. The control section 11 can repeatedly adjust the value of each operation parameter of the third discriminator 432 by the above-described series of processing until a prescribed condition is satisfied, similarly to the above-described step S1021 and so on.

[0295] Thus, the control section 11 can construct the third discriminator 432 trained to recognize whether the input input value is the sample value 531 obtained from any one of the plurality of third learning data sets 53 or the pseudo value 535 generated by the third generator 431. After the training processing of this third discriminator 432 is completed, the control section 11 proceeds to the next step S1222.

[0296] (Step S1222)

[0297] In step S1222, the control section 11 trains the third generator 431 to generate the pseudo value 535 such as to reduce the recognition performance of the third discriminator 432. In other words, the control section 11 trains the third generator 431 to generate the pseudo value 535 such as to be recognized as "true" (i.e., recognized as coming from the learning data) by the third discriminator 432.

[0298] The processing of step S1222 can be the same as the above-described step S1022 and the like except for the point of replacing the learning data. That is, the control unit 11 supplies the generated pseudo value 535, the corresponding reflection level 532, and the corresponding second label 522 to the third recognizer 432, acquires an output value corresponding to the recognition result of the origin of the input value, and calculates the error of the acquired output value from the correct answer (recognized as coming from the learning data). The control unit 11 backpropagates the gradient of the calculated error of the output value to each operation parameter of the third generator 431 via the third recognizer 432 by the error backpropagation method, and calculates the error of the value of each operation parameter of the third generator 431. The control unit 11 updates the value of each operation parameter of the third generator 431 on the basis of the calculated error. After fixing the value of each operation parameter of the third recognizer 432, the control unit 11 adjusts the value of each operation parameter of the third generator 431 by the above-described series of update processing to reduce the sum of the calculated errors. The control unit 11 can repeatedly adjust the value of each operation parameter of the third generator 431 by the above-described series of processing until a prescribed condition is satisfied, similarly to the above-described step S1021 and the like.

[0299] Thus, the control unit 11 can construct the third generator 431 trained to generate a pseudo value 535 such as to reduce the recognition performance of the third recognizer 432. After the training processing of this third generator 431 is completed, the control unit 11 proceeds to the next step S1223.

[0300] (Step S1223)

[0301] In step S1223, the control unit 11 determines whether to repeatedly perform the processing of step S1221 and step S1222. The criterion for repeatedly performing the processing can be appropriately determined according to the embodiment, similarly to the above-described step S1023 and the like. In the case of determining to repeatedly perform the processing, the control unit 11 returns to step S1221 and performs the processing of step S1221 and step S1222 again. On the other hand, in the case of determining not to repeatedly perform the processing, the control unit 11 completes the processing of machine learning of the third learning model 43 and proceeds to the next step S123.

[0302] Thus, the control unit 11 alternately and repeatedly performs the training step of the third recognizer 432 and the training step of the third generator 431. Thus, it is possible to alternately and gradually improve the performance of the third recognizer 432 and the third generator 431. As a result, it is possible to construct the third generator 431 trained to completion for each third learning data set 53, which has acquired the ability to generate a pseudo value of the attribute of the second feature matching the sample value 531 from the reflection level 532 and the second label 522 (i.e., a pseudo value of the attribute of the second feature of the class and the degree corresponding to the input value).

[0303] Note that, in a case where the first label 512 is further provided to each of the third learning data sets 53, the first label 512 can be further added to each of the inputs to the third generator 431 and the third discriminator 432 in each of the above training steps. Thereby, the control unit 11 can construct the third generator 431 that has completed training, for each of the third learning data sets 53, the third generator 431 having the ability to generate the pseudo value of the attribute of the second feature that matches the sample value 531, from the reflection level 532, the first label 512, and the second label 522.

[0304] (Step S123)

[0305] Return Figure 12A In step S123, the control unit 11 functions as the saving processing unit 119, generates information related to the third generator 431 that has completed training constructed by machine learning, as third learning result data 123. Further, the control unit 11 saves the generated third learning result data 123 in a predetermined storage area. The predetermined storage area can be, for example, a RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination of these.

[0306] Thereby, after the third learning result data 123 is saved, the control unit 11 ends the series of processes related to machine learning of the third learning model 43. Note that, the generated third learning result data 123 can be provided to the data generation device 2 at an arbitrary point in time, similarly to the first learning result data 121 and the like. Further, the processes of steps S121 to S123 can be repeatedly performed periodically or aperiodically. Further, by providing the updated or newly generated third learning result data 123 to the data generation device 2 by an arbitrary method, the third learning result data 123 held by the data generation device 2 can be updated.

[0307] (D) Machine Learning of Estimator

[0308] Figure 13 is a flowchart showing an example of a processing flow related to machine learning of the estimator 44 by the learning device 1 of the present embodiment. Note that, the processing flow described below is merely an example, and each step can be changed as long as it is permitted. Further, with respect to each processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment.

[0309] (Step S131)

[0310] In step S131, the control section 11 functions as a fourth acquisition section 114 and acquires a plurality of fourth learning data sets 54. Each fourth learning data set 54 is constituted by a combination of a sample that is input to the first generator 411 that has completed training and a pseudo sample 542 that is generated by providing the sample to the first generator 411 that has completed training. In the present embodiment, the input sample is constituted by a noise 540 and a sample value 541.

[0311] Each fourth learning data set 54 can be appropriately generated by using the first generator 411 that has completed training. In addition, similarly to the first learning data set 51 described above, the generation of each fourth learning data set 54 can be performed by the learning device 1 or can be performed by another computer other than the learning device 1. It can also be that some of the fourth learning data sets 54 are generated by the learning device 1 and the other fourth learning data sets 54 are generated by one or more other computers.

[0312] The number of acquired fourth learning data sets 54 can not be particularly limited and can be appropriately selected according to the embodiment. After a plurality of fourth learning data sets 54 are acquired, the control section 11 proceeds to the next step S132.

[0313] (Step S132)

[0314] In step S132, the control section 11 functions as a fourth training section 118 and performs machine learning of the predictor 44 using the plurality of fourth learning data sets 54 that are acquired. In this machine learning, the control section 11 trains the predictor 44 with respect to each fourth learning data set 54 so that each estimated value obtained by the predictor 44 estimating the input to the first generator 411 that has completed training from the pseudo sample 542 matches the corresponding noise 540 and sample value 541, respectively.

[0315] As an example, first, the control section 11 inputs the pseudo sample 542 to the input layer 441 of the predictor 44 with respect to each fourth learning data set 54 and sequentially performs discharge determination of each neuron included in each layer 441 to 443 from the input side. Thereby, the control section 11 acquires, from the output layer 443, output values corresponding to the first estimated value and the second estimated value of the input to the first generator 411 that has completed training when the pseudo sample 542 is generated, respectively. The control section 11 calculates errors of each estimated value and the corresponding noise 540 and sample value 541, respectively. Similarly to step S1021 described above, a loss function can be used when the errors are calculated.

[0316] The control section 11 calculates the error of the value of each operation parameter of the predictor 44 using the gradient of the calculated error by the error backpropagation method. The control section 11 updates the value of each operation parameter of the predictor 44 based on the calculated error. The degree of updating the value of each operation parameter can be adjusted by the learning rate, similarly to step S1021 and the like.

[0317] The control section 11 adjusts the value of each operation parameter of the predictor 44 by the above series of update processing to reduce the sum of the calculated errors. The control section 11 can repeatedly adjust the value of the operation parameter of the predictor 44 by the above series of processing until a prescribed condition is satisfied, similarly to step S1021 and the like.

[0318] Thus, the control section 11 can configure the predictor 44 that has completed training with respect to each fourth learning data set 54, and the predictor 44 has acquired the ability to estimate the input (the noise 540 and the sample value 541) provided to the first generator 411 that has completed training when the pseudo sample 542 generated by the first generator 411 that has completed training is generated. After the machine learning of the predictor 44 is completed, the control section 11 proceeds to the next step S133 to perform processing.

[0319] (Step S133)

[0320] In step S133, the control section 11 functions as a saving processing section 119 and generates information related to the predictor 44 that has completed training constructed by machine learning as fourth learning result data 124. Next, the control section 11 saves the generated fourth learning result data 124 in a prescribed storage area. The prescribed storage area can be, for example, a RAM in the control section 11, the storage section 12, an external storage device, a storage medium, or a combination thereof.

[0321] Thus, after the fourth learning result data 124 is saved, the control section 11 ends the series of processing related to the machine learning of the predictor 44. Note that, similarly to the first learning result data 121, the generated fourth learning result data 124 can be provided to the estimation device 3 at an arbitrary point in time. In addition, the processing of steps S131 to S133 can be repeatedly performed periodically or aperiodically. Furthermore, the fourth learning result data 124 that is updated or newly generated can be provided to the estimation device 3 by any method to update the fourth learning result data 124 held by the estimation device 3.

[0322] (E) Generation of Second Sample

[0323] Figure 14is a flowchart showing an example of a processing flow related to generation of the second sample 521. Note that the processing flow described below is merely an example, and each step can be changed as appropriate. Also, for each processing flow described below, steps can be omitted, replaced, and added as appropriate, depending on the embodiment. The processing of steps S141 to S144 below can be performed within the step S111 described above.

[0324] (Step S141)

[0325] In step S141, the control unit 11 acquires a plurality of third samples 551 each including the first feature and the second feature.

[0326] The method of generating and acquiring each third sample 551 can be the same as the method of generating and acquiring the first sample 511 of the first learning data set 51 described above. As a specific example, in the example described above in which the product P1 having a defect is imaged by the camera, the third sample 551 can be generated. Also, in the example described above in which the operation sound of the machine in which an anomaly exists is recorded by the microphone, the third sample 551 can be generated. Figure 2A Figure 2B

[0327] The number of acquired third samples 551 can not be particularly limited, and can be selected as appropriate depending on the embodiment. After a plurality of third samples 551 are acquired, the control unit 11 proceeds to the next step S142 and performs processing. Note that the control unit 11 can perform the processing of step S142 below each time a third sample 551 is acquired, or can perform the processing of step S142 below for each third sample 551 after a plurality of third samples 551 are acquired.

[0328] (Step S142)

[0329] In step S142, the control unit 11 estimates each estimated value of the input to be provided to the trained first generator 411 in order to generate a pseudo sample corresponding to each third sample 551 by providing each third sample 551 to the trained predictor 44. That is, the control unit 11 inputs each third sample 551 to the input layer 441 of the trained predictor 44, and sequentially performs discharge determination of each neuron included in each layer 441 to 443 from the input side. Thus, if a pseudo sample corresponding to each third sample 551 is generated by the trained first generator 411, the control unit 11 acquires an output value corresponding to each estimated value of the input to be provided to the trained first generator 411 from the output layer 443. After each estimated value is acquired, the control unit 11 proceeds to the next step S143 and performs processing.

[0330] (Step S143)

[0331] ​​In step S143, the control section 11 generates the pseudo sample 555 for each third sample 551 by providing the obtained each estimated value to the first generator 411 that has completed the training. That is, the control section 11 inputs each estimated value to the input layer 4111 of the first generator 411 that has completed the training, and sequentially performs discharge determination of each neuron contained in each layer 4111 to 4113 from the input side. Thereby, the control section 11 acquires the output value corresponding to the pseudo sample 555 generated from each estimated value from the output layer 4113. After the pseudo sample 555 is acquired, the control section 11 proceeds to the next step S144.

[0332] (Step S144)

[0333] In step S144, the control section 11 calculates the difference between the obtained pseudo sample 555 and the corresponding third sample 551. As described above, by this difference processing, a sample that does not include the first feature and includes the second feature, which can be used as the second sample 521, can be appropriately generated. Thereby, after the generation of the second sample 521 is completed, the control section 11 ends the series of processes related to the generation of the second sample 521. According to this series of processes, the third sample 551 to the second sample 521 can be automatically generated, and thus the second sample 521 can be obtained at low cost.

[0334] Note that the processing flow related to the machine learning of the first to third learning models 41 to 43 and the processing flow related to the generation of the second sample 521 are examples of the "learning method" of the present application. The learning method does not necessarily include all the processes described above, and can be configured by appropriately combining each process of the processing flow described above. The learning method can also be referred to as a model generation method for generating each machine learning model that has completed the training, or simply as a generation method.

[0335] [Data generation device]

[0336] Figure 15 is a flowchart showing an example of a processing flow of the data generation device 2 of the present embodiment. The processing flow described below is an example of the "data generation method" of the present application. However, each processing flow described below is only an example, and each step can be changed as long as it is permitted. Also, with respect to each processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment.

[0337] (Step S201)

[0338] In step S201, the control section 21 functions as the accepting section 211 and accepts designation of the first input value 221, the second input value 223, and the third input value 225. The first input value 221, the second input value 223, and the third input value 225 in the generation section of the sample correspond to the first label 512, the second label 522, and the reflection level 532 in the learning section. That is, the first input value 221, the second input value 223, and the third input value 225 are designated in accordance with the class of the first feature, the class of the second feature, and the degree of reflection of the second feature to be included in the sample to be generated.

[0339] Figure 16 An example of the accepting screen for accepting designation of each input value (221, 223, 225) in the context of the example of the new sample 65 of the generated image data is shown in Figure 2A The accepting screen is an example of a user interface for accepting input of each input value (221, 223, 225). In a case where a display device or the like is included in the output device 26, in step S201, the control section 21 can output the accepting screen to the output device 26. Thereby, the control section 21 can accept designation of each input value (221, 223, 225) by the operator via the input device 25.

[0340] Figure 16 The example of the accepting screen includes a first input field 261, a second input field 262, a slide bar 263, a display area 264, a generation button 265, and a save button 266. The first input field 261 is used to designate the first input value 221. In the first input field 261, the class of the first feature is designated, and accordingly, the control section 21 can acquire the first input value 221. Similarly, the second input field 262 is used to designate the second input value 223. In the second input field 262, the class of the second feature is designated, and accordingly, the control section 21 can acquire the second input value 223.

[0341] The slide bar 263 is used to designate the third input value 225. The third input value 225, which designates the degree of reflection of the second feature, can be frequently changed compared to the first input value 221 and the second input value 223, which designate the classes of the first feature and the second feature. Therefore, by using a user interface in the form of a slide bar in designation of the third input value 225, it is possible to reduce the work of designating input values when generating the new sample 65. Note that, although the slide bar 263 is configured so that the degree of reflection of the second feature can be set in the range of 0 to 10 in the example of Figure 16 The range of values of the degree of reflection of the second feature can not be limited to such an example and can be arbitrarily set.

[0342] The new sample 65 generated by the process described later is displayed using the specified respective input values (221, 223, 225) in the display area 264. The form of the display area 264 can be appropriately determined in accordance with the kind of data of the generated sample 65. The generation button 265 is used to start the process for generating the new sample 65 using the respective input values (221, 223, 225) specified by the respective input fields (261, 262) and the slide bar 263. The control section 21 can enter the next step S202 to perform the process in accordance with the operation of the generation button 265. The save button 266 is used to save the generated sample 65.

[0343] Note that the user interface for accepting the input of the respective input values (221, 223, 225) can not be limited to such an example, and can be appropriately determined in accordance with the embodiment. For example, in the case where the prescribed kind of data is sound data, a reproduction button for reproducing the generated new sample 65 can be further provided. Further, as another example, the respective input values (221, 223, 225) can be specified by simple numerical input. After the specification of the respective input values (221, 223, 225) is accepted, the control section 21 enters the next step S202 to perform the process.

[0344] (Step S202)

[0345] In step S202, the control section 21 operates as the first generation section 212, and sets the first generator 411 completed with the training with reference to the first learning result data 121. The control section 21 extracts the noise 220 from the prescribed probability distribution. Next, the control section 21 inputs the extracted noise 220 and the acquired first input value 221 to the input layer 4111 of the first generator 411 completed with the training, and performs the discharge determination of the respective neurons contained in the respective layers 4111 to 4113 in this order from the input side. Thereby, the control section 21 acquires the output value corresponding to the first pseudo sample 61 generated from the noise 220 and the first input value 221 from the output layer 4113, the first pseudo sample 61 including the first feature of the class specified by the first input value 221. After the first pseudo sample 61 is acquired, the control section 21 enters the next step S203 to perform the process.

[0346] (Step S203)

[0347] In step S203, the control section 21 functions as a second generation section 213, and sets the second generator 421 completed with the training with reference to the second learning result data 122. The control section 21 extracts the noise 222 from a prescribed probability distribution. Next, the control section 21 inputs the extracted noise 222 and the acquired second input value 223 to the input layer 4211 of the second generator 421 completed with the training, and sequentially performs discharge determination of each neuron included in each layer 4211 to 4213 from the input side. Thereby, the control section 21 acquires the output value corresponding to the second pseudo sample 62 generated from the noise 222 and the second input value 223, which includes the second feature of the class specified by the second input value 223, from the output layer 4213. After acquiring the second pseudo sample 62, the control section 21 proceeds to the next step S204.

[0348] (Step S204)

[0349] In step S204, the control section 21 functions as a data synthesis section 214, and sets the third generator 431 completed with the training with reference to the third learning result data 123. The control section 21 extracts the noise 224 from a prescribed probability distribution. Next, the control section 21 inputs the extracted noise 224, the acquired second input value 223, and the acquired third input value 225 to the input layer 4311 of the third generator 431 completed with the training, and sequentially performs discharge determination of each neuron included in each layer 4311 to 4313 from the input side. Thereby, the control section 21 acquires the output value corresponding to the pseudo value 63 generated from the noise 224, the second input value 223, and the third input value 225, which is the pseudo value of the attribute of the second feature of the class and the degree specified by the second input value 223 and the third input value 225, from the output layer 4313.

[0350] In a case where the machine learning of the third generator 431 is further performed using the first label 512, the control section 21 further inputs the acquired first input value 221 to the input layer 4311 of the third generator 431 completed with the training, and performs the above-described operation processing of the third generator 431 completed with the training. Thereby, the control section 21 can acquire the pseudo value 63 for reflecting the second feature in the first feature specified by the first input value 221 in the class and the degree specified by the second input value 223 and the third input value 225.

[0351] After the pseudo value 63 is acquired, the control section 21 proceeds to the next step S205 to perform processing. Note that the processing order of steps S202 to S204 can not be limited to this example, and can be arbitrary. Each noise (220, 222, 224) can be acquired by or separately. In addition, in a case where step S204 is performed after step S202, the learning device 1 can perform machine learning of the above-described estimator 44 between step S202 and step S204. In this case, the first pseudo sample 61 generated in step S202 can also be used as the pseudo sample 542. Alternatively, by including step S201 in the above-described step S131, the generated pseudo sample 542 can also be used as the first pseudo sample 61 in step S202.

[0352] (Step S205)

[0353] In step S205, the control section 21 functions as the data synthesizing section 214, converts the second pseudo sample 62 so that the value of the attribute of the second feature contained in the second pseudo sample 62 matches the generated pseudo value 63, and synthesizes the second pseudo sample 62 with the first pseudo sample 61. The specific processing contents of the conversion and the synthesis can be appropriately determined according to the kind of data, respectively. For example, in a case where the kind of data prescribed is image data, the conversion can be constituted by scaling, parallel movement, rotation, or a combination thereof. In addition, for example, in a case where the kind of data prescribed is signal data, the conversion can be constituted by increase or decrease of amplitude, change of frequency, increase or decrease of occurrence frequency, or a combination thereof. The synthesis can be constituted by, for example, overlap addition.

[0354] Further, a machine learning model that has completed training, which has acquired the ability to synthesize two samples so that the result matches the input attribute value, can be used in the synthesis. The machine learning model that has completed training can be constructed by performing machine learning using a plurality of learning data sets each constituted by a combination of training data (input data) and correct answer data (teacher signal). The training data is constituted by, for example, a first training sample including a first feature, a second training sample including a second feature, and a sample value of an attribute of the second feature corresponding to the degree of reflection. The correct answer data is constituted by, for example, a sample of the corresponding synthesis result.

[0355] The control section 21 generates a new sample 65 of the prescribed kind of data by the synthesis. After the generation of the new sample 65 is completed, the control section 21 ends the processing flow in the present action example. After the generation processing is completed, the control section 21 can display the generated new sample 65 on the display area 264 of the above-described reception screen.

[0356] The control section 21 can execute the above series of processes to generate a new sample 65 according to the change of the third input value 225 by operating the slide bar 263, and display the generated new sample 65 in the display area 264. Thereby, the operator expresses the variation of the degree of the second feature in the generated sample 65, which corresponds to the adjustment of the degree of the second feature by operating the slide bar 263.

[0357] In addition, the control section 21 can save the generated new sample 65 in a prescribed storage area according to the operation of the save button 266. The prescribed storage area can be, for example, a RAM in the control section 21, the storage section 22, an external storage device, a storage medium, or a combination thereof. The generated new sample 65 can be provided to the estimation device 3 at an arbitrary point in time.

[0358] Also, the control section 21 can extract a plurality of each noise (220, 222, 224) for each input value (221, 223, 225). Thereby, the control section 21 can generate a plurality of new samples 65 using the same each input value (221, 223, 225). The generated plurality of new samples 65 can be displayed in the display area 264, and the selection of the sample 65 to be saved therefrom can be accepted. In this case, the control section 21 can save one or more samples 65 selected from the generated plurality of samples 65 in the prescribed storage area according to the operation of the save button 266.

[0359] [Estimation Device]

[0360] Figure 17A is a flowchart showing an example of a processing flow of the estimation device 3 of the present embodiment. The processing flow described below is an example of an estimation method. However, each processing flow described below is merely an example, and each step can be changed as long as it is permitted. Also, with respect to each processing flow described below, steps can be appropriately omitted, replaced, and added according to the embodiment.

[0361] (Step S301)

[0362] In step S301, the control section 31 functions as an acquisition section 311, and acquires an object sample 325. The method of acquiring the object sample 325 can be appropriately determined according to the kind of data. In a case where the object sample 325 is sensing data, the object sample 325 can be acquired by observing an object of estimation related to the second feature with a sensor. In the example described above Figure 2A , the object sample 325 can be acquired by capturing a product to be inspected with a camera. In the example described above Figure 2BIn the example of the machine, the object sample 325 can be acquired by recording the operation sound of the machine to be inspected using a microphone. The control section 31 can acquire the object sample 325 directly or indirectly from a sensor. After acquiring the object sample 325, the control section 31 proceeds to the next step S302.

[0363] (Step S302)

[0364] In step S302, the control section 31 functions as the estimation section 312 and analyzes the object sample 325 using the data set 320, thereby performing estimation processing related to the second feature on the object sample 325. In the present embodiment, as the estimation processing, the control section 31 performs processing of detecting the second component on the object sample 325 using the first generator 411 and the estimator 44 that have completed training.

[0365] Specifically, the control section 31 sets the first generator 411 and the estimator 44 that have completed training with reference to the first / fourth learning result data (121, 124). The control section 31 inputs the object sample 325 to the input layer 441 of the estimator 44 that has completed training, and sequentially performs discharge determination of each neuron included in each layer 441 to 443 from the input side. Thereby, if a pseudo sample corresponding to the object sample 325 is generated by the first generator 411 that has completed training, the control section 31 acquires an output value corresponding to each estimation value to be input to the input of the first generator 411 that has completed training from the output layer 443. Next, the control section 31 inputs the acquired each estimation value to the input layer 4111 of the first generator 411 that has completed training, and sequentially performs discharge determination of each neuron included in each layer 4111 to 4113 from the input side. Thereby, the control section 31 acquires an output value corresponding to the pseudo sample 326 generated from each estimation value from the output layer 4113.

[0366] Further, the control section 31 generates a difference sample 327 by taking a difference between the object sample 325 and the pseudo sample 326. Note that, before performing the difference processing, a pre-processing can be applied to the object sample 325 and the pseudo sample 326. For example, in a case where the object sample 325 and the pseudo sample 326 are time series data and the second component is detected in a frequency region, a pre-processing of converting the time series data to frequency data such as Fourier transform can be applied to the object sample 325 and the pseudo sample 326. The control section 31 detects the second component using the generated difference sample 327.

[0367] Here, further using Figure 17B an example of processing of detecting the second component using the difference sample 327 will be described. Figure 17B The schematic example is described above Figure 2AAn example of a process of detecting the second component (i.e., the defect) in the above-described example situation. The control unit 31 generates the pseudo sample I11 corresponding to the object sample I10 using the first generator 411 and the inferencer 44 trained as described above. The object sample I10 is an example of the above-described object sample 325, and the pseudo sample I11 is an example of the above-described pseudo sample 326.

[0368] The control unit 31 generates the difference sample I12 by calculating the difference between the object sample I10 and the pseudo sample I11. The difference sample I12 is an example of the above-described difference sample 327. The more the difference between the object sample I10 and the pseudo sample I11, the larger the value in the difference sample I12. On the other hand, the less the difference between the object sample I10 and the pseudo sample I11, the smaller the value in the difference sample I12. The method of expressing the difference can be appropriately selected according to the type of data. As an example, Figure 17B In this example, the type of data is image data, and the more the difference in the pixel, the more white is formed, and the less the difference in the pixel, the more black is formed.

[0369] As described above, basically, the pseudo sample I11 reproduces the first feature but does not reproduce the second feature. Therefore, in the difference sample I12, the portion including the second component (exhibiting the second feature) in the object sample I10 causes a large difference. However, the reason for the difference in the difference sample I12 is not limited to the inclusion of the second component in the object sample I10. As other reasons, for example, noise and the like can be cited that occur in the process of generating the pseudo sample I11. However, the degree of the difference caused by the noise is lower than that caused by the inclusion of the second component. Therefore, these differences can be distinguished based on the value of the difference sample I12. Thus, the control unit 31 removes the difference caused by the noise using the threshold value T1. In Figure 17B In this example, the control unit 31 binarizes each pixel of the difference sample I12 using the threshold value T1. Thus, the control unit 31 can obtain the sample I13 from which the noise is removed.

[0370] In the sample I13, mainly, the difference caused by the second component and the difference further caused by other reasons (e.g., insufficient learning of at least any one of the first generator 411 and the inferencer 44) can occur. Among them, the difference caused by the second component occurs in a state corresponding to the attribute of the second feature, such as size, (temporal or regional) position, brightness, amplitude, frequency (temporal or regional), and the like. In the above-described example situation, Figure 2AIn the example of the defect, the defect can have shape-related attributes such as an area, a width, a height, a perimeter, an aspect ratio, a circularity, and the like. That is, in a case where the defect exists in the product presented in the object sample I10, a region in which white pixels are aggregated appears on the position corresponding to the sample I13, and the region has shape-related attributes equivalent to the shape. Thus, by setting the threshold value T2 for the attribute of the second feature, the second component can be detected.

[0371] As an example of the processing, Figure 17B In the example of the defect, the control section 31 regards regions of continuous white pixels within the sample I13 as one region, and determines whether each region of white pixels satisfies the threshold value T2. Next, the control section 31 retains the regions that satisfy the threshold value T2, and deletes the regions that do not satisfy the threshold value T2. Thus, the control section 31 can obtain a detection sample I14 in which the regions that do not satisfy the value of the attribute of the second feature are removed from the sample I13.

[0372] The control section 31 detects the second component based on the obtained detection sample I14. In Figure 17B In the example of the defect, in a case where a region of white pixels exists in the detection sample I14, the control section 31 determines that the defect exists in the product presented in the object sample I10. On the other hand, in a case where a region of white pixels does not exist in the detection sample I14, the control section 31 determines that the defect does not exist in the product presented in the object sample I10. Thus, after the detection processing of the second component is completed, the control section 31 proceeds to the next step S303 to perform processing.

[0373] Note that the above-described each threshold value (T1, T2) is an example of a parameter in the estimation processing. The parameter in the estimation processing can be appropriately set according to the content of the estimation, the kind of data, and the like. In a case where the kind of data is image data, and the estimation processing performs processing of detecting an object presented in the image, the above-described each threshold value (T1, T2) can be used as an example of a parameter in the estimation processing, not limited to the above-described context of the defect detection. As another example, in a case where the kind of data is signal data, and the estimation processing performs processing of detecting an object signal included in the signal, a difference in spectral intensity can be used as a parameter in the estimation processing.

[0374] The control section 31, which functions as a setting section 314 at an arbitrary timing (for example, before starting the processing of step S301), can set the value of the parameter based on the sample 321 included in the data group 320. That is, in the present embodiment, the analysis of the object sample 325 by the data group 320 corresponds to setting the value of the parameter based on the sample 321 included in the data group 320 so that the second component can be detected. In the above-described Figure 17B In the example of the defect, the control section 31 can appropriately set each threshold value (T1, T2) so that the defect presented in the sample 321 can be detected.

[0375] Here, further using Figure 18A and Figure 18B , an example of a method of setting parameters of the presumption process is described. Figure 18A and Figure 18B respectively schematically illustrate an example of an output screen for displaying the samples 321 included in the data group 320 in the context illustrated in the above Figure 2A . In a case where the output device 36 includes a display or the like display device, the control section 31 can output each output screen to the output device 36 according to an operation of the operator.

[0376] In the output screen of Figure 18A , the samples 321 included in the data group 320 are displayed in tabular form according to the categories of the respective features. The operator can confirm, through this output screen, the combination of the categories of the first feature and the second feature for which the sample 321 is not obtained and the sample 321 that is not obtained (the combination of the categories of the defect and the product in the example of the above Figure 2A .

[0377] In the above data generation device 2, by specifying the first input value 221 and the second input value 223 corresponding to the categories of the first feature and the second feature for which the sample 321 is not obtained, a new sample 65 exhibiting the combination can be generated. Alternatively, in the output screen or the like of Figure 18A , the control section 31 accepts the specification of the combination of the categories of the first feature and the second feature for which the sample 321 is not obtained. Also, the control section 31 can delegate the generation of a new sample 65 exhibiting the combination by notifying the above data generation device 2 of the specified combination. Thereby, a new sample 65 exhibiting the combination can be generated. The generated new sample 65 can be provided to the presumption device 3 by an arbitrary method, and added to the data group 320. By adding this new sample 65, it is possible to seek to diversify the samples 321 included in the data group 320.

[0378] On the other hand, in the output screen of Figure 18B , each sample 321 including the categories of the first feature and the second feature of the object is displayed in parallel according to the attribute value of the second feature. If the values of the parameters are set in a manner that the second component can be detected to a smaller degree, it is possible to erroneously detect the second component for the object sample 325 that does not truly include the second component. As a specific example, in the example of the above Figure 17B , the larger the size of the defect, the easier it is to detect the defect, and the smaller the size of the defect, the more difficult it is to detect the defect. If each threshold value (T1, T2) is set in a manner that a defect of a smaller size can be detected, it is possible to mistake a difference that occurs due to other factors as a defect. Therefore, as one of the reasons for reducing the probability of generating this false detection, in this output screen, the control section 31 can accept the specification of the sample 321 corresponding to the limit case of the detection of the second component.

[0379] Furthermore, the control unit 31 can set the parameter value in a manner capable of detecting the second component contained in the sample 321 designated as a limit case. In the above... Figure 17B In the example, the control unit 31 can set each threshold (T1, T2) in a manner that enables the detection of defects presented on the sample 321 designated as a limit case. The method for setting the parameter values ​​is not particularly limited and can be appropriately determined according to the implementation method. For example, multiple candidates can be appropriately prepared for the parameter values. The control unit 31 can determine the candidate with the highest detection rate for the second component contained in the sample 321 designated as a limit case by performing the above detection process using each candidate, and select the determined candidate as the parameter value.

[0380] In the aforementioned data generation apparatus 2, a third input value 225 is provided based on the detection limit of the second component, thereby generating a new sample 65 that can be set as a limit case. Alternatively, in Figure 18B On the output screen, etc., the control unit 31 can appropriately accept the specification of the degree of the second feature required in the extreme cases. Furthermore, the control unit 31 can entrust the generation of a new sample 65 by notifying the data generation device 2 of the category of the first feature, the category of the second feature, and the specified degree of the second feature. Thus, a new sample 65 that can be set as an extreme case can be generated. The generated new sample 65 can be provided to the inference device 3 by any method and added to the data set 320. By adding this new sample 65, the degree of the second feature appearing in the samples 321 contained in the data set 320 is diversified, and by setting appropriate extreme cases, the detection accuracy of the second component can be improved.

[0381] It should be noted that the series of processes for setting this parameter value can be performed by the data generation device 2 described above. In this case, the data generation device 2 will... Figure 18A and Figure 18B The example output screens are sent to output device 26, while the aforementioned series of processes for generating new sample 65 are executed. Additionally, in data generation device 2, when generating new sample 65, samples 321 contained in data group 320 can be viewed through the aforementioned output screens. In this case, each output screen can be used as a reference for specifying each input value (221, 223, 225).

[0382] (Step S303)

[0383] return Figure 17A In step S303, the control unit 11 operates as the output unit 313 and outputs information related to the above-mentioned predicted result (in this embodiment, the detection result of the second component).

[0384] The output target and the content of the output information can be appropriately determined according to the embodiment. For example, the control section 31 can output the result obtained by performing the prescribed inference on the object sample 325 by the step S302 directly to the output device 36. Alternatively, for example, the control section 31 can perform some kind of information processing based on the result obtained by performing the prescribed inference. Also, the control section 31 can output the result of performing the information processing as information related to the result of the inference. The output of the result of performing the information processing can include, for example, outputting a specific message according to the result of the inference, and controlling the operation of the control target device according to the result of the inference. The output target can be, for example, the output device 36, the output device of another computer, the control target device, or the like.

[0385] After the output of the information related to the result of the inference is completed, the control section 31 ends the processing flow of the present action example. Note that the control section 31 can continue to repeat the series of information processing of the steps S301 to S303 for a prescribed period. The timing of the repetition can be arbitrary. Thus, the inference device 3 can continue to perform the prescribed inference.

[0386] [Features]

[0387] As described above, the learning device 1 of the present embodiment can construct the first generator 411 of which the training is completed by the processing of the steps S101 to S103, and the first generator 411 of which the training is completed acquires the ability to generate the pseudo sample including the first feature of the class specified by the input value. Also, the learning device 1 of the present embodiment can construct the second generator 421 of which the training is completed by the processing of the steps S111 to S113, and the second generator 421 of which the training is completed acquires the ability to generate the pseudo sample including the second feature of the class specified by the input value. In the data generation device 2 of the present embodiment, the classes of the first feature and the second feature each included in the generated new sample 65 can be controlled based on the input values (221, 223) respectively provided to the respective generators (411, 421) of which the training is completed. That is, the new sample 65 including a combination of different classes of the first feature and the second feature can be freely generated. For example, in the case where there is a combination in which the sample 321 is not obtained or the number of samples 321 is small in the data group 320, the new sample 65 including the combination can be generated by the data generation device 2. Further, the processing of the steps S201 to S205 can be mostly automated. Thus, according to the present embodiment, the samples including a plurality of different combinations can be freely and automatically generated, and therefore, the cost of collecting a plurality of different samples for a prescribed kind of data exhibiting at least two or more features can be reduced.

[0388] Further, in the estimation device 3 of the present embodiment, by adding the generated new sample 65 to the data group 320, it is possible to seek to diversify the combination of the first feature and the second feature exhibited in the samples 321 contained in the data group 320. That is, in the data group 320, it is possible to enrich the samples 321 for which the combination has not been obtained or for which the number of combinations is small. Thus, by reflecting the generated new sample 65 in the estimation processing of the estimation device 3, it is possible to expect an improvement in the accuracy of the estimation processing related to the second feature.

[0389] Further, the learning device 1 of the present embodiment can construct the third generator 431 for which the training is completed through the processing of the above-described steps S121 to S123, and the third generator 431 for which the training is completed acquires the ability to generate the pseudo value of the attribute of the second feature of the class and the degree specified by the input value. In the data generation device 2 of the present embodiment, it is possible to control the value of the attribute of the second feature contained in the generated new sample 65 on the basis of each input value (223, 225) provided to the third generator 431 for which the training is completed. That is, in addition to the first generator 411 and the second generator 421 for which the training is completed, by further utilizing the third generator 431 for which the training is completed, it is possible to freely generate the new sample 65 including the second feature having an arbitrary attribute value. Thus, according to the present embodiment, it is possible to generate more different samples with respect to the data of the prescribed kind capable of exhibiting at least two or more features.

[0390] Note that, by providing the third input value 225 in accordance with the limit of the detection of the second component, it is possible to generate a plurality of different samples corresponding to the limit case of the detection. Further, in the case where the machine learning of the third generator 431 is implemented using the first label 512 in the processing of the above-described steps S121 to S123, the third generator 431 for which the training is completed that is constructed further acquires the ability to generate the pseudo value of the attribute of the second feature corresponding to the class of the first feature. Thus, in the data generation device 2 of the present embodiment, by utilizing the third generator 431 for which the training is completed, it is possible to generate the new sample 65 including the second feature having the attribute value suitable for the class of the first feature.

[0391] §4 Modification

[0392] The embodiments of the present application have been described in detail thus far, but the description thus far is merely an example of the present application in all respects. Of course, various modifications or changes can be made within the scope of the present application. For example, the following modifications can be made. Note that, hereinafter, the same symbols will be used with respect to the members identical to those of the above-described embodiments, and the description will be appropriately omitted with respect to the aspects identical to those of the above-described embodiments. The following modifications can be appropriately combined.

[0393] <4.1>

[0394] The learning device 1, the data generation device 2, and the estimation device 3 of the above-described embodiments can be applied to all situations in which a plurality of different samples of data of a prescribed kind that can exhibit at least two or more features are collected. For example, the above-described embodiments can be applied to situations in which a plurality of different samples of sensor data obtained by observing an object using a sensor are collected. Hereinafter, a modification example that limits the application situation will be shown.

[0395] (A) Situation of appearance inspection

[0396] Figure 19 An example of an application situation of the inspection system 100A of the first modification example of the schematic example. This modification example is an example in which the above-described embodiments are applied in a situation in which appearance inspection of a product is performed using image data that presents the product. The inspection system 100A of this modification example is provided with the learning device 1, the data generation device 2, and the inspection device 3A. The learning device 1, the data generation device 2, and the inspection device 3A can be connected to each other via a network, similarly to the above-described embodiments. Figure 2A An example of an application situation of the inspection system 100A of the first modification example of the schematic example. This modification example is an example in which the above-described embodiments are applied in a situation in which appearance inspection of a product is performed using image data that presents the product. The inspection system 100A of this modification example is provided with the learning device 1, the data generation device 2, and the inspection device 3A. The learning device 1, the data generation device 2, and the inspection device 3A can be connected to each other via a network, similarly to the above-described embodiments.

[0397] The prescribed kind of data processed in this modification example is image data in which a product RA is presented. This image data can be obtained by capturing the product RA using a camera SA. The first feature is related to a background that includes the product RA, and the second feature is related to a defect of the product RA. The kind of product RA and the kind of defect can not be particularly limited and can be appropriately selected according to the embodiments. As described above, the product RA can be, for example, a product conveyed on a production line, such as an electronic device, an electronic component, an automobile component, a medicine, a food, or the like. The defect can be, for example, a scratch, a stain, a crack, a dent, a burr, uneven color, foreign matter contamination, or the like. In addition to these limitations, the inspection system 100A of this modification example can be configured identically to the estimation system 100 of the above-described embodiments.

[0398] (Learning device)

[0399] In this modification example, the first sample 511 of the first learning data set 51 can be a sample of image data in which the product RA is presented, and the first label 512 can indicate the category of the product RA. The learning device 1 can construct the first generator 411 that has completed training by performing the processes of steps S101 to S103, and the first generator 411 that has completed training has acquired the ability to generate a pseudo sample in which a background that includes a product RA of a category specified by an input value is presented.

[0400] In the present modification example, the second sample 521 of the second learning data set 52 can be a sample of image data in which a defect of the product RA is present, and the second label 522 can indicate a category of the defect. The learning device 1 can construct the second generator 421 of which the training is completed by performing the processes of the above-described steps S111 to S113, and the second generator 421 of which the training is completed acquires the ability to generate a pseudo sample in which a defect of a category specified by an input value is present.

[0401] In the present modification example, the sample value 531 of the third learning data set 53 can indicate a degree of an attribute of a defect. The attribute of the defect can be, for example, an area, a width, a height, a circumference, an aspect ratio, a circularity, or the like, which is a shape-related attribute. The reflection level 532 can indicate a degree of reflection of the defect. The reflection level 532 can be set in association with a degree of the defect, such as a degree of severity of the defect, a quality level of the product, or the like. The learning device 1 can construct the third generator 431 of which the training is completed by performing the processes of the above-described steps S121 to S123, and the third generator 431 of which the training is completed acquires the ability to generate a pseudo value of an attribute of a defect of a category and a degree specified by an input value.

[0402] In the present modification example, the learning device 1 can construct the predictor 44 of which the training is completed by performing the processes of the above-described steps S131 to S133, and the predictor 44 of which the training is completed acquires the ability to generate a pseudo sample 542 by the first generator 411 of which the training is completed. In addition, in the present modification example, the learning device 1 can generate the third sample 551 to the second sample 521 in which the product RA and the defect are present by performing the processes of the above-described steps S141 to S144 using the first generator 411 of which the training is completed and the predictor 44.

[0403] (Data generation device)

[0404] In the present modification example, the first input value 221, the second input value 223, and the third input value 225 can be specified in accordance with a category of the product RA, a category of the defect, and a degree of reflection of the defect, which are present in a sample that is desired to be generated. The data generation device 2 can generate a new sample 65 in which a defect of a specified category and the product RA are present while controlling a degree of reflection of the defect by performing the processes of the above-described steps S201 to S205 using the respective generators (411, 421, 431) of which the training is completed.

[0405] (Inspection device)

[0406] Figure 20A An example of a hardware configuration of the inspection device 3A of the present modification example is schematically shown. As shown in FIG. 18, the inspection device 3A includes a CPU 301, a ROM 302, a RAM 303, a display 304, an input device 305, a storage device 306, a communication device 307, and a bus 308. Figure 20BAs shown, the inspection device 3A of the present modified example is configured of a computer to which a control section 31, a storage section 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a driver 37 are electrically connected, similarly to the above-described estimation device 3.

[0407] In the present modified example, the storage section 32 stores various information such as an inspection program 83A. The inspection program 83A corresponds to the above-described estimation program 83 of the embodiment. In addition, in the present modified example, the inspection device 3A is connected to a camera SA via the external interface 34. The camera SA can be, for example, a normal RGB camera, a depth camera, an infrared camera, or the like. The camera SA can be appropriately disposed at a place where the product RA can be photographed. For example, the camera SA can be disposed near a conveyance device that conveys the product RA.

[0408] Note that the hardware configuration of the inspection device 3A can not be limited to such an example. With regard to the specific hardware configuration of the inspection device 3A, components can be appropriately omitted, replaced, and added according to the embodiment. The inspection device 3A can also be an information processing device designed specifically for a service provided, as well as a general-purpose server device, a general-purpose PC, a PLC, or the like.

[0409] Figure 21 An example of the software configuration of the inspection device 3A of the present modified example is schematically shown. Similarly to the above-described embodiment, the software configuration of the inspection device 3A is realized by the control section 31 executing the inspection program 83A. The software configuration of the inspection device 3A is the same as the above-described estimation device 3 except that the data processed is limited as described above. The inspection section 312A corresponds to the above-described estimation section 312. Thus, the inspection device 3A can perform a series of processes related to appearance inspection, similarly to the estimation process of the above-described estimation device 3.

[0410] That is, in step S301, the control section 31 acquires, from the camera SA, an object sample 325 on which the product RA to be inspected is present. In step S302, the control section 31 functions as the inspection section 312A and determines whether or not there is a defect in the product RA to be inspected by analyzing the object sample 325 using the data set 320. The sample 321 included in the data set 320 presents the product RA and the defect. The sample 321 of the limit case can be selected for each category of the detected defect. The respective threshold values (T1, T2) as the detection parameters can be set based on the sample 321 selected as the limit case. Thus, the control section 31 can estimate the state of the product RA.

[0411] In step S303, the control section 31 outputs information related to the result of the estimation of the state of the product RA. For example, the control section 31 can output the result of the estimation of the state of the product RA directly to the output device 36. In addition, for example, in a case where it is determined that the product RA includes a defect, the control section 31 can output a warning for notifying this content to the output device 36. In addition, for example, in a case where the inspection device 3A is connected to a conveyance device that conveys the product RA, the conveyance device can be controlled on the basis of the result of the estimation of the state of the product RA to convey the product RA having a defect and the product RA having no defect through different lines.

[0412] (Characteristics)

[0413] According to the present modification example, in the data generation device 2, a new sample 65 in which an arbitrary combination of the product RA and the defect is presented can be freely and automatically generated. Therefore, it is possible to reduce the cost of collecting a plurality of different samples that exhibit various combinations of the product RA and the defect. In addition, by adding the generated new sample 65 to the data group 320, it is possible to reduce or eliminate the combination of the product RA and the defect that is not obtained as a sample 321. Thus, it is possible to seek to improve the accuracy of the appearance inspection of the inspection device 3A.

[0414] [Other Modes]

[0415] In the above-described modification example, the defect of the product RA is an example of a foreground. That is, the inspection processing in the above-described modification example is an example of processing of detecting a target foreground from a sample of image data of a background and a foreground. That is, the prescribed kind of data that is processed can be image data of a background and a foreground. The first feature can be related to the background, and the second feature can be related to the foreground. The kinds of the background and the foreground can each not be limited to the examples of the product RA and the defect, and can be appropriately selected according to the implementation.

[0416] (A-1) Context of Image Diagnosis

[0417] Figure 21 The schematic example is an example of another context of detecting a foreground from image data. Figure 21 The diagnostic system of the example includes the learning device 1, the data generation device 2, and an image diagnosis device 3A1. The image diagnosis device 3A1 corresponds to the above-described inspection device 3A. The configuration of the image diagnosis device 3A1 can be the same as that of the above-described inspection device 3A. Figure 21 The prescribed kind of data that is processed in the example of the image diagnosis device 3A1 can be image data for medical use (hereinafter, also referred to as medical image) in which an organ is presented. The organ can be, for example, a digestive system, a circulatory system, a respiratory system, a urinary system, a reproductive system, an endocrine system, a sensory system, a nervous system, a motor system (bone, joint, ligament, muscle), or the like.

[0418] The medical image can be acquired using a medical examination apparatus SA1. The medical examination apparatus SA1 is an example of a sensor. The medical examination apparatus SA1 can be, for example, an X-ray apparatus, a CT apparatus, an MRI apparatus, or the like. Accordingly, the obtained medical image can be, for example, an X-ray image, a CT image, an MRI image, or the like. In Figure 22 In the example of FIG. 10, the image diagnostic apparatus 3A1 is connected to the medical examination apparatus SA1, whereby the object sample 325 of the medical image can be acquired.

[0419] The first feature can be related to an organ, and the second feature can be related to a lesion (for example, a shadow generated due to a disease). An image of an organ is an example of a background, and an image of a lesion is an example of a foreground. The attribute of the lesion can be related to a shape of a shadow appearing on the medical image. The reflection level 532 can be set in accordance with, for example, a severity of a disease, a progression of a disease, or the like. The image diagnostic apparatus 3A1 can perform a process of detecting a lesion from the object sample 325 of the medical image by the same flow as the examination apparatus 3A described above. In a case where it is determined that there is a lesion, the image diagnostic apparatus 3A1 can output a diagnosis result such as a category, a position, or the like of the lesion in step S303.

[0420] (A-2) A situation of monitoring a road condition

[0421] Figure 22 An example of the situation in which the illustrative example detects a foreground from image data. Figure 22 The monitoring system of the example of FIG. 10 has the learning apparatus 1, the data generation apparatus 2, and a monitoring apparatus 3A2. The monitoring apparatus 3A2 corresponds to the examination apparatus 3A described above. The monitoring apparatus 3A2 can have the same configuration as the examination apparatus 3A described above. Figure 23 The prescribed kind of data processed in the example of FIG. 10 can be image data in which a road to be monitored is presented (hereinafter, also referred to as a monitoring image).

[0422] The monitoring image can be acquired using a camera SA2 configured to capture the road to be monitored. The monitoring apparatus 3A2 can be connected to the camera SA2 via, for example, a network or the like as appropriate. Thereby, the monitoring apparatus 3A2 can acquire the object sample 325 of the monitoring image.

[0423] The first feature can be related to a background such as a road, a building, and the like, and the second feature can be related to a condition of the road. The condition of the road can be expressed by, for example, a degree of mixture of moving bodies (a degree of congestion), a degree of danger of crime, and the like. The moving bodies can be pedestrians, vehicles, and the like. The degree of danger of crime can be set, for example, such that an image of a road with a small number of pedestrians and darkness is high, and an image of a road with a large number of pedestrians and brightness is low. The attribute of the condition of the road can be expressed by, for example, an attribute of the moving bodies, an attribute of the environment, and the like. The attribute of the moving bodies can be related to, for example, a position, a number, a speed, and the like. In addition thereto, in a case where the moving bodies are persons, the attribute of the moving bodies can be related to, for example, a gender, a physique, and the like. The attribute of the environment can be related to, for example, a time, a season, a presence or absence of a streetlight, and the like, which are brightness elements. The image related to the condition of the road is an example of the foreground. The reflection level 532 can be set in accordance with the condition of the road. In this case, the monitoring device 3A2 can estimate the condition of the object road (for example, the degree of mixture of the moving bodies, the degree of danger of crime) from the object sample 325 of the monitoring image by the same flow as the above-described checking device 3A.

[0424] According to each of the above-described modes, in the data generation device 2, a new sample 65 in which an arbitrary combination of a background and a foreground is presented can be freely and automatically generated. In the above-described context of image diagnosis, a plurality of different medical images exhibiting various disease states can be obtained based on images of organs and images of lesions obtained by different persons, respectively. In the context of monitoring the above-described road conditions, a plurality of different monitoring images exhibiting various road conditions can be obtained based on images obtained under different conditions (places, times, and the like). Thus, it is possible to reduce a cost required for collecting a plurality of different samples exhibiting various combinations of backgrounds and foregrounds. In addition, by adding the generated new sample 65 to the data group 320, it is possible to reduce or eliminate a combination of a background and a foreground for which a sample 321 is not obtained. Thus, it is possible to seek to improve the accuracy of processing for detecting each foreground.

[0425] (B) Context of monitoring a driving state

[0426] Figure 24A An example of an application context of the monitoring system 100B of the illustrative example second modification. This modification is an example in which the above-described embodiment is applied in a context of monitoring a driving state of a vehicle by a driver using sensing data obtained by a vehicle-mounted sensor provided in the vehicle. The monitoring system 100B of this modification is provided with the learning device 1, the data generation device 2, and the monitoring device 3B. The learning device 1, the data generation device 2, and the monitoring device 3B can be connected to each other via a network, similarly to the above-described embodiment.

[0427] The prescribed type of data processed in this modification example is acceleration data (sensing data) of the vehicle obtained by the acceleration sensor SB. The first feature is related to a transition of acceleration generated at the time of normal driving, and the second feature is related to a transition of acceleration generated due to a sudden event or the like. Other than these limitations, the monitoring system 100B of this modification example can be configured identically to the inference system 100 of the above-described embodiment.

[0428] (Learning device)

[0429] In this modification example, the first sample 511 of the first learning data set 51 can be a sample of acceleration data that exhibits a transition of acceleration generated at the time of normal driving, and the first label 512 can indicate a category of a factor (hereinafter, an influence factor) that affects normal driving, such as a type of driver, a type of vehicle, and the like. The type of driver can be set according to a personality exhibited at the time of driving, such as cautious, rough, and the like. The learning device 1 can construct the first generator 411 that has completed training by executing the processes of steps S101 to S103, and the first generator 411 that has completed training has acquired the ability to generate a pseudo sample that exhibits an influence factor of a category specified by an input value, that is, a transition of acceleration generated at the time of normal driving.

[0430] In this modification example, the second sample 521 of the second learning data set 52 can be a sample of acceleration data that exhibits a transition of acceleration generated due to a sudden event, and the second label 522 can indicate a category of a sudden event. The learning device 1 can construct the second generator 421 that has completed training by executing the processes of steps S111 to S113, and the second generator 421 that has completed training has acquired the ability to generate a pseudo sample that exhibits a transition of acceleration generated due to an event of a category specified by an input value.

[0431] In this modification example, the sample value 531 of the third learning data set 53 can indicate a degree of a transition of acceleration. The transition of acceleration can be expressed by, for example, an amplitude, a frequency, and the like. The reflection level 532 can indicate a degree of reflection of the transition of acceleration. The reflection level 532 can be set in association with a degree of a possibility of an accident, such as a degree of severity of a sudden event. The learning device 1 can construct the third generator 431 that has completed training by executing the processes of steps S121 to S123, and the third generator 431 that has completed training has acquired the ability to generate a pseudo value of a degree of a transition of acceleration of an event and a degree specified by an input value.

[0432] In the present modification, the learning device 1 can construct the trained predictor 44 by executing the processes of steps S131 to S133 described above, and the trained predictor 44 has acquired the ability to estimate the input provided to the trained first generator 411 when the pseudo sample 542 generated by the trained first generator 411 is generated. In addition, in the present modification, the learning device 1 can generate the third sample 551 to the second sample 521 including the transition of acceleration generated during normal driving and the transition of acceleration due to an emergency event by executing the processes of steps S141 to S144 described above using the trained first generator 411 and the predictor 44.

[0433] (Data generation device)

[0434] In the present modification, the first input value 221, the second input value 223, and the third input value 225 can be specified in accordance with the category of the influence factor, the category of the emergency event, and the degree of reflection of the emergency event in the sample to be generated. The data generation device 2 can generate a new sample 65 including the transition of acceleration generated during normal driving and the transition of acceleration due to an emergency event while controlling the degree of reflection of the emergency event by executing the processes of steps S201 to S205 described above using the trained generators (411, 421, 431).

[0435] (Monitoring device)

[0436] Figure 24A An example of a hardware configuration of the monitoring device 3B of the present modification is schematically shown. As shown in FIG. 18, like the estimation device 3 described above, the monitoring device 3B of the present modification can be composed of a computer to which a control section 31, a storage section 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a driver 37 are electrically connected. Figure 24B

[0437] In the present modification, the storage section 32 stores various information such as a monitoring program 83B. The monitoring program 83B corresponds to the estimation program 83 of the embodiment described above. In addition, in the present modification, the monitoring device 3B is connected to an acceleration sensor SB via the external interface 34. The acceleration sensor SB can be appropriately arranged so as to be able to measure acceleration during travel of a vehicle.

[0438] Note that the hardware configuration of the monitoring device 3B can not be limited to this example. With respect to the specific hardware configuration of the monitoring device 3B, components can be appropriately omitted, replaced, and added according to the embodiment. The monitoring device 3B is an information processing device designed specifically for a service provided, and a general-purpose computer, a mobile phone including a smartphone, a vehicle-mounted device, and the like. ​

[0439] Figure 25 An example of a software configuration of the monitoring device 3B of the illustrative example of the present modification. The software configuration of the monitoring device 3B is realized by the control section 31 executing the monitoring program 83B, as in the above-described embodiment. The software configuration of the monitoring device 3B is the same as that of the above-described estimation device 3 except for the point that the data to be processed is limited as above. The monitoring section 312B corresponds to the above-described estimation section 312. Thus, the monitoring device 3B can perform a series of processes related to monitoring of the driving state of the driver in the same manner as the estimation process of the above-described estimation device 3.

[0440] That is, in step S301, the control section 31 acquires, from the acceleration sensor SB, an object sample 325 of acceleration data that exhibits a transition of acceleration of the estimation object. In step S302, the control section 31 functions as the monitoring section 312B and detects a transition of acceleration due to a sudden event from the object sample 325 by analyzing the object sample 325 using the data group 320. The samples 321 included in the data group 320 include a transition of acceleration generated at the time of normal driving and a transition of acceleration generated due to a sudden event. The samples 321 of limit cases can be selected for each detected sudden event. The parameters of detection can be set based on the samples 321 selected as limit cases. Thus, the control section 31 can estimate the driving state of the driver. Note that the estimation of the driving state can be performed in real time or for the purpose of evaluating the driving state at a past time.

[0441] In step S303, the control section 31 outputs information related to the estimation result of the driving state of the driver. For example, the control section 31 can create a history related to the transition of acceleration due to a sudden event and save the created history in a prescribed storage area. The history can directly represent the transition of acceleration obtained in step S302 or can represent a result of performing some kind of information processing on the obtained transition of acceleration. As an example, the control section 31 can evaluate the degree of possibility of sending an accident based on the transition of acceleration due to a sudden event and create a history representing the evaluation result. Further, the control section 31 can recognize the action of the driver such as sudden braking, sudden acceleration, sudden turning, and the like based on the transition of acceleration due to a sudden event and create a history representing the recognition result. The index for recognizing the action can be appropriately set. The monitoring device 3B can continuously monitor the driving state of the driver by repeatedly performing the processes of steps S301 to S303 during the driving of the vehicle by the driver.

[0442] (Features)

[0443] According to the present modification example, in the data generation device 2, a new sample 65 that exhibits a transition of acceleration generated by an arbitrary combination of a category of an influence factor and a sudden event can be freely and automatically generated. Therefore, it is possible to reduce the cost of collecting a plurality of different samples that exhibit various combinations of a category of an influence factor and a sudden event. In addition, by adding the generated new sample 65 to the data group 320, it is possible to reduce or eliminate a combination of a category of an influence factor and a sudden event for which a sample 321 is not obtained. Thus, it is possible to improve the accuracy of the process of estimating the driving state of the driver in the monitoring device 3B.

[0444] [Other modes]

[0445] Note that the acceleration sensor SB is an example of an in-vehicle sensor. In addition to the acceleration sensor SB, an in-vehicle sensor can use, for example, a camera, a Lidar sensor, a millimeter wave radar, an ultrasonic sensor, or the like. Thus, instead of the driving state of the driver, a situation outside the vehicle can be estimated, or a situation outside the vehicle can be estimated in conjunction with the driving state. In this case, as with the above, the first feature can be related to various transitions generated during normal driving, and the second feature can be related to various transitions generated due to a sudden event. The category of the first feature can be expressed by a category of an influence factor during normal driving, and the category of the second feature can be expressed by a category of a sudden event.

[0446] (C) Situation of detecting an anomaly of a machine

[0447] Figure 2B An example of an application situation of the detection system 100C of the illustrative example third modification example. The present modification example is a situation in which the above-described embodiment is applied in detecting an anomaly of a machine using sound data of a running sound of a machine that includes Figure 26A An example of a situation in which the above-described embodiment is applied in detecting an anomaly of a machine using sound data of a running sound of a machine that includes

[0448] The prescribed kind of data processed in this modification example is sound data including the running sound of a machine RC. This sound data can be obtained by recording the running sound of the machine RC using a microphone SC. The first feature is related to the running sound produced when the machine RC is normally running, ambient sound such as noise around the machine RC, and the like, and the second feature is related to abnormal sound produced due to abnormalities such as aging, malfunction, and the like of the machine RC. The kind of machine RC and abnormality can not be particularly limited and can be appropriately selected according to the embodiment. As described above, the machine RC can be, for example, a device that constitutes a production line, such as a conveyance device, an industrial robot, and the like. The machine RC can be the entire device or a part of the device such as a motor. The abnormality can be, for example, a malfunction, a foreign object mixed in, dirt attached, and a worn-out component. In addition to these limitations, the detection system 100C of this modification example can be configured in the same manner as the inference system 100 of the above-described embodiment.

[0449] (Learning device)

[0450] In this modification example, the first sample 511 of the first learning data set 51 can be a sample of sound data including ambient sound, and the first label 512 can indicate the category of the machine RC and the surrounding environment. As described above, the surrounding environment of the machine RC can be distinguished by, for example, the place where the machine RC is driven, the time, and the like. The learning device 1 can construct the first generator 411 that has completed training by performing the processes of steps S101 to S103 described above, and the first generator 411 that has completed training has acquired the ability to generate a pseudo sample including ambient sound of the machine RC and the surrounding environment of the category specified by the input value.

[0451] In this modification example, the second sample 521 of the second learning data set 52 can be a sample of sound data including abnormal sound, and the second label 522 can indicate the category of the abnormality. The learning device 1 can construct the second generator 421 that has completed training by performing the processes of steps S111 to S113 described above, and the second generator 421 that has completed training has acquired the ability to generate a pseudo sample including abnormal sound produced due to the abnormality of the category specified by the input value.

[0452] In this modification example, the sample value 531 of the third learning data set 53 can indicate the degree of the attribute of the abnormal sound. The attribute of the abnormal sound can be expressed by, for example, the amplitude, the frequency, and the like. The reflection level 532 can indicate the degree of reflection of the abnormal sound. The reflection level 532 can be set in association with the degree of the abnormality, such as the severity of the abnormality, the normality of the machine RC, and the like. The learning device 1 can construct the third generator 431 that has completed training by performing the processes of steps S121 to S123 described above, and the third generator 431 that has completed training has acquired the ability to generate a pseudo value of the attribute of the abnormal sound of the category and the degree specified by the input value.

[0453] In this modification, the learning device 1 can construct the trained predictor 44 by performing the processes of steps S131 to S133 described above, and the trained predictor 44 has acquired the ability to estimate the input provided to the first trained generator 411 when the pseudo sample 542 generated by the first trained generator 411 is generated. In addition, in this modification, the learning device 1 can generate the third to second samples 551 to 521 including the environmental sound and the abnormal sound by performing the processes of steps S141 to S144 described above using the first trained generator 411 and the predictor 44.

[0454] (Data generation device)

[0455] In this modification, the first to third input values 221, 223, and 225 can be specified in accordance with the category of the machine and the surrounding environment, the category of the abnormality, and the degree of reflection of the abnormal sound in the sample to be generated. The data generation device 2 can generate a new sample 65 including the environmental sound and the abnormal sound of the specified category while controlling the degree of reflection of the abnormal sound by performing the processes of steps S201 to S205 described above using the trained generators (411, 421, 431).

[0456] (Abnormality detection device)

[0457] Figure 26A An example of a hardware configuration of the abnormality detection device 3C of the illustrative example of this modification is shown. As shown in FIG. 18, like the estimation device 3 described above, the abnormality detection device 3C of this modification can be configured of a computer to which a control section 31, a storage section 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a driver 37 are electrically connected. Figure 26B

[0458] In this modification, the storage section 32 stores various information such as an abnormality detection program 83C. The abnormality detection program 83C corresponds to the estimation program 83 of the embodiment described above. In addition, in this modification, the abnormality detection device 3C is connected to a microphone SC via the external interface 34. The microphone SC can be appropriately disposed at a place where the running sound of the machine RC can be recorded.

[0459] Note that the hardware configuration of the abnormality detection device 3C can not be limited to this example. With respect to the specific hardware configuration of the abnormality detection device 3C, components can be appropriately omitted, replaced, and added according to the embodiment. The abnormality detection device 3C can be an information processing device designed specifically for the service provided, and a general-purpose server device, a general-purpose PC, a PLC, and the like.

[0460] Figure 27 ​An example of a software configuration of the abnormality detection device 3C of the illustrative example of the present variation. The software configuration of the abnormality detection device 3C is realized by the control section 31 executing the abnormality detection program 83C, as with the above-described embodiment. The software configuration of the abnormality detection device 3C is the same as the above-described estimation device 3 except for the data processed being defined as above. The detection section 312C corresponds to the above-described estimation section 312. Thus, the abnormality detection device 3C can perform a series of processes related to detection of an abnormality of the machine RC in the same manner as the estimation process of the above-described estimation device 3.

[0461] That is, in step S301, the control section 31 acquires the object sample 325 including the operation sound of the machine RC to be inspected from the microphone SC. In step S302, the control section 31 functions as the detection section 312C, and determines whether or not the machine RC to be inspected has generated an abnormality by analyzing the object sample 325 using the data group 320. The samples 321 included in the data group 320 include the environmental sound and the abnormal sound. The samples 321 of the limit case can be selected for each category of the abnormality detected. The parameters of detection can be set based on the samples 321 selected as the limit case. Thus, the control section 31 can estimate the state of operation of the machine RC. In the case where the operation of the machine RC has generated an abnormality, the control section 31 can detect the abnormality.

[0462] In step S303, the control section 31 outputs information related to the estimation result of the state of operation of the machine RC. For example, the control section 31 can directly output the estimation result of the state of operation of the machine RC to the output device 36. In addition, for example, in the case where it is detected that an abnormality has been generated by the process of step S302, the control section 31 can output a warning for notifying that the abnormality has been generated to the output device 36. Also, in the case where the abnormality detection device 3C is configured to be able to control the operation of the machine RC, the control section 31 can stop the operation of the machine RC in accordance with the detection of the generation of an abnormality. Further, the control section 31 can output information of the category of the abnormality generated on the machine RC and a maintenance method for coping with the abnormality to the output device 36. The information indicating the maintenance method for coping with the abnormality can be saved in a prescribed storage area of the storage section 32, the storage medium 93, an external storage device, a storage medium, or the like. The control section 31 can appropriately acquire the maintenance method for coping with the abnormality from the prescribed storage area.

[0463] (Features)

[0464] According to the present modification, in the data generation device 2, a new sample 65 including a combination of an arbitrary category of environmental sound and abnormal sound can be freely and automatically generated. Therefore, it is possible to reduce the cost required for collecting a plurality of different samples including various combinations of environmental sound and abnormal sound. In addition, by adding the generated new sample 65 to the data group 320, it is possible to reduce or eliminate the combination of environmental sound and abnormal sound for which a sample 321 is not obtained. Thus, it is possible to improve the accuracy of the process of detecting an abnormality of the machine RC by the abnormality detection device 3C.

[0465] [Other methods]

[0466] Note that the microphone SC is an example of a sensor for monitoring the state of the machine RC. The method of detecting an abnormality of the machine RC by sensing data can not be limited to the method of using sound data including the operating sound of the machine RC described above. In addition to the microphone SC, a sensor that measures the vibration of the machine, such as an acceleration sensor, a vibration sensor, or the like, can also be used to detect an abnormality of the machine RC. In this case, only the sound (vibration of air) is replaced with the physical vibration of the machine RC, and it is possible to detect an abnormality of the machine RC by the same method as described above.

[0467] <4.2>

[0468] In the above-described embodiment, each of the generators (411, 421, 431), each of the recognizers (412, 422, 432), and the predictor 44 uses a fully coupled neural network. However, the type of neural network that configures each of the generators (411, 421, 431), each of the recognizers (412, 422, 432), and the predictor 44 can not be limited to such an example. Each of the generators (411, 421, 431), each of the recognizers (412, 422, 432), and the predictor 44 can use a convolutional neural network, a recurrent neural network, or the like.

[0469] In addition, in the above-described embodiment, the type of machine learning model that configures each of the generators (411, 421, 431), each of the recognizers (412, 422, 432), and the predictor 44 can not be limited to a neural network. Each of the recognizers (412, 422, 432) and the predictor 44 can use, for example, a support vector machine, a regression model, a decision tree model, or the like, in addition to a neural network. The method of machine learning can be appropriately selected according to the type of each machine learning model.

[0470] Further, in the above-described embodiment, the form of the input and output of each of the generators (411, 421, 431), each of the discriminators (412, 422, 432), and the inferrer 44 can not be particularly limited and can be appropriately determined according to the embodiment. For example, the first generator 411 can be configured to further accept the input of information other than the class of the first feature and the noise. Further, for example, the first generator 411 can be configured to output information other than the pseudo sample. The same applies to the others.

[0471] <4.3>

[0472] In the above-described embodiment, each of the learning models 41 to 43 is constituted by each of the generators (411, 421, 431) and each of the discriminators (412, 422, 432). However, as the constitution of each of the learning models 41 to 43, as long as each of the generators (411, 421, 431) can be trained to acquire the ability to generate corresponding false data, it can not be limited to such an example and can be appropriately determined according to the embodiment.

[0473] Figure 28 An example of the constitution of the first learning model 41D of the illustrative example of the present modification. The first learning model 41D of the present modification is constituted by an encoder 415 and a decoder 411D. The encoder 415 is configured to convert a sample including a first feature and information indicating the class of the first feature into a feature amount. The decoder 411D is configured to restore a sample including a corresponding first feature from the feature amount obtained by the encoder 415 and information indicating the class of the corresponding first feature. In the present modification, the decoder 411D corresponds to the first generator 411. The feature amount obtained by the encoder 415 corresponds to the noise (latent variable) in the above-described embodiment.

[0474] The encoder 415 and the decoder 411D are each constituted by a machine learning model. The kind of the machine learning model that constitutes each of the encoder 415 and the decoder 411D can not be particularly limited and can be appropriately selected according to the embodiment. As in the above-described embodiment, the machine learning model that constitutes each of the encoder 415 and the decoder 411D can use, for example, a neural network. In the case where each of the encoder 415 and the decoder 411D is constituted by a neural network, in the above-described step S102, the learning device 1 can implement the machine learning of the first learning model 41D as described below.

[0475] That is, the control section 11 inputs the first sample 511 and the first label 512 to the encoder 415 with respect to each of the first learning data sets 51 and executes the operation processing of the encoder 415. Thereby, the control section 11 acquires an output value corresponding to the result of converting the first sample 511 and the first label 512 into a feature amount from the encoder 415.

[0476] Next, the control section 11 inputs the feature quantity obtained from the encoder 415 and the corresponding first label 512 to the decoder 411D, and performs the operation processing of the decoder 411D. Thereby, the control section 11 acquires the output value corresponding to the restored sample 515D restored from the feature quantity and the first label 512 from the decoder 411D.

[0477] The control section 11 calculates the error of the restored sample 515D obtained with respect to each first learning data set 51 and the first sample 511. The control section 11 calculates the error of the value of the operation parameter of each of the encoder 415 and the decoder 411D using the gradient of the calculated error by the error back propagation method. The control section 11 updates the value of the operation parameter of each of the encoder 415 and the decoder 411D based on the calculated error. The control section 11 adjusts the value of the operation parameter of each of the encoder 415 and the decoder 411D by this series of update processing to reduce the sum of the calculated errors.

[0478] Note that, in addition to the above update processing, the control section 11 can further calculate the error of the feature quantity obtained from the encoder 415 and the value obtained from a prescribed probability distribution (for example, a Gaussian distribution or the like), and adjust the value of the operation parameter of the encoder 415 to reduce the sum of the errors. Thereby, the control section 11 can normalize the output value of the encoder 415.

[0479] By the above machine learning, the trained encoder 415 can be constructed, and the trained encoder 415 obtains the ability to convert a sample including a first feature and information indicating the category of the first feature into a suitable feature quantity. At the same time, the trained decoder 411D can be constructed, and the trained decoder 411D obtains the ability to generate a sample including a corresponding first feature from a feature quantity and information indicating the category of the first feature.

[0480] The feature quantity corresponds to the above noise, and therefore, the trained decoder 411D can perform processing identically to the above trained first generator 411. The data generation device 2 can generate the first pseudo sample 61 using the trained decoder 411D by the same processing flow as the above embodiment. In addition, the learning device 1 can generate the third sample 551 to the second sample 521 using the trained decoder 411D and the estimator 44 by the same processing flow as the above embodiment. Note that the second learning model 42 and the third learning model 43 can also be respectively constructed like the first learning model 41D of the present modified example.

[0481] <4.4>

[0482] In the above-described embodiment, the third learning model 43 can be omitted. In this case, the third acquisition section 113 and the third training section 117 can be omitted from the software configuration of the learning device 1. In the processing flow of the learning device 1, the processing of steps S121 to S123 can be omitted. In the processing flow of the data generation device 2, the processing of step S204 can be omitted. The control section 21 can appropriately synthesize the first pseudo sample 61 and the second pseudo sample 62 in step S205. The control section 21 can synthesize the first pseudo sample 61 and the second pseudo sample 62 by, for example, directly overlapping and adding the first pseudo sample 61 and the second pseudo sample 62. Alternatively, the parameters of the synthesis can be appropriately designated by input of an operator or the like. The control section 21 can synthesize the first pseudo sample 61 and the second pseudo sample 62 on the basis of the designated parameters.

[0483] <4.5>

[0484] In the above-described embodiment, the machine learning of the predictor 44 can be performed by another computer other than the learning device 1. Alternatively, the second sample 521 can be generated by data processing or the like using an operation of an operator in a method other than the above-described embodiment. For example, the control section 11 can generate the second sample 521 by calculating a difference between the third sample 551 and a sample including only the first feature provided in advance. In these cases, the fourth acquisition section 114 and the fourth training section 118 can be omitted from the software configuration of the learning device 1. In the processing flow of the learning device 1, the processing of steps S131 to S133 and steps S141 to S144 can be omitted.

[0485] <4.6>

[0486] The inference device 3 of the above-described embodiment utilizes the first generator 411 and the predictor 44 completed with training in the inference processing related to the second feature in step S302. However, the inference processing related to the second feature can not be limited to such an example. For example, a sample including the first feature corresponding to the object sample 325 can be provided in advance. The inference device 3 can use this pre-provided sample instead of the pseudo sample 326. In addition, for example, the inference device 3 can generate a sample including the first feature corresponding to the object sample 325 from the object sample 325 by another model generated by a partial space method or the like. Note that, as exemplified in these, in a case where the first generator 411 and the predictor 44 completed with training are not utilized in the inference processing, the first learning result data 121 and the fourth learning result data 124 can be omitted from the information held by the inference device 3.

[0487] Furthermore, in the above embodiment, the inference processing related to the second feature is implemented through data parsing of the difference sample 327. The new sample 65 can be reflected in the parameters of this data parsing. However, the way in which the new sample 65 is reflected in the parameters of the inference processing is not limited to this example. As another example, the inference processing related to the second feature can use a fully trained machine learning model that has acquired the ability to perform the inference processing. The new sample 65 can be reflected in the values ​​of the operational parameters of the fully trained machine learning model through machine learning used for this machine learning model. In this method, the operational parameters of the fully trained machine learning model are equivalent to the parameters in the inference processing.

[0488] Figure 29 This example illustrates an application scenario of the inference system 100E in this modified embodiment. This modified embodiment is an example of a method for using a trained machine learning model for inference processing. The inference system 100E in this modified embodiment includes a learning device 1, a data generation device 2, an inference device 3E, and other learning devices 7. The inference device 3E corresponds to the inference device 3 in the above embodiment. The other learning devices 7 are computers configured to generate trained machine learning models that can be used for inference processing related to a second feature. Similar to the above embodiment, the learning device 1, data generation device 2, inference device 3E, and other learning devices 7 are interconnected via a network. Furthermore, at least any pair of the learning device 1, data generation device 2, inference device 3E, and other learning devices 7 constitutes a single computer.

[0489] [Other learning devices]

[0490] (Hardware Configuration)

[0491] Figure 29 This is an illustrative example of the hardware configuration of another learning device 7 in this variation. For example... Figure 31 As shown, the other learning device 7 in this modification is composed of a computer electrically connected to a control unit 71, a storage unit 72, a communication interface 73, an external interface 74, an input device 75, an output device 76, and a driver 77. The control unit 71 to the driver 77 and the storage medium 97 of the other learning device 7 can be configured in the same way as the control unit 11 to the driver 17 and the storage medium 91 of the learning device 1 described above.

[0492] In this variant, the storage unit 72 stores various information such as the learning program 87, multiple learning datasets 721, and learning result data 725. The learning program 87 is used to enable other learning devices 7 to perform information processing related to the generation of a trained machine learning model that can be used for inference processing (described later). Figure 30The learning program 87 includes a series of instructions of the information processing. The learning program 87 can be referred to as a "model generation program" or simply a "generation program" as with the learning program 81 described above. The plurality of learning data sets 721 are used for machine learning of the machine learning model. The learning result data 725 represents information related to the trained machine learning model constructed by the machine learning. The learning result data 725 is generated as a result of execution of the learning program 87. At least any one of the learning program 87 and the plurality of learning data sets 721 can be stored in the storage medium 97. The other learning device 7 can acquire at least any one of the learning program 87 and the plurality of learning data sets 721 from the storage medium 97.

[0493] Note that the specific hardware configuration of the other learning device 7 can omit, replace, and add components as appropriate depending on the embodiment. For example, the control section 71 can include a plurality of hardware processors. The hardware processor can be constituted by a microprocessor, an FPGA, a DSP, or the like. The storage section 72 can be constituted by a RAM and a ROM included in the control section 71. At least any one of the communication interface 73, the external interface 74, the input device 75, the output device 76, and the driver 77 can be omitted. The other learning device 7 can be constituted by a plurality of computers. In this case, the hardware configurations of the respective computers can or can not be identical. The other learning device 7 can be an information processing device designed specifically for a service provided, as well as a general-purpose server device, a general-purpose PC, or the like.

[0494] (Software Configuration)

[0495] Figure 31 An example of the software configuration of the other learning device 7 of the illustrative example of the present modification. As with the learning device 1 and the like described above, the learning program 87 is executed by the control section 71. Thereby, the other learning device 7 functions as a computer that has the acquisition section 711, the training section 712, and the saving processing section 713 as software modules. That is, in the present modification, each software module of the other learning device 7 is realized by the control section 71 (CPU).

[0496] The acquisition section 711 acquires the plurality of learning data sets 721 each constituted by a combination of the training sample 7211 and the correct answer data 7212. The training sample 7211 includes the first feature. On the other hand, the training sample 7211 can be any one of a sample including the second feature and a sample not including the second feature. The new sample 65 can be used as the training sample 7211. The correct answer data 7212 represents a result (correct answer) obtained by performing a prescribed inference related to the second feature on the training sample 7211.

[0497] The training section 712 performs machine learning of the machine learning model 45 using the acquired plurality of learning data sets 721. In the machine learning, the training section 712 trains the machine learning model 45 to output an output value that matches the correct answer data 7212 after inputting the training sample 7211 for each learning data set 721. Thereby, it is possible to generate the trained machine learning model 45 that has acquired the ability to perform the inference processing related to the second feature. The saving processing section 713 generates information related to the trained machine learning model 45 as learning result data 725, and saves the generated learning result data 725 to a prescribed storage area.

[0498] The kind of the machine learning model 45 can not be particularly limited, and can be appropriately selected according to the embodiment. The method of the machine learning can be selected according to the kind of the machine learning model 45. The machine learning model 45 can be configured by, for example, a neural network, similarly to the above-described first generator 411 and the like. Hereinafter, for convenience of explanation, it is assumed that the machine learning model 45 is configured by a neural network.

[0499] Note that, in the present modification example, similarly to the above-described learning device 1 and the like, an example in which each software module of the other learning device 7 is implemented by a general-purpose CPU is described. However, part or all of the above-described software modules can also be implemented by one or a plurality of dedicated processors. In addition, regarding the software configuration of the other learning device 7, the software modules can be appropriately omitted, replaced, and added according to the embodiment.

[0500] (Action Example)

[0501] Figure 2A is a flowchart showing an example of a processing flow related to machine learning of the machine learning model 45 by the other learning device 7 of the present modification example. Note that the processing flow described below is merely an example, and each step can be changed as long as it is permitted. Also, regarding each processing flow described below, the steps can be appropriately omitted, replaced, and added according to the embodiment.

[0502] In step S701, the control section 71 functions as an acquisition section 711 and acquires a plurality of learning data sets 721. Each learning data set 721 can be generated as appropriate. As an example, a training sample 7211 is acquired as appropriate. The training sample 7211 can be acquired by the same method as the first sample 511 or the third sample 551 described above. Next, a prescribed inference related to the second feature is performed on the acquired training sample 7211. In this context, the prescribed inference can be performed manually by an operator or the like. Also, a result (correct answer) after the prescribed inference is performed on the training sample 7211 is associated with the training sample 7211. Thus, each learning data set 721 can be generated. Each learning data set 721 can be automatically generated by the operation of a computer, as with the first learning result data 51 described above, or can be manually generated by the operation of an operator. In addition, each learning data set 721 can be performed by the other learning device 7, or can be performed by another computer other than the other learning device 7. The control section 71 acquires each learning data set 721 generated.

[0503] In addition, the new sample 65 can be used as the training sample 7211. For example, in the case where the number of learning data sets 721 is insuff...

Claims

1. A data generation system comprising: a first acquisition unit configured to acquire a plurality of first learning data sets each composed of a combination of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data; a second acquisition unit configured to acquire a plurality of second learning data sets each composed of a combination of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data; a first training unit configured to implement machine learning of a first learning model including a first generator using the acquired plurality of first learning data sets, the machine learning of the first learning model including training the first generator for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample; a second training unit configured to implement machine learning of a second learning model including a second generator using the acquired plurality of second learning data sets, the machine learning of the second learning model including training the second generator for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample; a first generation unit configured to generate a first pseudo sample including the first feature by providing a first input value corresponding to the first label to the trained first generator; a second generation unit configured to generate a second pseudo sample including the second feature by providing a second input value corresponding to the second label to the trained second generator; and a data synthesis unit configured to generate a new sample of the prescribed kind of data by synthesizing the generated second pseudo sample with the generated first pseudo sample, a third acquisition unit configured to acquire a plurality of third learning data sets each composed of a combination of a sample value of an attribute of the second feature, a reflection level indicating a degree of reflection of the second feature, and the second label; and a third training unit configured to implement machine learning of a third learning model including a third generator using the acquired plurality of third learning data sets, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value, wherein the prescribed kind of data includes a first component and a second component different from the first component, the second component being an object of a prescribed speculation, the first feature is related to the first component, the second feature is related to the second component, the prescribed speculation refers to detection of the second component, and the synthesis includes: generating the pseudo value of the attribute of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level, i.e., a third input value provided in accordance with the detected limit, to the trained third generator. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ transforming the second pseudo sample so as to match a value of an attribute of the second feature contained in the second pseudo sample with the generated pseudo value, and synthesizing the second pseudo sample to the first pseudo sample, the data is image data of a product taken by a camera, the first label represents a category of a product, the second label represents a category of a defect, and the new sample represents a product and a defect of a specified category, the new sample is used as a training sample to train a machine learning model for detecting a defect in the image data.

2. The data generation system according to claim 1, wherein the first learning model further includes a first discriminator, training the first generator so as to generate a pseudo sample that reduces performance of the identification by the first discriminator is constituted by alternately repeating the following steps: training the first discriminator to identify whether an input sample input to the first discriminator is the first sample obtained from any one of the plurality of first learning data sets or a pseudo sample generated by the first generator; and training the first generator so as to generate a pseudo sample that reduces performance of the identification by the first discriminator.

3. The data generation system according to claim 1, wherein the second learning model further includes a second discriminator, training the second generator so as to generate a pseudo sample that reduces performance of the identification by the second discriminator is constituted by alternately repeating the following steps: training the second discriminator to identify whether an input sample input to the second discriminator is the second sample obtained from any one of the plurality of second learning data sets or a pseudo sample generated by the second generator; and training the second generator so as to generate a pseudo sample that reduces performance of the identification by the second discriminator.

4. The data generation system according to claim 1, wherein the third learning model further includes a third discriminator, training the third generator so as to generate a pseudo value of an attribute of the second feature that reduces performance of the identification by the third discriminator is constituted by alternately repeating the following steps: training the third discriminator to identify whether an input value input to the third discriminator is the sample value obtained from any one of the plurality of third learning data sets or a pseudo value generated by the third generator; and training the third generator so as to generate a pseudo value of an attribute of the second feature that reduces performance of the identification by the third discriminator.

5. The data generation system according to claim 1, wherein each of the third learning data sets further has the first label, the machine learning that implements the third learning model includes: training the third generator so as to match a pseudo value of an attribute of the second feature generated by the third generator from the reflection level, the second label, and the first label with the sample value.

6. The data generation system according to claim 1, wherein the reflection level is constituted by a continuous value.

7. The data generation system according to claim 1, wherein the acquiring of the plurality of second learning data sets includes the following step: acquiring a plurality of third samples each including the first feature and the second feature, the third samples being third samples of the specified category of data; the predictor is trained to estimate the input provided to the first generator when generating the pseudo sample from the pseudo sample generated by the first generator trained; the third pseudo sample is generated for each of the third samples by providing the estimated input to the first generator trained; and the second sample of each of the second learning data sets is generated by subtracting the third pseudo sample from the third sample.

8. The data generating system of claim 7, wherein, Further provided are: a fourth acquisition unit that acquires a plurality of fourth learning data sets each composed of a sample value of the input provided to the first generator trained and a fourth pseudo sample generated by providing the sample value to the first generator trained; a fourth training unit that constructs the predictor trained by performing machine learning of the predictor using the acquired plurality of fourth learning data sets, the machine learning of the predictor including training the predictor for each of the fourth learning data sets so that an estimated value obtained by estimating the input provided to the first generator trained from the fourth pseudo sample using the predictor matches the sample value.

9. The data generation system according to claim 1, wherein the prescribed kind of data is image data including a background and a foreground, the first feature is related to the background, the second feature is related to the foreground.

10. The data generation system according to claim 1, wherein the prescribed kind of data is image data representing a product, the first feature is related to a background including the product, the second feature is related to a defect of the product.

11. A data generation apparatus comprising: a first generation unit having a first generator trained by machine learning using a plurality of first learning data sets each composed of a first sample including a first feature and a first label representing a category of the first feature, the first sample being a first sample of a prescribed kind of data, by the machine learning, the first generator is trained for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, the first generation unit generates a first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first generator trained. a second generation unit configured to generate a second pseudo sample including a second feature by providing a second input value corresponding to a second label to a second generator trained by machine learning using a plurality of second learning data sets each composed of a second sample including the second feature different from the first feature and the second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, the second generator being trained for each of the second learning data sets by the machine learning so that a pseudo sample of the second feature generated by the second generator from the second label matches the second sample, and a data synthesis unit configured to generate a new sample of the prescribed kind of data by synthesizing the generated second pseudo sample to the generated first pseudo sample, a second acquisition unit configured to acquire a plurality of third learning data sets each composed of a sample value of an attribute of the second feature, a reflection level indicating a degree of reflection of the second feature, and the second label, and a second training unit configured to perform machine learning of a third learning model including a third generator using the acquired plurality of third learning data sets, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value, the prescribed kind of data including a first component and a second component different from the first component, the second component being an object of a prescribed speculation, the first feature being related to the first component, the second feature being related to the second component, the prescribed speculation being detection of the second component, the synthesis including: generating the pseudo value of the attribute of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level, i.e., a third input value provided in accordance with a limit of the detection, to the third generator trained, and converting the second pseudo sample so that a value of the attribute of the second feature contained in the second pseudo sample matches the generated pseudo value, and synthesizing the second pseudo sample to the first pseudo sample, the data being image data of a product captured by a camera, the first label indicating a category of a product, the second label indicating a category of a defect, the new sample indicating a product and a specified category of a defect, the new sample being used as a training sample to train a machine learning model for detecting a defect in the image data.

12. A data generation method, a computer executing the following steps: generating a first pseudo sample including a first feature using a first generator trained, The first generator trained is constructed by machine learning using a plurality of first learning data sets each composed of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data, In the machine learning, the first generator is trained for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, The computer generates the first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first generator trained, A second pseudo sample including a second feature is generated using a second generator trained, The second generator trained is constructed by machine learning using a plurality of second learning data sets each composed of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, In the machine learning, the second generator is trained for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample, The computer generates the second pseudo sample including the second feature by providing a second input value corresponding to the second label to the second generator trained, and A new sample of the prescribed kind of data is generated by synthesizing the generated second pseudo sample to the generated first pseudo sample, A plurality of third learning data sets each composed of a sample value of an attribute of the second feature, a reflection level indicating a degree of reflection of the second feature, and the second label are acquired; and Machine learning of a third learning model including a third generator is implemented using the acquired plurality of third learning data sets, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value, The prescribed kind of data includes a first component and a second component different from the first component, the second component being an object of a prescribed speculation, The first feature is related to the first component, The second feature is related to the second component, The prescribed speculation refers to detection of the second component, The synthesis includes: generating a pseudo value of the attribute of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level, i.e., a third input value provided in accordance with a limit of the detection, to the third generator trained; and converting the second pseudo sample so that a value of the attribute of the second feature contained in the second pseudo sample matches the generated pseudo value, and synthesizing the second pseudo sample to the first pseudo sample, The data is image data of a product photographed by a camera, the first label indicates a category of the product, the second label indicates a category of a defect, and the new sample indicates a product and a defect of a specified category, The new sample is used as a training sample to train a machine learning model for detecting defects in the image data.

13. A program product including a data generation program for causing a computer to execute the following steps: generating, by a first trained generator, a first pseudo sample including a first feature, The first trained generator is constructed by machine learning using a plurality of first learning data sets each composed of a first sample including a first feature and a first label indicating a category of the first feature, the first sample being a first sample of a prescribed kind of data, In the machine learning, the first generator is trained for each of the first learning data sets so that a pseudo sample generated by the first generator from the first label matches the first sample, The computer generates the first pseudo sample including the first feature by providing a first input value corresponding to the first label to the first trained generator; generating, by a second trained generator, a second pseudo sample including a second feature, The second trained generator is constructed by machine learning using a plurality of second learning data sets each composed of a second sample including a second feature different from the first feature and a second label indicating a category of the second feature, the second sample being a second sample of the prescribed kind of data, In the machine learning, the second generator is trained for each of the second learning data sets so that a pseudo sample generated by the second generator from the second label matches the second sample, The computer generates the second pseudo sample including the second feature by providing a second input value corresponding to the second label to the second trained generator, and generates a new sample of the prescribed kind of data by synthesizing the generated second pseudo sample with the generated first pseudo sample, acquiring a plurality of third learning data sets each composed of a sample value of an attribute of the second feature, a reflection level indicating a degree of reflection of the second feature, and the second label, and using the acquired plurality of third learning data sets, implementing machine learning of a third learning model including a third generator, the machine learning of the third learning model including training the third generator for each of the third learning data sets so that a pseudo value of the attribute of the second feature generated by the third generator from the reflection level and the second label matches the sample value, The prescribed kind of data includes a first component and a second component different from the first component, the second component being an object of a prescribed speculation, The first feature is related to the first component, The second feature is related to the second component, The prescribed speculation refers to detecting the second component, The synthesizing includes: generating pseudo values of attributes of the second feature by providing a second input value corresponding to the second label and a third input value corresponding to the reflection level, i.e., the third input value provided according to the detected limit, to the third generator which is completed training; and transforming the second pseudo sample so that the values of the attributes of the second feature contained in the second pseudo sample match the generated pseudo values, and synthesizing the second pseudo sample to the first pseudo sample, The data is image data of a product taken by a camera, the first label represents a category of a product, the second label represents a category of a defect, and the new sample represents a product and a specified category of a defect. The new sample is used as a training sample to train a machine learning model for detecting defects in the image data.

Images