Label generation method, label generation device, and label generation program

Abstract

To generate a dataset for machine learning including a highly reliable correct label, at a low cost.SOLUTION: A label generation method includes: preparing a first inference model trained on a first dataset obtained from a source domain, and a second inference model trained on a second dataset including second training data generated by adding a disturbance to first training data; and generating a third correct label for third training data, on the basis of a consensus of the prepared trained first inference model and second inference model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a label generation method, a model generation method, a label generation device, a label generation program, a model generation device, and a model generation program. [Background technology]

[0002] Conventionally, in product manufacturing situations such as on a production line, a technology has been used in which a photograph of a manufactured product is taken with an imaging device and the quality of the product is inspected based on the obtained image data. For example, Patent Document 1 proposes an appearance inspection device that performs an appearance inspection of an inspection object by photographing the appearance of the inspection object irradiated with inspection light from a light source while moving the light source with an articulated robot and analyzing the obtained images. Also, for example, Patent Document 2 proposes an inspection device that determines whether an inspection object shown in an image is normal or abnormal based on a trained first neural network, and if the inspection object is determined to be abnormal, classifies the type of the abnormality based on a trained second neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-045330 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-026982 [Non-patent literature]

[0004] [Non-Patent Document 1] Dong-hyun Lee, "Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks", [online], [Retrieved July 7, 2021], Internet <url: http: citeseerx.ist.psu.edu viewdoc summary?doi="10.1.1.664.3543"> Summary of the Invention [Problem to be solved by the invention]

[0005] According to conventional methods, it is possible to perform visual inspection of a product from an image obtained by photographing. Furthermore, according to a method using a machine learning model such as a neural network, as disclosed in Patent Document 2, it is possible to perform visual inspection of a product based on the output of a trained machine learning model, without manually specifying in detail the image processing of the obtained image. This simplifies the information processing for visual inspection and reduces the effort required to create an inspection program. However, the present inventors have discovered the following problems with the conventional inspection methods described above.

[0006] When visual inspection is performed based on captured images, image data showing products containing defects is collected in advance as training data. A correct answer label indicating the correct answer (true value) for the task of detecting defects shown in the training data is associated with the training data, and a learning dataset is generated from this. In the case of defect detection, the generated multiple datasets are used as learning data in machine learning. In other words, the generated multiple datasets are used to adjust the parameter values ​​for defect detection in machine learning. Basically, the more datasets collected, the more accurate the visual inspection can be.

[0007] However, the greater the number of defect types, product appearances, and background types, the greater the number of combinations, and the greater the effort required to collect a dataset. In particular, generating correct labels to assign to the collected training data is time-consuming. To address this issue, a semi-supervised learning method, such as that described in Non-Patent Document 1, can be adopted. In the method proposed in Non-Patent Document 1, a small dataset is generated by assigning correct labels to a portion of the acquired training data (here, the domain from which the training data is acquired is referred to as the "source domain"). This small dataset is then used to perform machine learning, thereby generating a provisional trained machine learning model. The generated trained machine learning model is then used to obtain inference results for the remaining training data that has not been assigned correct labels. A pseudo dataset is generated by associating labels indicating the obtained inference results with the training data as pseudo labels (pseudo correct labels). The generated pseudo dataset is then used to further perform machine learning on the machine learning model. This automates the process of generating correct labels to assign to training data for at least a portion of the dataset, thereby increasing the number of datasets available for machine learning. This allows for improved inference accuracy of the generated trained machine learning model while reducing the cost of collecting datasets.

[0008] The above-mentioned problem can also occur in the context of domain adaptation. That is, the source domain from which training data for a training dataset is obtained may differ from the target domain from which target data for performing an inference task using a trained machine learning model is obtained. For example, the source domain and the target domain may differ if the conveyor belt patterns (backgrounds) that transport products are different between the training data and the target data, if the performance of the cameras used to photograph the products is different, or if the camera installation angles are different. In such cases, if a trained machine learning model is generated using only the dataset obtained in the source domain, the inference accuracy of the generated trained machine learning model for the target data may be low due to the influence of the domain differences. Therefore, in order to improve the inference accuracy of the trained machine learning model, it is possible to obtain a training dataset in the target domain and perform machine learning (e.g., additional learning, re-learning, generation of a new trained model, etc.) using the obtained dataset.

[0009] However, collecting a dataset in the target domain is time-consuming. In particular, generating correct labels to be assigned to training data obtained in the target domain is time-consuming. Therefore, as a method for solving this problem, similar to the above method, a trained machine learning model generated by machine learning using a dataset obtained in the source domain can be used to generate pseudo labels to be assigned to training data obtained in the target domain. This automates the task of generating correct labels to be assigned to training data for at least a portion of the dataset in the target domain, thereby increasing the number of datasets in the target domain used for machine learning. This improves the inference accuracy of trained machine learning for target data obtained in the target domain and reduces the cost of collecting datasets in the target domain.

[0010] As described above, the method of generating pseudo-ground truth labels using a trained machine learning model generated using a partial dataset, such as that described in Non-Patent Document 1, can reduce the cost of collecting datasets in semi-supervised learning and domain adaptation. It can also improve the inference accuracy of the trained machine learning model. However, the present inventors have discovered the following problem with this method. That is, in both the semi-supervised learning and domain adaptation scenarios, the inference accuracy of the trained machine learning model generated using a partial dataset is not high. As a result, the reliability of the pseudo-ground truth labels generated by the trained machine learning model may be low (i.e., the number of training data items assigned with erroneous ground truth labels may increase). If unreliable pseudo-labels are assigned, the inference accuracy of the trained machine learning model that is ultimately generated may actually be worse.

[0011] Note that this problem is not specific to situations in which a trained machine learning model that can be used for visual inspection is generated. Furthermore, this problem is not specific to situations in which image data is used as training data. The training data may be composed of, for example, image data, sound data, numerical data, text data, or other sensing data obtained by a sensor. The inference task may be composed of, for example, extracting a region containing a feature in the data, identifying a class of a feature contained in the data, or the like. For example, a similar problem may arise in any situation in which a trained machine learning model that has acquired the ability to perform an inference task on any data is generated, such as extracting a region containing a feature in image data (segmentation), identifying a class of a feature contained in image data, extracting a region containing a feature in sound data, identifying a class of a feature contained in sound data, extracting a region containing a feature in other sensing data, or identifying a class of a feature contained in sensing data.

[0012] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide a technology that can generate a machine learning dataset containing highly reliable correct answer labels at low cost, or a technology that uses the resulting dataset for machine learning to improve the performance of the generated trained model. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, the present invention employs the following configuration.

[0014] That is, a label generation method according to one aspect of the present invention is an information processing method in which a computer executes the following steps: acquiring a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct label indicating a correct answer to an inference task for the first training data; acquiring a trained second inference model generated by machine learning using a plurality of second datasets, each of which is composed of a combination of second training data generated by adding disturbances to the first training data and a second correct label indicating a correct answer to the inference task for the second training data; acquiring third training data; using the trained first inference model to acquire a first inference result by performing the inference task on the acquired third training data; using the trained second inference model to acquire a second inference result by performing the inference task on the acquired third training data; and generating a third correct label for the third training data based on a match between the first inference result and the second inference result.

[0015] In this configuration, a different dataset is prepared by adding external disturbances to the first training data. Since the addition of external disturbances can be automated, different datasets can be generated easily and at low cost. By preparing a first inference model and a second inference model trained on different datasets, it is possible to obtain inference results by performing an inference task on the training data (third training data) from different perspectives. Then, by generating a correct answer label (i.e., deriving a correct answer) based on a match between the inference results obtained from the different perspectives, it is possible to increase the likelihood of obtaining an appropriate correct answer from features common to the different perspectives (i.e., data features truly relevant to the inference task). As a result, a highly reliable correct answer label (third correct answer label) can be generated. Furthermore, at least a portion of the process of generating the correct answer label can be automated. Therefore, this configuration makes it possible to generate a dataset for machine learning containing a highly reliable correct answer label at low cost. The third training data may be obtained from a source domain or a target domain different from the source domain.

[0016] In the label generation method according to the above aspect, the third training data may be acquired in a target domain different from the source domain. In this configuration, by using the first inference model and the second inference model trained to perform inference tasks from different perspectives, it is possible to increase the likelihood of obtaining an appropriate correct answer from common features without being affected by differences in domains. Therefore, when assigning correct labels to training data (third training data) obtained in a target domain different from the source domain, it is possible to generate a machine learning dataset containing highly reliable correct labels at low cost.

[0017] In the label generation method according to the above aspect, adding disturbances to the first training data may be configured by transforming the first training data using a trained transformation model. The trained transformation model may be generated by machine learning using the first training data and the third training data, so as to acquire the ability to transform the style of the first training data into the style of the third training data. In this configuration, a trained second inference model is prepared by using second training data having a style that matches the style of the third training data for machine learning. The trained second inference model has the ability to solve an inference task on the style of the third training data. By using this trained second inference model together with the trained first inference model to generate correct labels, it is possible to increase the likelihood of obtaining an appropriate correct answer to the inference task for training data (third training data) obtained in the target domain. Therefore, this configuration makes it possible to generate a dataset for machine learning that includes more reliable correct labels.

[0018] In the label generation method according to the above aspect, the first inference model and the second inference model may be further trained by adversarial learning between the first inference model and a discriminative model. The adversarial learning may include: training the discriminative model using the first training data and the third training data to identify whether an inference result of the first inference model is related to the first training data or the third training data; training the first inference model using the first training data and the third training data to reduce the discriminative performance of the discriminative model; training the discriminative model using the second training data and the third training data to identify whether an inference result of the second inference model is related to the second training data or the third training data; and training the second inference model using the second training data and the third training data to reduce the discriminative performance of the discriminative model. In this configuration, the adversarial learning may be performed to prepare the first inference model and the second inference model trained to capture features common to the source domain, the state in which the disturbance is applied, and the target domain. As a result of this adversarial learning, each inference model can be made less susceptible to the effects of gaps between the source domain, external disturbances, and the target domain when solving an inference task. Therefore, this configuration provides a trained inference model with high inference accuracy, and as a result, it is possible to generate a machine learning dataset that includes more reliable ground truth labels.

[0019] In the label generation method according to the above aspect, the computer may further execute a step of outputting the generated third correct label. With this configuration, an operator can confirm the automatically generated pseudo-correct label (third correct label) based on the output. As a result, the operator can correct or delete the correct label indicating an incorrect correct answer.

[0020] In the label generation method according to the above aspect, the inference task may be to extract a region containing a feature, and generating the third correct label based on the match may be configured to identify an overlapping portion between the region extracted as the first inference result and the region extracted as the second inference result, and, if the size of the identified overlapping portion exceeds a threshold, generate the third correct label to indicate the overlapping portion as a correct answer to the inference task. With this configuration, when generating a trained machine learning model capable of extracting regions containing features, it is possible to generate a machine learning dataset containing highly reliable correct labels at low cost.

[0021] In the label generation method according to the above aspect, the inference task may be identifying a class of a feature included in the data, and generating the third correct label based on the match may be configured to generate the third correct label to indicate a matched class when the class identified as the first inference result and the class identified as the second inference result match. This configuration allows for low-cost generation of a machine learning dataset including highly reliable correct labels when generating a trained machine learning model capable of identifying the class of a feature included in data. Extracting the region including the feature and identifying the class of the feature may be performed simultaneously.

[0022] In the label generation method according to the above aspect, each of the training data may be composed of image data, and the inference task may be composed of at least one of extracting a region containing a feature in the image data and identifying a class of the feature contained in the image data. With this configuration, when generating a trained machine learning model that has acquired the ability to perform an inference task on image data, it is possible to generate a machine learning dataset containing highly reliable correct labels at low cost.

[0023] In the label generation method according to the above aspect, each of the training data may be image data, and the inference task may be performed by extracting a region containing a feature from the image data. The first inference model and the second inference model may be further trained by adversarial learning between the first inference model and a discriminative model. The adversarial learning may include: training the discriminative model using the first training data and the third training data to distinguish, for each pixel, whether an inference result of the first inference model is related to the first training data or the third training data; training the first inference model using the first training data and the third training data to reduce the discriminative performance of the discriminative model; training the discriminative model using the second training data and the third training data to distinguish, for each pixel, whether an inference result of the second inference model is related to the second training data or the third training data; and training the second inference model using the second training data and the third training data to reduce the discriminative performance of the discriminative model.

[0024] If a discriminative model is configured to identify features by capturing overall features, adversarial learning may result in the discriminative model acquiring the ability to identify features based on differences in label distributions. Once the discriminative model has acquired such an ability, each inference model may be trained by adversarial learning to reduce the discriminative model's discrimination performance, potentially acquiring the ability to extract features that eliminate differences in label distributions. As a result, the inference accuracy of each inference model may be reduced. In contrast, with this configuration, by configuring the discriminative model to identify each pixel, the discrimination of the discriminative model can be made independent of differences in label distributions, thereby preventing a decrease in the inference accuracy of each inference model. As a result, a first inference model and a second inference model with high inference accuracy can be prepared, and by using them, highly reliable correct labels can be generated.

[0025] In the label generation method according to the above aspect, the training data may be composed of sound data, and the inference task may be composed of at least one of extracting a region containing a feature in the sound data and identifying a class of the feature contained in the sound data. With this configuration, when generating a trained machine learning model that has acquired the ability to perform an inference task on sound data, it is possible to generate a machine learning dataset containing highly reliable correct labels at low cost.

[0026] In the label generation method according to the above aspect, the training data may be composed of sensing data, and the inference task may be composed of at least one of extracting a region containing a feature in the sensing data and identifying a class of the feature contained in the sensing data. With this configuration, when generating a trained machine learning model that has acquired the ability to perform an inference task on the sensing data, it is possible to generate a machine learning dataset containing highly reliable correct labels at low cost.

[0027] The present invention is not limited to a label generation method configured to execute the above series of processes by a computer. One aspect of the present invention may be a model generation method configured to generate a trained machine learning model using correct labels generated by the label generation method according to any of the above aspects. Furthermore, another aspect of the present invention may be an inference method configured to perform an inference task using the generated trained machine learning model.

[0028] For example, a model generation method according to one aspect of the present invention is an information processing method that includes the steps of: acquiring, by a computer, multiple third datasets each generated by associating the third correct label generated by the label generation method according to any of the above aspects with the third training data; and performing machine learning of a third inference model using the acquired multiple third datasets, the machine learning being configured to train the third inference model so that, for each of the third datasets, an inference result obtained by performing the inference task using the third inference model on the third training data matches the correct answer indicated by the third correct label. With this configuration, the performance of the generated trained model can be improved by using a dataset including highly reliable correct labels for machine learning.

[0029] Furthermore, as another form of each information processing method according to each of the above-mentioned forms, one aspect of the present invention may be an information processing device that realizes all or part of each of the above-mentioned configurations, an information processing system, a program, or a storage medium that stores such a program and is readable by a computer or other device, machine, etc. Here, a storage medium that is readable by a computer, etc. is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0030] For example, a label generation device according to one aspect of the present invention includes: a first model acquisition unit configured to acquire a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct label indicating a correct answer to an inference task for the first training data; a second model acquisition unit configured to acquire a trained second inference model generated by machine learning using a plurality of second datasets, each of which is composed of a combination of second training data generated by adding disturbances to the first training data and a second correct label indicating a correct answer to the inference task for the second training data; a data acquisition unit configured to acquire third training data; a first inference unit configured to acquire a first inference result by performing the inference task on the acquired third training data using the trained first inference model; a second inference unit configured to acquire a second inference result by performing the inference task on the acquired third training data using the trained second inference model; and a generation unit configured to generate a third correct label for the third training data based on a match between the first inference result and the second inference result.

[0031] Furthermore, for example, a label generation program according to one aspect of the present invention is a program that causes a computer to execute the following steps: acquiring a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct label indicating a correct answer to an inference task for the first training data; acquiring a trained second inference model generated by machine learning using a plurality of second datasets, each of which is composed of a combination of second training data generated by adding disturbances to the first training data and a second correct label indicating a correct answer to the inference task for the second training data; acquiring third training data; using the trained first inference model to acquire a first inference result by performing the inference task on the acquired third training data; using the trained second inference model to acquire a second inference result by performing the inference task on the acquired third training data; and generating a third correct label for the third training data based on a match between the first inference result and the second inference result.

[0032] Also, for example, a model generation device according to one aspect of the present invention includes a data acquisition unit configured to acquire multiple third datasets each generated by associating the third correct label generated by the label generation method according to any of the above-mentioned embodiments with the third training data, and a learning processing unit configured to perform machine learning of a third inference model using the acquired multiple third datasets, wherein the machine learning is configured by training the third inference model so that, for each of the third datasets, the inference result obtained by performing the inference task using the third inference model on the third training data matches the correct answer indicated by the third correct label.

[0033] Furthermore, for example, a model generation program according to one aspect of the present invention is a program for causing a computer to execute the steps of: acquiring a plurality of third datasets, each of which is generated by associating the third correct label generated by the label generation method according to any of the above-described forms with the third training data; and performing machine learning of a third inference model using the acquired plurality of third datasets, wherein the machine learning is configured to train the third inference model so that, for each of the third datasets, the inference result obtained by performing the inference task using the third inference model on the third training data matches the correct answer indicated by the third correct label. [Effects of the Invention]

[0034] According to the present invention, it is possible to generate a machine learning dataset containing highly reliable ground truth labels at low cost, and by using the resulting dataset for machine learning, it is possible to improve the performance of the generated trained model. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a label generating apparatus according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of a model generating device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of an inference device according to an embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the software configuration of the label generating device according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of a method for adding a disturbance according to an embodiment. [Figure 7A] FIG. 7A schematically shows an example of the configuration of a first inference model and the machine learning process according to an embodiment. [Figure 7B] FIG. 7B schematically shows an example of the configuration of a second inference model and the machine learning process according to the embodiment. [Figure 7C] FIG. 7C schematically illustrates an example of the process of adversarial learning between a first inference model and a discriminative model according to an embodiment. [Figure 7D] FIG. 7D schematically illustrates an example of the process of adversarial learning between the second inference model and the discriminative model according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a software configuration of the model generating device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of the software configuration of the inference device according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of a processing procedure of the label generating device according to the embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a processing procedure of the model generating device according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a processing procedure of the inference device according to the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of another situation to which the present invention is applied. [Figure 14] FIG. 14 is a schematic diagram showing an example of another situation to which the present invention is applied. [Figure 15] FIG. 15 shows a schematic example of another situation to which the present invention is applied. [Figure 16] FIG. 16 shows the calculation results of the average IoU in the second experiment. DETAILED DESCRIPTION OF THE INVENTION

[0036] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0037] §1 Application Examples 1 is a diagram illustrating an example of a situation in which the present invention is applied. As shown in FIG. 1, an inference system 100 according to this embodiment includes a label generation device 1, a model generation device 2, and an inference device 3.

[0038] (Label generation device) The label generation device 1 according to this embodiment is a computer configured to generate a correct label for training data to which no correct label has been assigned using a trained inference model. Specifically, the label generation device 1 acquires a trained first inference model 61 generated by machine learning using a plurality of first data sets 51, each of which is composed of a combination of first training data 511 in the source domain and a first correct label 513 indicating a correct answer (true value) of an inference task for the first training data 511. The label generation device 1 also acquires a trained second inference model 62 generated by machine learning using a plurality of second data sets 52, each of which is composed of a combination of second training data 521 generated by adding a disturbance to the first training data 511 and a second correct label 523 indicating a correct answer (true value) of the inference task for the second training data 521. The label generation device 1 also acquires one or more pieces of third training data 531.

[0039] The source domain is a target domain from which first training data 511 is collected. The first training data 511 may be appropriately acquired from the source domain. A method for collecting the first training data 511 in the source domain is not particularly limited and may be appropriately selected depending on the embodiment. The first training data 511 may be, for example, image data, sound data, numerical data, text data, graph data (e.g., data showing chemical structures, graphs showing relationships between objects / people, etc.), or other measurement data (sensing data) obtained by various sensors. In one example, the first training data 511 may be generated by observing a real environment using sensors such as a camera, a microphone, an encoder, an environmental sensor, a vital sign sensor, a medical testing device, an in-vehicle sensor, or a home security sensor. The environmental sensor may be, for example, a barometer, a thermometer, a hygrometer, a sound pressure meter, a sound sensor, an ultraviolet sensor, a light meter, a rain gauge, a gas sensor, etc. The vital sensor may be, for example, a blood pressure monitor, a pulse rate monitor, a heart rate monitor, an electrocardiograph, an electromyograph, a thermometer, a skin electrodermal response monitor, a microwave sensor, an electroencephalograph, a magnetoencephalograph, an activity monitor, a blood glucose level monitor, an electrooculography sensor, an eye movement monitor, or the like. The medical examination device may be, for example, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, or the like. The in-vehicle sensor may be, for example, an image sensor, a light detection and ranging (Lidar) sensor, a millimeter-wave radar, an ultrasonic sensor, an acceleration sensor, or the like. The home security sensor may be, for example, an image sensor, an infrared sensor, an activity (audio) sensor, a gas (CO2, etc.) sensor, a current sensor, a smart meter (a sensor that measures the amount of power used by home appliances, lighting, etc.), or the like. In one example, the image data may be, for example, photographic data such as satellite photographs. In another example, the first training data 511 may be generated by information processing such as image generation processing, sound generation processing, simulation in a virtual environment, or the like.

[0040] The domain may be defined by the conditions for acquiring data, such as the sensor attributes, the observation target, the observation environment, the conditions of the virtual environment, and the generation conditions. The sensor attributes may include attributes related to observation capabilities, such as the measurement range, the resolution of the measurement range (e.g., resolution), and the sampling frequency. In one example, the observation target may be defined so as to distinguish between the individuality of objects (e.g., specific people). In another example, the observation target may be defined so as not to distinguish between the individuality of objects (treating similar objects the same). The observation environment may be defined by the attributes of the observed environment, such as the time of day, season (year), weather, location, and sensor installation angle. The time of day may be defined by expressions such as morning, noon, and night, or by precise time intervals, such as from 1:00 to 3:00. The weather may be defined by weather conditions, such as sunny, cloudy, rainy, and snowy. When the first training data 511 includes text data, the generation conditions may include, for example, conditions such as language, cultural area, generation, gender, purpose, writing style, posting platform (e.g., media such as social networking services, newspapers, distribution services, etc.), etc. A difference in domain may be constituted by a difference in at least one of these conditions (e.g., a difference to the extent that it affects the inference task).

[0041] The third training data 531 is composed of the same type of data as the first training data 511. In one example, when this embodiment is used in the semi-supervised learning scenario, the third training data 531 may be acquired in the same source domain as the first training data 511. In another example, when this embodiment is used in the domain adaptation scenario, the third training data 531 may be acquired in a target domain different from the source domain. The target domain is a target domain for generating a trained model and performing an inference task using the generated trained model. The target domain may differ from the source domain in at least one of the above conditions.

[0042] The label generation device 1 obtains a first inference result by performing an inference task on the acquired third training data 531 using the trained first inference model 61. The label generation device 1 also obtains a second inference result by performing an inference task on the acquired third training data 531 using the trained second inference model 62. The label generation device 1 then generates a third correct label 533 for the third training data 531 based on a match between the first inference result and the second inference result. That is, the label generation device 1 derives a correct answer to the inference task for the third training data 531 based on a consensus between the trained first inference model 61 and the trained second inference model 62, and generates a pseudo correct label (third correct label 533) configured to indicate the derived correct answer.

[0043] The content of the inference task is not particularly limited as long as it is capable of deriving the overlapping (matching) portion between the first inference result and the second inference result, and may be determined appropriately depending on the embodiment. For example, the inference task may be configured by at least one of extracting a region containing a feature in data and identifying a class of a feature contained in the data. As specific examples, the inference task may be, for example, extracting a region containing a feature in image data (segmentation), identifying a class of a feature contained in image data, extracting a region containing a feature in sound data, identifying a class of a feature contained in sound data, extracting a region containing a feature in other sensing data, or identifying a class of a feature contained in sensing data. As other specific examples, the inference task may be, for example, identifying attributes (e.g., harmfulness level, emotion, etc.) appearing in text, completing missing text, etc.

[0044] (Model generation device) The model generation device 2 according to this embodiment is a computer configured to generate a trained model using the third correct labels 533 generated by the label generation device 1. Specifically, the model generation device 2 acquires multiple third data sets 53, each generated by associating the third correct labels 533 generated by the label generation method with corresponding third training data 531. The model generation device 2 performs machine learning of a third inference model 63 using the acquired multiple third data sets 53. The machine learning of the third inference model 63 is performed by training the third inference model 63 so that, for each third data set 53, an inference result obtained by performing an inference task using the third training data 531 matches the correct answer indicated by the corresponding third correct label 533. As a result of performing this machine learning, a trained third inference model 63 can be generated. The generated trained third inference model 63 may be provided to the inference device 3 at any time.

[0045] (Inference device) The inference device 3 according to this embodiment is a computer configured to perform an inference task using a trained third inference model 63 generated by the model generation device 2. Specifically, the inference device 3 acquires target data 321 on which the inference task is to be performed. The target data 321 is the same type of data as the respective training data (511, 521, 531). The target data 321 may be acquired in any domain. When this embodiment is used in the above-described domain adaptation scenario, the target data 321 may be acquired in a target domain. The inference device 3 performs an inference task on the acquired target data 321 using the trained third inference model 63. As a result, the inference device 3 acquires an inference result obtained by performing the inference task on the target data 321. The inference device 3 outputs information related to the inference result.

[0046] (Features) As described above, in this embodiment, different datasets (the first dataset 51 and the second dataset 52) ​​are prepared for machine learning by adding disturbances to the first training data 511. Because the addition of disturbances can be automated, the second dataset 52 can be generated easily and at low cost from the first dataset 51. Furthermore, by preparing a trained first inference model 61 and a trained second inference model 62 derived from different datasets, it is possible to obtain inference results obtained by performing an inference task on the third training data 531 from different perspectives. Then, by generating the third correct answer label 533 based on a match between the inference results obtained from the different perspectives, it is possible to increase the likelihood of obtaining an appropriate correct answer from features common to the different perspectives (i.e., features of the data truly related to the inference task). As a result, it is possible to generate a highly reliable third correct answer label 533. In addition, it is possible to automate at least a part of the process of generating the third correct answer label 533. Therefore, the label generation device 1 according to this embodiment can generate a third dataset 53 for machine learning, including a highly reliable third correct answer label 533, at low cost. Furthermore, according to the model generation device 2 of this embodiment, by using the third dataset 53 including the highly reliable third correct label 533 for machine learning, it is possible to improve the inference performance of the generated trained third inference model 63. Furthermore, according to the inference device 3 of this embodiment, by using the trained third inference model 63 generated in this way, it is possible to expect the execution of highly accurate inference tasks for the target data 321.

[0047] 1, the label generation device 1, the model generation device 2, and the inference device 3 may be connected to one another via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between the label generation device 1, the model generation device 2, and the inference device 3 is not limited to this example and may be appropriately selected depending on the embodiment. In another example, data may be exchanged between the label generation device 1, the model generation device 2, and the inference device 3 using a storage medium.

[0048] In the example of FIG. 1 , the label generation device 1, the model generation device 2, and the inference device 3 are each separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. In another example, at least any pair of the label generation device 1, the model generation device 2, and the inference device 3 may be a single computer. As a specific example, the label generation device 1, the model generation device 2, and the inference device 3 may be configured as a single computer. The label generation device 1 and the model generation device 2 may be configured as a single computer. Alternatively, the model generation device 2 and the inference device 3 may be configured as a single computer. In yet another example, at least any one of the label generation device 1, the model generation device 2, and the inference device 3 may be configured as multiple computers.

[0049] §2 Configuration example [Hardware configuration] <Label generating device> Fig. 2 schematically illustrates an example of the hardware configuration of the label producing apparatus 1 according to this embodiment. As shown in Fig. 2, the label producing apparatus 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a drive 17 are electrically connected. In Fig. 2, the communication interface and the external interface are referred to as a "communication I / F" and an "external I / F." Similar notations are used in Figs. 3 and 4, which will be described later.

[0050] The control unit 11 includes a hardware processor such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 is an example of a memory resource, and is configured, for example, with a hard disk drive or a solid-state drive. In this embodiment, the storage unit 12 stores various information such as a label generation program 81, first learning result data 71, second learning result data 72, and third training data 531.

[0051] The label generation program 81 is a program for causing the label generation device 1 to execute information processing (see FIG. 10 described below) for generating a third correct label 533 for the third training data 531. The label generation program 81 includes a series of instructions for the information processing. The first learning result data 71 indicates information related to the trained first inference model 61. The second learning result data 72 indicates information related to the trained second inference model 62.

[0052] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for performing wired or wireless communication via a network. The label generating device 1 may perform data communication with another computer (for example, the model generating device 2) via the communication interface 13.

[0053] The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 14 may be selected arbitrarily. The label generation device 1 may be connected to a sensor for obtaining training data via the communication interface 13 or the external interface 14.

[0054] The input device 15 is a device for inputting, such as a mouse or keyboard. The output device 16 is a device for outputting, such as a display or speaker. An operator can operate the label generating device 1 by using the input device 15 and the output device 16. The input device 15 and the output device 16 may be integrated into one device, such as a touch panel display.

[0055] The drive 17 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading various information such as programs stored in a storage medium 91. At least one of the label generation program 81, the first learning result data 71, the second learning result data 72, and the third training data 531 may be stored in this storage medium 91.

[0056] The storage medium 91 is a medium that stores various types of information, such as programs, by electrical, magnetic, optical, mechanical, or chemical action so that the stored information, such as programs, can be read by a computer or other device, machine, etc. The label generation device 1 may acquire at least one of the label generation program 81, the first learning result data 71, the second learning result data 72, and the third training data 531 from the storage medium 91.

[0057] 2 illustrates a disk-type storage medium such as a CD or DVD as an example of the storage medium 91. However, the type of storage medium 91 is not limited to a disk type, and may be a type other than a disk. Examples of storage media other than a disk type include semiconductor memories such as flash memories. The type of drive 17 may be selected appropriately depending on the type of storage medium 91.

[0058] Note that, with regard to the specific hardware configuration of the label producing apparatus 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), etc. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the communication interface 13, the external interface 14, the input device 15, the output device 16, and the drive 17 may be omitted. The label producing apparatus 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the label producing apparatus 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), etc.

[0059] <Model generation device> Fig. 3 schematically illustrates an example of the hardware configuration of the model generation device 2 according to this embodiment. As shown in Fig. 3, the model generation device 2 according to this embodiment is a computer to which a control unit 21, a storage unit 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a drive 27 are electrically connected.

[0060] The control unit 21 to the drive 27 and the storage medium 92 of the model generating device 2 may be configured similarly to the control unit 11 to the drive 17 and the storage medium 91 of the label generating device 1, respectively. The control unit 21 includes a hardware processor such as a CPU, RAM, and ROM, and is configured to execute various information processes based on programs and data. The storage unit 22 is configured, for example, with a hard disk drive or a solid state drive. In this embodiment, the storage unit 22 stores various information such as a model generation program 82, a plurality of third data sets 53, and third learning result data 73.

[0061] The model generation program 82 is a program for causing the model generation device 2 to execute information processing (see FIG. 11 described below) related to machine learning to generate a trained third inference model 63. The model generation program 82 includes a series of instructions for the information processing. A plurality of third datasets 53 are used to generate the trained third inference model 63 (i.e., machine learning of the third inference model 63). The third learning result data 73 indicates information related to the generated trained third inference model 63. In this embodiment, the third learning result data 73 is generated as a result of executing the model generation program 82.

[0062] At least one of the model generation program 82 and the plurality of third data sets 53 may be stored in a storage medium 92. Furthermore, the model generation device 2 may acquire at least one of the model generation program 82 and the plurality of third data sets 53 from the storage medium 92. The third learning result data 73 may be stored in the storage medium 92.

[0063] The model generation device 2 may be connected to a device (sensor, another computer, external storage device, etc.) for acquiring the third data set 53 via at least one of the communication interface 23 and the external interface 24. The model generation device 2 may receive operations and inputs from an operator using the input device 25 and the output device 26.

[0064] Note that, with regard to the specific hardware configuration of the model generation device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, etc. The storage unit 22 may be configured with RAM and ROM included in the control unit 21. At least one of the communication interface 23, the external interface 24, the input device 25, the output device 26, and the drive 27 may be omitted. The model generation device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the model generation device 2 may be an information processing device designed specifically for the service to be provided, as well as a general-purpose server device, a general-purpose PC, etc.

[0065] <Inference device> Fig. 4 illustrates a schematic example of a hardware configuration of the inference device 3 according to this embodiment. As shown in Fig. 4, the inference device 3 according to this embodiment is a computer in which a control unit 31, a storage unit 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a drive 37 are electrically connected.

[0066] The control unit 31 to the drive 37 and the storage medium 93 of the inference device 3 may be configured similarly to the control unit 11 to the drive 17 and the storage medium 91 of the label generating device 1, respectively. The control unit 31 includes a hardware processor such as a CPU, RAM, and ROM, and is configured to execute various information processes based on programs and data. The storage unit 32 is configured, for example, with a hard disk drive or a solid state drive. In this embodiment, the storage unit 32 stores various information such as an inference program 83 and third learning result data 73.

[0067] The inference program 83 is a program for causing the inference device 3 to execute information processing (see FIG. 12 described below) that performs an inference task using the trained third inference model 63. The inference program 83 includes a series of instructions for the information processing. At least one of the inference program 83 and the third learning result data 73 may be stored in a storage medium 93. Furthermore, the inference device 3 may acquire at least one of the inference program 83 and the third learning result data 73 from the storage medium 93.

[0068] The inference device 3 may be connected to a device (sensor, other computer, external storage device, etc.) for acquiring target data 321 via at least one of a communication interface 33 and an external interface 34. The inference device 3 may receive operations and inputs from an operator using an input device 35 and an output device 36. Note that the operators of at least any pair of the label generation device 1, the model generation device 2, and the inference device 3 may be the same. Alternatively, the operators of each device 1 to 3 may not be the same.

[0069] Note that, with regard to the specific hardware configuration of the inference device 3, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 31 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, etc. The storage unit 32 may be configured with RAM and ROM included in the control unit 31. At least one of the communication interface 33, the external interface 34, the input device 35, the output device 36, and the drive 37 may be omitted. The inference device 3 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the inference device 3 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a mobile terminal (e.g., a smartphone), an industrial PC, a programmable logic controller (PLC), etc.

[0070] [Software configuration] <Label generating device> 5 schematically illustrates an example of the software configuration of the label generating device 1 according to this embodiment. The control unit 11 of the label generating device 1 loads the label generation program 81 stored in the storage unit 12 into the RAM. The control unit 11 then executes instructions included in the label generation program 81 loaded into the RAM using the CPU. As a result, as shown in FIG. 5, the label generating device 1 according to this embodiment operates as a computer including, as software modules, a first model acquisition unit 111, a second model acquisition unit 112, a data acquisition unit 113, a first inference unit 114, a second inference unit 115, a generation unit 116, and an output unit 117. That is, in this embodiment, each software module of the label generating device 1 is realized by the control unit 11 (CPU).

[0071] The first model acquisition unit 111 is configured to acquire a trained first inference model 61 generated by machine learning. The second model acquisition unit 112 is configured to acquire a trained second inference model 62 generated by machine learning.

[0072] The trained first inference model 61 is generated by machine learning using a plurality of first datasets 51. Each first dataset 51 is composed of a combination of first training data 511 and a first correct answer label 513. The first training data 511 of each first dataset 51 is collected in the source domain. The first correct answer label 513 is configured to indicate the correct answer (true value) of the inference task for the corresponding first training data 511.

[0073] On the other hand, the trained second inference model 62 is generated by machine learning using a plurality of second datasets 52. Each second dataset 52 is composed of a combination of second training data 521 and a second correct answer label 523. The second training data 521 of each second dataset 52 is generated by adding a disturbance to the first training data 511 included in one of the plurality of first datasets 51. The second correct answer label 523 is configured to indicate the correct answer (true value) of the inference task for the corresponding second training data 521.

[0074] The data acquiring unit 113 is configured to acquire third training data 531. The number of pieces of third training data 531 to be acquired may be determined appropriately depending on the embodiment. The first inference unit 114 is configured to acquire a first inference result by performing an inference task on the acquired third training data 531 using a trained first inference model 61. The second inference unit 115 is configured to acquire a second inference result by performing an inference task on the acquired third training data 531 using a trained second inference model 62. The generation unit 116 is configured to generate a third correct label 533 for the third training data 531 based on a match between the first inference result and the second inference result. The output unit 117 is configured to output the generated third correct label 533.

[0075] (One example of how to add disturbance) 6 schematically illustrates an example of a method for adding a disturbance according to the present embodiment. In the method of FIG. 6, a trained transformation model 65 generated by machine learning is used to add the disturbance. That is, adding the disturbance to the first training data 511 may be configured by transforming the first training data 511 using the trained transformation model 65.

[0076] The conversion model 65 may be configured by any machine learning model, such as a neural network. As long as the conversion model 65 can execute a calculation process for converting a style, the configuration and structure of the conversion model 65 are not particularly limited and may be determined appropriately depending on the embodiment. When a neural network is employed to configure the conversion model 65, the conversion model 65 may include any type of layer, such as a convolutional layer, a pooling layer, a dropout layer, a deconvolutional layer, an upsampling layer, or a fully connected layer. Furthermore, the number of layers in the conversion model 65, the number of nodes (neurons) in each layer, and the node connection relationships may be determined appropriately depending on the embodiment. The conversion model 65 may have at least one of a recursive structure and a residual structure. The machine learning model includes parameters for executing calculations to solve a task, which are adjusted by machine learning. When a neural network is employed, the connection weights between each node, the thresholds of each node, etc. are examples of parameters.

[0077] The trained conversion model 65 may be generated by machine learning using the first training data 511 and the third training data 531 so as to acquire the ability to convert the style of the first training data 511 into the style of the third training data 531. That is, to acquire such ability, the parameter values ​​of the conversion model 65 may be adjusted by machine learning using the first training data 511 and the third training data 531. The machine learning method may be determined appropriately depending on the configuration of the machine learning model to be adopted. The style is, for example, an attribute that defines a format or expression such as appearance or texture.

[0078] Methods for generating such trained conversion models65 include, for example, Reference 1 "Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge, "Image style transfer using convolutional neural networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016" and Reference 2 "Xun Huang, Serge Belongie, "Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization," [online], [Retrieved July 7, 2021], Internet.<URL:https: / / arxiv.org / abs / 1703.06868> ”, Reference 3 “Yijun Li, Ming-Yu Liu, Xueting Li, Ming-Hsuan Yang, Jan Kautz, “A Closed-form Solution to Photorealistic Image Stylization”, [online], [Retrieved July 7, 2021], Internet<URL:https: / / arxiv.org / abs / 1802.06474> ”, Reference 4 “Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, Jung-Woo Ha, “Photorealistic Style Transfer via Wavelet Transforms”, [online], [Retrieved July 7, 2021], Internet<URL:https: / / arxiv.org / abs / 1903.09760> " or the like may be adopted. This makes it possible to generate a trained transformation model 65. As a specific example of a configuration, the transformation model 65 may be configured by an encoder and a decoder. The encoder may be configured to receive content image data and style image data as input.The conversion model 65 may be trained to preserve content-related features of the content image data. Additionally, the conversion model 65 may be trained to match style-related features between the content image data and the style image data. In this training, the first training data 511 may be used for the content image data, and the third training data 531 may be used for the style image data. According to this trained conversion model 65, the first training data 511 and the third training data 531 are each converted into features by a trained encoder. The training data is restored from the obtained features of the first training data 511 by a trained decoder. This makes it possible to generate second training data 521 having content that matches the content of the first training data 511 and a style that matches the style of the third training data 531. Note that when the inference task is to extract a region containing a feature, information indicating the region containing the feature (correct label) may be reflected in the machine learning of the conversion model 65. In this case, the third training data 531 used in the machine learning of the conversion model 65 may be appropriately assigned a correct label.

[0079] The second training data 521 may be generated by converting the style of the first training data 511 included in at least one of the multiple first datasets 51 using the generated trained conversion model 65. That is, the first training data 511 is input to the trained conversion model 65, and calculation processing of the trained conversion model 65 is executed. The second training data 521 may be obtained as a result of executing the calculation processing of this trained conversion model 65.

[0080] According to this method of adding disturbances, a trained second inference model 62 is prepared by using second training data 521 having a style that matches the style of third training data 531 for machine learning. The trained second inference model 62 has acquired the ability to perform an inference task on the style of the third training data 531. Therefore, when third training data 531 is acquired in a target domain different from the source domain, the trained second inference model 62 is used together with the trained first inference model 61 to generate third correct labels 533, thereby increasing the likelihood of obtaining an appropriate correct answer to the inference task for the third training data 531. Therefore, by employing this method of adding disturbances when acquiring the second training data 521, a third dataset 53 for machine learning including a more reliable third correct label 533 can be generated.

[0081] The trained conversion model 65 may have the ability to convert style without changing the boundary of an instance. As an example of such conversion, if the data to be converted is image data and the inference task is to extract a region containing a feature in the image data, the trained conversion model 65 may be configured to change the surface texture without changing the position of the edge of the region. As another example, if the data to be converted is audio data and the inference task is to identify features contained in the audio data (e.g., performing speech recognition), the trained conversion model 65 may be configured to change the appearance, such as the pitch of the voice, without changing the content of the voice. As yet another example, if the data to be converted is text data and the inference task is to identify features contained in the text data (e.g., performing character recognition), the trained conversion model 65 may be configured to change the appearance, such as the tone of the text, without changing the meaning of the sentence. In these cases, the first correct answer label 513 can be used as the second correct answer label 523 for the converted second training data 521.

[0082] As a result of the above machine learning, learning result data 75 indicating the trained conversion model 65 may be generated. The configuration of the learning result data 75 is not particularly limited as long as it can hold information for executing calculations of the trained conversion model 65, and may be determined appropriately depending on the embodiment. As an example, the learning result data 75 may be configured to include information indicating the configuration of the conversion model 65 (e.g., the structure of a neural network) and the values ​​of parameters adjusted by machine learning. The learning result data 75 may be stored in any storage area. The learning result data 75 may be referenced as appropriate to set the trained conversion model 65 in a usable state on a computer.

[0083] In one example, the process related to machine learning of the conversion model 65 and the process of generating the second training data 521 may be executed on the same computer. In another example, the process related to machine learning of the conversion model 65 and the process of generating the second training data 521 may be executed on different computers. When a first computer that executes the process related to machine learning of the conversion model 65 and a second computer that generates the second training data 521 are different computers, the trained conversion model 65 (learning result data 75) may be provided from the first computer to the second computer at any timing. The first computer and the second computer may each be at least one of the label generation device 1, the model generation device 2, and the inference device 3, or may be a computer other than these.

[0084] However, the capabilities acquired by the conversion model 65 need not be limited to the above example. In another example, adding disturbances may be configured by converting the data into a style that does not depend on the third training data 531, such as a random style. The conversion model may be generated as appropriate to acquire the ability to perform such style conversion. Furthermore, the method of adding disturbances need not be limited to the method using such a conversion model. As another example, adding disturbances may be achieved by any randomization or expansion method, such as adding noise, changing jitter (in the case of image data, adjusting it in color space), grayscaling, contrast normalization, gamma correction, (color) histogram equalization, or geometric transformation. These methods can also increase the style diversity of the training data (first training data 511, second training data 521) and improve the robustness of the inference results obtained by the trained first inference model 61 and the trained second inference model 62. As a result, the reliability of the generated third correct answer label 533 can be improved.

[0085] Note that adding noise to the first training data 511 may be configured by adding perturbations (adversarial noise) to the first training data 511 according to the gradient descent method, which maximizes the loss function of the inference task. The adversarial noise makes it difficult for the first inference model 61 to solve the inference task, and the second inference model 62 trained using second training data 521 including this noise can acquire the ability to solve the inference task from a different perspective than the first inference model 61. As a result, a highly reliable third correct answer label 533 can be generated using the trained first inference model 61 and the trained second inference model 62.

[0086] When performing machine learning of the second inference model 62, the number of second data sets 52 to be generated is not particularly limited and may be determined appropriately depending on the embodiment. The number of second data sets 52 may or may not match the number of first data sets 51. Multiple second training data 521 may be generated from one piece of first training data 511 by adding different disturbances to one piece of first training data 511. There may be first training data 511 that is not used to generate the second training data 521.

[0087] In one example, the process related to machine learning of the conversion model 65 and the process of generating the second training data 521 (second data set 52) ​​may be executed in the label generating device 1 (second model acquisition unit 112). In this case, each process may be executed as part of the process of acquiring the trained second inference model 62. In another example, at least one of the process related to machine learning of the conversion model 65 and the process of generating the second training data 521 may be executed in a computer other than the label generating device 1.

[0088] Furthermore, the second correct labels 523 for each second data set 52 may be generated appropriately. In one example, at least a portion of the process for generating the second correct labels 523 may be performed manually. In another example, the second correct labels 523 may be automatically generated from the corresponding first correct labels 513. For example, if the transformation model 65 is trained to transform the style without changing the boundaries of the above-mentioned instances, the added disturbance may not change the correct answer to the inference task, such as when the inference task is to extract a region containing a feature in image data and the disturbance is at least one of color conversion and noise addition. In such a case, the first correct label 513 associated with the disturbed first training data 511 may be used as the second correct label 523 as is. In other cases, such as when the inference task is to extract a region containing a feature in image data and the disturbance is performed by performing a geometric transformation, the correct answer to the inference task for the disturbed second training data 521 may be derivable from the corresponding first correct label 513 based on the added disturbance. In such a case, the second ground truth label 523 may be generated by applying the disturbance-induced modification to the corresponding first ground truth label 513.

[0089] (An example of an inference model and machine learning method) 7A and 7B schematically illustrate an example of the configuration of each inference model (61, 62) and the machine learning process according to this embodiment.

[0090] Each inference model (61, 62) is configured by a machine learning model with parameters adjusted by machine learning. Each inference model (61, 62) may be configured by any machine learning model, such as a neural network. As long as the inference models (61, 62) can execute computational processing to perform an inference task, the configuration and structure of each inference model (61, 62) are not particularly limited and may be determined appropriately depending on the embodiment. The configuration of the first inference model 61 and the configuration of the second inference model 62 may be the same as or different from each other. In the example of FIGS. 7A and 7B, a neural network is used for each inference model (61, 62). As with the conversion model 65, the type of layers, the number of layers, the number of nodes in each layer, and the node connection relationships included in each inference model (61, 62) may be determined appropriately depending on the embodiment. Each inference model (61, 62) may have at least one of a recursive structure and a residual structure.

[0091] As shown in FIG. 7A , multiple first datasets 51 are used in machine learning of the first inference model 61. Each first dataset 51 is composed of a combination of first training data 511 and first correct labels 513. The multiple first datasets 51 may be collected as needed. As described above, the first training data 511 may be acquired as needed from a source domain. The corresponding first correct labels 513 may be obtained by operator input or by computer information processing using an arbitrary model. Each first dataset 51 may be automatically generated by computer operation, or may be manually generated at least partially including operator operation.

[0092] The machine learning of the first inference model 61 is configured by training the first inference model 61 so that, for each first data set 51, a result of performing an inference task using the first inference model 61 on the first training data 511 matches the correct answer indicated by the corresponding first correct answer label 513. That is, in the machine learning, the parameter values ​​of the first inference model 61 are adjusted (optimized) so that the error between the inference result for the first training data 511 of each first data set 51 and the correct answer indicated by the first correct answer label 513 is reduced. The machine learning method may be appropriately determined depending on the configuration of the machine learning model to be adopted. For example, stochastic gradient descent, mini-batch gradient descent, or the like may be used for the training process.

[0093] As an example of a training process when a neural network is employed, the first training data 511 of each first data set 51 is input to the first inference model 61, and a forward propagation calculation process of the first inference model 61 is performed. As a result of this forward propagation calculation, an inference result for the first training data 511 is obtained from the first inference model 61. The error between the obtained inference result and the correct answer indicated by the corresponding first correct answer label 513 is calculated, and the gradient of the calculated error is further calculated. Next, the error gradient is backpropagated using the error backpropagation method to calculate the error of the parameter value of the first inference model 61. The parameter value is then updated based on the calculated error. This series of update processes adjusts the parameter value of the first inference model 61 so that the sum of the errors between the inference result and the correct answer is reduced. This parameter value adjustment may be repeated until a predetermined condition is met, such as a predetermined number of times or the calculated sum of errors is equal to or less than a threshold. Furthermore, machine learning conditions, such as a loss function and a learning rate, may be set appropriately depending on the embodiment. This machine learning process makes it possible to generate a trained first inference model 61 that has acquired the ability to perform inference tasks depending on the multiple first datasets 51 used.

[0094] 7B, multiple second data sets 52 are used for machine learning of the second inference model 62. Each second data set 52 is composed of a combination of second training data 521 and second correct labels 523. As described above, the second training data 521 is generated by adding disturbances to the first training data 511. The second correct labels 523 may be generated as needed.

[0095] The machine learning of the second inference model 62 is configured by training the second inference model 62 so that, for each second data set 52, the result of performing an inference task using the second inference model 62 on the second training data 521 matches the correct answer indicated by the corresponding second correct answer label 523. That is, in the machine learning, the parameter values ​​of the second inference model 62 are adjusted (optimized) so that the error between the inference result for the second training data 521 of each second data set 52 and the correct answer indicated by the second correct answer label 523 is reduced.

[0096] Except for the fact that the data used for machine learning is different, the machine learning method of the second inference model 62 may be the same as that of the first inference model 61. As an example of adopting a neural network, the parameter values ​​of the second inference model 62 may be adjusted by backpropagation so as to reduce the error between the inference result for the second training data 521 of each second dataset 52 and the correct answer indicated by the corresponding second correct answer label 523.

[0097] Note that the training included in the machine learning of each inference model (61, 62) does not have to be limited to training for acquiring the ability to perform the above-mentioned inference task. For example, the machine learning of each inference model (61, 62) may further include additional training for any purpose, such as improving the accuracy of performing the inference task. In one example, the machine learning of each inference model (61, 62) may further include adversarial learning, as illustrated in Figures 7C and 7D below.

[0098] 7C and 7D schematically illustrate an example of the process of adversarial learning between the inference models (61, 62) and the discriminative models (67, 68) according to this embodiment. The first discriminative model 67 is configured to identify, from the inference result, which training data (511, 531) the inference result of the first inference model 61 corresponds to. The second discriminative model 68 is configured to identify, from the inference result, which training data (521, 531) the inference result of the second inference model 62 corresponds to.

[0099] Like the inference models (61, 62), the identification models (67, 68) are configured by machine learning models with parameters adjusted by machine learning. Like the inference models (61, 62), the identification models (67, 68) may be configured by any machine learning model, such as a neural network. As long as the identification models (67, 68) can execute the above-described identification calculation process, the configuration and structure of the identification models (67, 68) are not particularly limited and may be determined appropriately depending on the embodiment. In the examples of FIGS. 7C and 7D, a neural network is employed for each of the identification models (67, 68). The structure of the neural network (e.g., type of layer, number of layers, number of nodes in each layer, node connection relationship, presence or absence of a recursive structure, presence or absence of a residual structure, etc.) may be determined appropriately depending on the embodiment.

[0100] The adversarial learning of the first inference model 61 is configured by using the first training data 511 and the third training data 531 to train the first discriminative model 67 to distinguish whether the inference result of the first inference model 61 is for the first training data 511 or the third training data 531, and by using the first training data 511 and the third training data 531 to train the first inference model 61 to degrade the discriminative performance of the first discriminative model 67. Furthermore, the adversarial learning of the second inference model 62 is configured by using the second training data 521 and the third training data 531 to train the second discriminative model 68 to distinguish whether the inference result of the second inference model 62 is for the second training data 521 or the third training data 531, and by using the second training data 521 and the third training data 531 to train the second inference model 62 to degrade the discriminative performance of the second discriminative model 68. That is, similar to the machine learning process for acquiring the ability to perform the above-mentioned inference task, in each training process, the parameter values ​​of each inference model (61, 62) and each discrimination model (67, 68) are adjusted (optimized) to achieve each condition. Each training process may be similar to the training process for acquiring the ability to perform the above-mentioned inference task.

[0101] As an example of a case where a neural network is employed, in adversarial learning of a first inference model 61, first training data 511 and third training data 531 are input to the first inference model 61, and a forward propagation calculation process of the first inference model 61 is performed. As a result of this calculation process, inference results for each of the first training data 511 and the third training data 531 are obtained. Next, the inference results for each data are input to a first discrimination model 67, and a forward propagation calculation process of the first discrimination model 67 is performed. As a result of this calculation process, a discrimination result of the first discrimination model 67 for each inference result is obtained. Then, the error between the obtained discrimination result and the correct answer (true value) is calculated.

[0102] In the example of FIG. 7C , a gradient inversion layer 671 is provided between the first inference model 61 and the first discrimination model 67. The gradient inversion layer 671 is configured to pass values ​​unchanged during forward propagation and to invert values ​​during backpropagation. Therefore, by adjusting the parameter values ​​of the first discrimination model 67 and the first inference model 61 so as to reduce the calculated error using the backpropagation algorithm, the training of the first discrimination model 67 and the training of the first inference model 61 can be achieved simultaneously. That is, by adjusting the parameter values ​​of the first discrimination model 67 so as to reduce the calculated error, the first discrimination model 67 can be trained to acquire the ability to perform the discrimination task. Furthermore, by using the gradient of the error inverted by the gradient inversion layer 671 to adjust the parameter values ​​of the first inference model 61, the first inference model 61 can be trained so as to reduce the discrimination performance of the first discrimination model 67.

[0103] Similarly, in the adversarial learning of the second inference model 62, the second training data 521 and the third training data 531 are input to the second inference model 62, and a forward propagation calculation process is performed on the second inference model 62. As a result of this calculation process, inference results for the second training data 521 and the third training data 531 are obtained. Next, the inference results for each data are input to the second discrimination model 68, and a forward propagation calculation process is performed on the second discrimination model 68. As a result of this calculation process, a discrimination result of the second discrimination model 68 for each inference result is obtained. Then, the error between the obtained discrimination result and the correct answer (true value) is calculated.

[0104] 7C, in the example of FIG. 7D, a gradient inversion layer 681 is provided between the second inference model 62 and the second discriminant model 68. The gradient inversion layer 681 is configured in the same manner as the gradient inversion layer 671. Therefore, by adjusting the parameter values ​​of the second discriminant model 68 and the second inference model 62 so that the calculated error is reduced by the backpropagation method, the training of the second discriminant model 68 and the training of the second inference model 62 can be achieved simultaneously.

[0105] The first and second discriminative models 67 and 68 may have the same configuration or may be different. In one example, the first and second discriminative models 67 and 68 may be provided separately. In another example, the first and second discriminative models 67 and 68 may be the same. That is, a common discriminative model may be prepared for the first and second inference models 61 and 62. In this case, at least a part of the adversarial learning process for the first and second inference models 61 and 62 may be executed simultaneously.

[0106] In this embodiment, each inference model (61, 62) may be further trained by the above-mentioned adversarial learning between the discriminative models (67, 68). When third training data 531 is acquired in a target domain different from the source domain, this adversarial learning can be performed to prepare each inference model (61, 62) trained to capture features common to the source domain, the state in which the disturbance is added, and the target domain. In other words, as a result of this adversarial learning, when performing an inference task, the trained inference models (61, 62) can be made less susceptible to the influence of gaps between the source domain, the disturbance, and the target domain. Therefore, by employing this adversarial learning, a third dataset 53 for machine learning including a more reliable third correct answer label 533 can be generated.

[0107] On the other hand, if each discriminative model (67, 68) is configured to identify origins by capturing the overall characteristics of the inference results, the discriminative models (67, 68) may acquire the ability to discriminate based on differences in label distributions during adversarial learning. Once each discriminative model (67, 68) acquires such an ability, the inference models (61, 62) may be trained by adversarial learning to reduce the discriminative performance of each discriminative model (67, 68), potentially acquiring the ability to extract features that eliminate differences in label distributions. As a result, the inference accuracy of each inference model (61, 62) may deteriorate. To address this, each discriminative model (67, 68) may be configured to discriminate origins for each predetermined unit. If each training data (511, 521, 531) is composed of image data, the predetermined unit may be, for example, a pixel. If each training data (511, 521, 531) is composed of sound data or sensing data, the predetermined unit may be, for example, a frequency component.

[0108] As an example, each training data (511, 521, 531) may be composed of image data, and the inference task may be performed by extracting a region containing a feature from the image data. In this case, in the adversarial learning described above, the first discriminative model 67 may be trained using the first training data 511 and the third training data 531 to discriminate, for each pixel, whether the inference result of the first inference model 61 is for the first training data 511 or the third training data 531. The second discriminative model 68 may be trained using the second training data 521 and the third training data 531 to discriminate, for each pixel, whether the inference result of the second inference model 62 is for the second training data 521 or the third training data 531. Other aspects may be similar to the adversarial learning described above.

[0109] By configuring each discriminant model (67, 68) to discriminate for each predetermined unit in this way, it is possible to make the discrimination by each discriminant model (67, 68) independent of the closeness of the label distribution. This makes it possible to prevent a deterioration in the inference accuracy of each of the inference models (61, 62) in adversarial learning. As a result, it is possible to prepare a trained first inference model 61 and a trained second inference model 62 with high inference accuracy, and by using these, it is possible to generate a third dataset 53 for machine learning that includes a highly reliable third ground truth label 533.

[0110] Note that the adversarial learning of at least one of the first inference model 61 and the second inference model 62 may be omitted. Furthermore, in the adversarial learning, the gradient inversion layers (671, 681) may be omitted. In this case, the training process of each inference model (61, 62) and the training process of each discriminative model (67, 68) may be executed alternately. In the training process of each discriminative model (67, 68), the parameter values ​​of each inference model (61, 62) may be fixed, and the parameter values ​​of each discriminative model (67, 68) may be adjusted to reduce the above-mentioned error. On the other hand, in the training process of each inference model (61, 62), the parameter values ​​of each discriminative model (67, 68) may be fixed, and an error may be calculated to reduce the discrimination performance of each discriminative model (67, 68), and the parameter values ​​of each inference model (61, 62) may be adjusted based on the calculated error. As an example, the error between the classification result of each discrimination model (67, 68) and an incorrect answer opposite to the correct answer (true value) may be calculated, and the gradient of the calculated error may further be calculated. Then, the error gradient may be backpropagated to each inference model (61, 62) via each discrimination model (67, 68) using an error backpropagation method, and the parameter values ​​of each inference model (61, 62) may be adjusted so that the calculated error becomes smaller.

[0111] As shown in FIGS. 7A to 7D, as a result of the above machine learning, first learning result data 71 indicating a trained first inference model 61 may be generated. Second learning result data 72 indicating a trained second inference model 62 may be generated. The configuration of each learning result data (71, 72) is not particularly limited as long as it can hold information for executing calculations of each trained inference model (61, 62) and may be determined appropriately depending on the embodiment. As an example, each learning result data (71, 72) may be configured to include information indicating the configuration of each inference model (61, 62) (e.g., neural network structure, etc.) and parameter values ​​adjusted by machine learning. Each learning result data (71, 72) may be stored in any storage area. Each learning result data (71, 72) may be referenced as appropriate to set each trained inference model (61, 62) in a usable state on a computer.

[0112] In one example, the process of collecting the plurality of first data sets 51, the process of generating the first inference model 61 trained by machine learning, the process of collecting the plurality of second data sets 52, and the process of generating the second inference model 62 trained by machine learning may be executed on the same computer. In another example, at least one of these processes may be executed on different computers. These processes may be executed on at least one of the label generation device 1, the model generation device 2, and the inference device 3. Alternatively, at least one of these processes may be executed on a computer other than the label generation device 1, the model generation device 2, and the inference device 3. The process of generating the first inference model 61 trained by machine learning and the process of generating the second inference model 62 trained by machine learning may be executed on the same computer or on different computers.

[0113] In one example, the trained first inference model 61 may be generated by the label generating device 1. In this case, acquiring the trained first inference model 61 by the first model acquisition unit 111 may include performing machine learning on the first inference model 61. In another example, the trained first inference model 61 may be generated by a computer other than the label generating device 1. In this case, the trained first inference model 61 (first learning result data 71) may be provided to the label generating device 1 at any timing. The label generating device 1 may acquire the trained first inference model 61 via, for example, a network, a storage medium 91, an external storage device (for example, a NAS (Network Attached Storage)), or the like. Alternatively, the trained first inference model 61 may be pre-installed in the label generating device 1.

[0114] Similarly, the trained second inference model 62 may be generated by the label generating device 1. In this case, acquiring the trained second inference model 62 by the second model acquisition unit 112 may include performing machine learning on the second inference model 62. In another example, the trained second inference model 62 may be generated by a computer other than the label generating device 1. In this case, the trained second inference model 62 (second learning result data 72) may be provided to the label generating device 1 at any timing. The label generating device 1 may acquire the trained second inference model 62 via, for example, a network, a storage medium 91, an external storage device (e.g., a NAS (Network Attached Storage)), etc. Alternatively, the trained second inference model 62 may be pre-installed in the label generating device 1.

[0115] Note that some of the multiple datasets used in the machine learning of each inference model (61, 62) may include datasets that do not satisfy the conditions of each of the above datasets (51, 52). That is, it is sufficient that the multiple datasets used in the machine learning of the first inference model 61 include the above first dataset 51, and the multiple datasets used in the machine learning of the second inference model 62 include the above second dataset 52, and a dataset that does not meet the conditions of each dataset (51, 52) may also be used in the machine learning of each inference model (61, 62).

[0116] <Model generation device> 8 schematically illustrates an example of the software configuration of the model generation device 2 according to this embodiment. The control unit 21 of the model generation device 2 loads a model generation program 82 stored in the storage unit 22 into the RAM. The control unit 21 then executes instructions included in the model generation program 82 loaded into the RAM using the CPU. As a result, as shown in FIG. 8, the model generation device 2 according to this embodiment operates as a computer including a data acquisition unit 211, a learning processing unit 212, and a storage processing unit 213 as software modules. That is, in this embodiment, each software module of the model generation device 2 is realized by the control unit 21 (CPU), similar to the label generation device 1 described above.

[0117] The data acquisition unit 211 is configured to acquire multiple third data sets 53, each generated by associating a third correct label 533 generated by the label generation device 1 with corresponding third training data 531. The learning processing unit 212 is configured to perform machine learning of a third inference model 63 using the acquired multiple third data sets 53. The storage processing unit 213 is configured to generate information about the trained third inference model 63 generated by machine learning as third learning result data 73, and to store the generated third learning result data 73 in a predetermined storage area.

[0118] (An example of an inference model and machine learning method) The third inference model 63 is configured by a machine learning model having parameters adjusted by machine learning. The third inference model 63 may be configured by any machine learning model, such as a neural network. As with the inference models (61, 62), the configuration and structure of the third inference model 63 are not particularly limited as long as it can execute computational processing to perform an inference task, and may be determined appropriately depending on the embodiment. The configuration of the third inference model 63 may be the same as at least one of the configurations of the first inference model 61 and the second inference model 62, or may be different from the configurations of the first inference model 61 and the second inference model 62. In the example of FIG. 8, a neural network is adopted for the third inference model 63. The structure of the neural network (e.g., type of layer, number of layers, number of nodes in each layer, node connection relationship, presence or absence of a recursive structure, presence or absence of a residual structure, etc.) may be determined appropriately depending on the embodiment.

[0119] The machine learning of the third inference model 63 is configured by training the third inference model 63 so that, for each third data set 53, an inference result obtained by performing an inference task using the third training data 531 by the third inference model 63 matches the correct answer indicated by the corresponding third correct answer label 533. That is, in the machine learning, the parameter values ​​of the third inference model 63 are adjusted (optimized) so as to reduce the error between the inference result for the third training data 531 of each third data set 53 and the correct answer indicated by the third correct answer label 533. Except for the fact that the data used for the machine learning is different, the training process of the third inference model 63 may be similar to the training process of the first inference model 61, etc. As an example of a case where a neural network is employed, the learning processing unit 212 may be configured to adjust the parameter values ​​of the third inference model 63 using an error backpropagation algorithm so as to reduce the error between the inference result for the third training data 531 of each third data set 53 and the correct answer indicated by the corresponding third correct answer label 533. This machine learning process can generate a trained third inference model 63 that has acquired the ability to perform inference tasks.

[0120] The storage processing unit 213 is configured to generate third learning result data 73 indicating the trained third inference model 63 as a result of the machine learning. The configuration of the third learning result data 73 is not particularly limited as long as it can hold information for executing calculations of the trained third inference model 63, and may be determined appropriately depending on the embodiment. As an example, the third learning result data 73 may be configured to include information indicating the configuration of the third inference model 63 (e.g., the structure of a neural network) and the values ​​of parameters adjusted by machine learning.

[0121] <Inference device> 9 illustrates a schematic example of the software configuration of the inference device 3 according to this embodiment. The control unit 31 of the inference device 3 loads the inference program 83 stored in the storage unit 32 into the RAM. The control unit 31 then executes the instructions included in the inference program 83 loaded into the RAM using the CPU. As a result, as shown in FIG. 9, the inference device 3 according to this embodiment operates as a computer including an acquisition unit 311, an inference unit 312, and an output unit 313 as software modules. That is, in this embodiment, each software module of the inference device 3 is realized by the control unit 31 (CPU), similar to the label generation device 1 and the like.

[0122] The acquisition unit 311 is configured to acquire target data 321. The inference unit 312 holds third learning result data 73, thereby providing a trained third inference model 63. The inference unit 312 is configured to acquire an inference result by performing an inference task on the acquired target data 321 using the trained third inference model 63. The output unit 313 is configured to output information related to the inference result.

[0123] <Other> The software modules of the label generation device 1, the model generation device 2, and the inference device 3 will be described in detail in the operation examples below. In this embodiment, an example is described in which the software modules of the label generation device 1, the model generation device 2, and the inference device 3 are all implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors (e.g., graphics processing units). Each of the above modules may be implemented as a hardware module. Furthermore, with regard to the software configurations of the label generation device 1, the model generation device 2, and the inference device 3, software modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0124] §3 Example of operation [Label generating device] 10 is a flowchart showing an example of a processing procedure for label generation by the label generating device 1 according to this embodiment. The processing procedure of the label generating device 1 described below is an example of a label generation method. However, the processing procedure of the label generating device 1 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0125] (Step S101) In step S101, the control unit 11 operates as the first model acquisition unit 111 and acquires a trained first inference model 61 generated by machine learning using the above-mentioned plurality of first data sets 51.

[0126] In one example, the control unit 11 may generate the trained first inference model 61 by executing the above-described machine learning as the acquisition process of step S101. In another example, the control unit 11 may acquire the trained first inference model 61 generated by another computer, for example, via a network, a storage medium 91, an external storage device, etc. In yet another example, if the first learning result data 71 is stored in advance in the storage unit 12 or the storage medium 91, for example, by executing machine learning in advance or by acquiring the data in advance from another computer, the control unit 11 may acquire the trained first inference model 61 from the storage unit 12 or the storage medium 91.

[0127] The first inference model 61 may be further trained by adversarial learning with the first discrimination model 67. Upon acquiring the trained first inference model 61, the control unit 11 proceeds to the next step S102.

[0128] (Step S102) In step S102, the control unit 11 operates as the second model acquisition unit 112 and acquires a trained second inference model 62 generated by machine learning using the plurality of second data sets 52 described above.

[0129] In one example, the control unit 11 may generate a trained second inference model 62 by executing the above-mentioned machine learning as the acquisition process of step S102. In this case, the control unit 11 may generate a plurality of second data sets 52 by the above-mentioned method. The above-mentioned trained transformation model 65 may be used to add a disturbance to the first training data 511. The control unit 11 may generate a trained transformation model 65 by executing the above-mentioned machine learning, and may generate second training data 521 for each second data set 52 using the generated trained transformation model 65. Alternatively, at least some of the plurality of second data sets 52 used for machine learning may be generated by another computer, and the control unit 11 may acquire at least some of the plurality of second data sets 52 from the other computer.

[0130] In another example, the control unit 11 may acquire the trained second inference model 62 generated by another computer, for example, via a network, a storage medium 91, an external storage device, etc. In yet another example, if the second learning result data 72 is stored in advance in the storage unit 12 or the storage medium 91, the control unit 11 may acquire the trained second inference model 62 from the storage unit 12 or the storage medium 91.

[0131] The second inference model 62 may be further trained by adversarial learning with the second discriminative model 68. Upon acquiring the trained second inference model 62, the control unit 11 proceeds to the next step S103.

[0132] (Step S103) In step S103, the control unit 11 operates as the data acquisition unit 113 to acquire third training data 531. The number of third training data 531 to be acquired may be determined appropriately depending on the embodiment.

[0133] The domain from which the third training data 531 is acquired may be selected depending on the embodiment. In one example, the third training data 531 may be acquired in the same source domain as the first training data 511. In another example, the third training data 531 may be acquired in a target domain different from the source domain.

[0134] The method for collecting the third training data 531 may be selected appropriately depending on the embodiment. In one example, the third training data 531 may be generated by observing a real environment with a sensor. In another example, the third training data 531 may be generated by information processing such as a simulation, similar to the first training data 511.

[0135] As the processing of step S103, the control unit 11 may generate third training data 531 by the above-mentioned collection method. The control unit 11 may acquire the third training data 531 generated by another computer, for example, via a network, a storage medium 91, an external storage device, etc. Alternatively, if the third training data 531 has been collected in advance, the control unit 11 may acquire the third training data 531 from the storage unit 12 or the storage medium 91. Upon acquiring the third training data 531, the control unit 11 proceeds to the next step S104.

[0136] (Step S104) In step S104, the control unit 11 operates as the first inference unit 114 and performs an inference task on the acquired third training data 531 using the trained first inference model 61. That is, the control unit 11 inputs the acquired third training data 531 into the trained first inference model 61 and executes calculation processing of the trained first inference model 61. As a result of this calculation processing, the control unit 11 acquires a first inference result for the third training data 531 from the trained first inference model 61. Upon acquiring the first inference result, the control unit 11 proceeds to the next step S105.

[0137] (Step S105) In step S105, the control unit 11 operates as the second inference unit 115 and performs an inference task on the acquired third training data 531 using the trained second inference model 62. That is, the control unit 11 inputs the acquired third training data 531 into the trained second inference model 62 and executes calculation processing of the trained second inference model 62. As a result of this calculation processing, the control unit 11 acquires a second inference result for the third training data 531 from the trained second inference model 62. Upon acquiring the second inference result, the control unit 11 proceeds to the next step S106.

[0138] The processing order of steps S101 to S105 is not limited to the above example. The processing of step S104 is executed after the processing of steps S101 and S103. The processing of step S105 is executed after the processing of steps S102 and S103. Except for these points, the order of each processing may be changed as appropriate. Each processing may be executed in parallel. In another example, the control unit 11 may execute the processing of step S103 first. After executing the processing of step S101, the control unit 11 may subsequently execute the processing of step S104. The processing of step S102 may be executed before step S101. After executing the processing of step S102, the control unit 11 may subsequently execute the processing of step S105.

[0139] (Step S106) In step S106, the control unit 11 operates as the generation unit 116 and generates a third correct answer label 533 for the third training data 531 based on the match between the first inference result and the second inference result. The method of deriving the correct answer for the third training data 531 from the match between the first inference result and the second inference result may be any method that achieves a consensus between the trained first inference model 61 and the trained second inference model 62, and may be determined appropriately depending on the format of each inference result and the content of the inference task.

[0140] In one example, the inference task may be to extract a region containing a feature. In the case of image data, the region containing a feature may be, for example, a region where a specific object, such as an object to be identified, is present. In the case of sound data, the region containing a feature may be, for example, a region where a specific sound (e.g., a speaker's speech, the sound of a machine malfunction) is emitted. In the case of sensing data, the region containing a feature may be, for example, a region where a given feature appears (for example, if the sensing data is vital sign data, a region where there is an abnormality in the vital signs or a sign of such an abnormality). Each inference result may be configured to indicate the result of extracting a region containing a feature in the third training data 531 (e.g., in the case of image data, a segmentation result).

[0141] In this case, the process of generating the third correct answer label 533 may be configured as follows. That is, the control unit 11 may identify an overlapping portion between the region extracted as the first inference result and the region extracted as the second inference result. Next, the control unit 11 may compare the size of the identified overlapping portion with a threshold and determine whether the size of the overlapping portion exceeds the threshold. The threshold may be set appropriately. Then, if the size of the overlapping portion exceeds the threshold, the control unit 11 may generate a third correct answer label 533 configured to indicate the overlapping portion as the correct answer to the inference task. On the other hand, if the size of the overlapping portion is less than the threshold, the control unit 11 may omit generating the third correct answer label 533 based on the match between the first inference result and the second inference result. This makes it possible to appropriately derive the correct answer to the inference task in the third training data 531 from the match between the first inference result and the second inference result, and generate a highly reliable third correct answer label 533. Note that if the size of the overlapping portion is equal to the threshold, the process may branch to either direction.

[0142] In another example, the inference task may be identifying a class (category) of a feature included in the data. In the case of image data, identifying the class of a feature may be, for example, identifying the type of object shown in the image data. In the case of a product, identifying the type of object may include, for example, identifying the presence or absence of a defect, the type of defect, or other visual inspection-related identification. In the case of sound data, identifying the class of a feature may be, for example, identifying a speaker's utterance, identifying a speaker, or identifying the state of a machine from its sound (e.g., the presence or absence of a malfunction or its precursor). In the case of sensing data, identifying the class of a feature may be, for example, identifying the state of an object shown in the sensing data (e.g., the health state of a target person if the sensing data is vital sign data). Each inference result may be configured to indicate the result of identifying the class of a feature included in the third training data 531.

[0143] In this case, the process of generating the third correct label 533 may be configured as follows. That is, the control unit 11 may determine whether the class identified as the first inference result and the class identified as the second inference result match. Then, if the class indicated by the first inference result and the class indicated by the second inference result match, the control unit 11 may generate the third correct label 533 configured to indicate the matched class. On the other hand, if the class indicated by the first inference result and the class indicated by the second inference result do not match, the control unit 11 may omit generating the third correct label 533 based on the match between the first inference result and the second inference result. This makes it possible to appropriately derive the correct answer to the inference task in the third training data 531 from the match between the first inference result and the second inference result and generate a highly reliable third correct label 533. Note that, for example, if the first inference result and the second inference result partially match, such as when multiple classes are identified and some of the classes match, the control unit 11 may determine the branch destination of the process depending on the number of matching classes. In one example, if the number of matching classes exceeds or is equal to or greater than a threshold, the control unit 11 may generate a third correct answer label 533 configured to indicate the partially matching class.

[0144] Note that extracting a region containing a feature and identifying the class of the feature contained in the data may be performed simultaneously. That is, the inference task may be configured by extracting a region containing a feature and identifying the class of the feature contained in the data. In this case, the control unit 11 may generate a third correct answer label 533 configured to indicate the overlapping portion and the matching class. After generating the third correct answer label 533, the control unit 11 proceeds to the next step S107.

[0145] (Step S107) In step S107, the control unit 11 operates as the output unit 117 and outputs the generated third correct label 533.

[0146] The output destination and output format of the third correct label 533 are not particularly limited as long as the operator can confirm the generated third correct label 533, and may be appropriately selected depending on the embodiment. As an example, the control unit 11 may output the third correct label 533 via the output device 16 or an output device of another computer. The third correct label 533 may be output, for example, by being displayed on a display or spoken through a speaker.

[0147] The third training data 531 may also be output together with the third correct label 533. This may prompt the operator to confirm the generated third correct label 533. After outputting the third correct label 533, the control unit 11 may accept correction or deletion of the generated third correct label 533 via the input device 15 or an input device of another computer. The correction or deletion of the generated third correct label 533 may be accepted by the other computer. If the generation of the third correct label 533 is omitted in step S106, the control unit 11 may output information indicating that the generation of the third correct label 533 has been omitted. In response to this, the control unit 11 or the other computer may accept input of the third correct label 533 for the third training data 531 from the operator. The control unit 11 or the other computer may generate the third dataset 53 at any timing by associating the finally obtained third correct label 533 with the third training data 531.

[0148] When the output of the third correct label 533 is completed, the control unit 11 ends the processing procedure of the label generating device 1 according to this operation example. The control unit 11 may execute the series of information processing from step S101 to step S107 in response to an instruction from an operator. Alternatively, the control unit 11 may execute the series of information processing from step S101 to step S107 in response to an instruction from another computer to generate the third correct label 533. The control unit 11 may generate the third correct label 533 for each of the multiple pieces of third training data 531 by repeatedly executing the information processing from step S103 to step S107.

[0149] [Model generation device] FIG. 11 is a flowchart showing an example of a processing procedure for machine learning by the model generation device 2 according to this embodiment. The processing procedure of the model generation device 2 described below is an example of a model generation method. However, the processing procedure of the model generation device 2 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0150] (Step S201) In step S201, the control unit 21 operates as a data acquisition unit 211 and acquires multiple third data sets 53 each generated by associating the third correct label 533 generated by the label generation device 1 with the corresponding third training data 531.

[0151] In one example, as the processing of step S201, the control unit 21 may acquire third training data 531, provide the acquired third training data 531 to the label generating device 1, and instruct the label generating device 1 to generate third correct labels 533. The control unit 21 may receive the generated third correct labels 533 from the label generating device 1 and generate the third data set 53 by associating the received third correct labels 533 with the corresponding third training data 531. In another example, the multiple third data sets 53 may be generated by at least one of the label generating device 1 and another computer. In this case, the control unit 21 may acquire the multiple third data sets 53 generated by at least one of the label generating device 1 and another computer via, for example, a network, a storage medium 92, an external storage device, or the like. In yet another example, when the multiple third data sets 53 are stored in advance in the storage unit 22 or the storage medium 92, the control unit 21 may acquire the multiple third data sets 53 from the storage unit 22 or the storage medium 92.

[0152] The number of third data sets to be acquired is not particularly limited and may be determined appropriately so as to enable machine learning. After acquiring multiple third data sets 53, the control unit 21 proceeds to the next step S202.

[0153] Note that some of the multiple datasets used for machine learning of the third inference model 63 may include datasets other than the above-mentioned third dataset 53. Furthermore, some of the multiple third datasets 53 may include datasets to which correct labels have been assigned by a method other than the above-mentioned label generation method (for example, manually). In this embodiment, it is sufficient that at least some of the multiple datasets used for machine learning of the third inference model 63 include a third dataset 53 including a third correct label 533 generated by a method other than the above-mentioned label generation method.

[0154] (Step S202) In step S202, the control unit 21 operates as the learning processing unit 212 and performs machine learning of the third inference model 63 using the acquired multiple third data sets 53. As described above, the control unit 21 adjusts the parameter values ​​of the third inference model 63 through machine learning so as to reduce the error between the inference result for the third training data 531 of each third data set 53 and the correct answer indicated by the third correct answer label 533. As a result of this machine learning, it is possible to generate a trained third inference model 63 that has acquired the ability to perform an inference task. When the machine learning process is complete, the control unit 21 proceeds to the next step S203.

[0155] (Step S203) In step S203, the control unit 21 operates as the storage processing unit 213 and generates information about the trained third inference model 63 generated by machine learning as third learning result data 73. Then, the control unit 21 stores the generated third learning result data 73 in a predetermined storage area.

[0156] The predetermined storage area may be, for example, RAM in the control unit 21, the storage unit 22, an external storage device, a storage medium, or a combination of these. The storage medium may be, for example, a CD or a DVD, and the control unit 21 may store the third learning result data 73 in the storage medium via the drive 27. The external storage device may be, for example, a data server such as a NAS. In this case, the control unit 21 may use the communication interface 23 to store the third learning result data 73 in the data server via a network. The external storage device may also be, for example, an external storage device connected to the model generation device 2 via the external interface 24.

[0157] When the storage of the third learning result data 73 is completed, the control unit 21 ends the processing procedure of the model generation device 2 according to this operation example.

[0158] The generated third learning result data 73 may be provided to the inference device 3 at any timing. For example, the control unit 21 may transfer the third learning result data 73 to the inference device 3 as part of the processing of step S203 or separately from the processing of step S203. The inference device 3 may acquire the third learning result data 73 by receiving this transfer. Alternatively, for example, the inference device 3 may acquire the third learning result data 73 by accessing the model generation device 2 or a data server via a network using the communication interface 33. Alternatively, for example, the inference device 3 may acquire the third learning result data 73 via a storage medium 93. Alternatively, for example, the third learning result data 73 may be pre-installed in the inference device 3.

[0159] Furthermore, the control unit 21 may periodically or irregularly repeat the processes of steps S201 to S203 to update or generate new third learning result data 73. During this repetition, at least a portion of the third data set 53 used for machine learning may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 21 may update the third learning result data 73 held by the inference device 3 by providing the updated or newly generated third learning result data 73 to the inference device 3 by any method.

[0160] [Inference device] 12 is a flowchart showing an example of a processing procedure for performing an inference task by the inference device 3 according to this embodiment. The processing procedure of the inference device 3 described below is an example of an inference method. However, the processing procedure of the inference device 3 described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0161] (Step S301) In step S301, the control unit 31 operates as the acquisition unit 311 and acquires the target data 321.

[0162] The target data 321 is the same type of data as the training data (511, 521, 531) and is the target for performing an inference task. The target data 321 may be acquired in any domain. In one example, the target data 321 may be acquired in a source domain. In another example, the subject data 321 may be acquired in a target domain. The subject data 321 may be acquired in a domain that is the same as or similar to the domain in which the third training data 531 is acquired.

[0163] The method for acquiring the target data 321 may be selected appropriately depending on the embodiment, similar to the third training data 531. In one example, the target data 321 may be generated by observing a real environment with a sensor. In another example, the target data 321 may be generated by information processing such as simulation.

[0164] As the processing of step S301, the control unit 31 may generate the target data 321 using the above-described generation method. The control unit 31 may acquire the target data 321 generated by another computer, for example, via a network, the storage medium 92, an external storage device, etc. Alternatively, if the target data 321 has been acquired in advance, the control unit 31 may acquire the target data 321 from the storage unit 32 or the storage medium 93. Upon acquiring the target data 321, the control unit 31 proceeds to the next step S302.

[0165] (Step S302) In step S302, the control unit 31 operates as the inference unit 312 and sets the trained third inference model 63 by referring to the third learning result data 73. Then, the control unit 31 uses the trained third inference model 63 to perform an inference task on the acquired target data 321. That is, the control unit 31 inputs the acquired target data 321 into the trained third inference model 63 and executes calculation processing of the trained third inference model 63. As a result of executing this calculation processing, the control unit 31 acquires an inference result of the inference task for the target data 321. Upon acquiring the inference result, the control unit 31 proceeds to the next step S303.

[0166] (Step S303) In step S303, the control unit 31 operates as the output unit 313 and outputs information related to the inference result.

[0167] The output destination of the inference result and the content of the information to be output may be determined appropriately depending on the embodiment. For example, the control unit 31 may output the inference result obtained in step S302 directly to the output device 36 or an output device of another computer. The control unit 31 may also perform some kind of information processing based on the obtained inference result. The control unit 31 may then output the result of the information processing as information related to the inference result. The output of the result of the information processing may include controlling the operation of the controlled device in accordance with the inference result. The output destination may be, for example, the output device 36, an output device of another computer, the controlled device, etc.

[0168] When the output of information related to the inference result is completed, the control unit 31 ends the processing procedure of the inference device 3 according to this operation example. The control unit 31 may continuously and repeatedly execute the series of information processes from step S301 to step S303. The timing of the repetition may be determined appropriately depending on the embodiment. In this way, the inference device 3 may be configured to continuously and repeatedly execute the above inference task.

[0169] [Features] As described above, in this embodiment, different data sets (the first data set 51 and the second data set 52) ​​are prepared for machine learning by adding disturbances to the first training data 511. Because the addition of disturbances can be automated, the second data set 52 can be generated easily and at low cost from the first data set 51. Furthermore, by the processes of steps S101 and S102, a trained first inference model 61 and a trained second inference model 62 derived from different data sets are prepared. By using these in the processes of steps S104 and S105, an inference result obtained by performing an inference task on the third training data 531 from different perspectives can be obtained. Then, in step S106, a third correct answer label 533 is generated based on a match between the inference results obtained from different perspectives, thereby increasing the likelihood of obtaining an appropriate correct answer from features common to the different perspectives (i.e., data features truly related to the inference task). As a result, a highly reliable third correct answer label 533 can be generated. In addition, at least a part of the process of generating the third correct answer label 533 can be automated. Therefore, according to the label generating device 1 according to this embodiment, the third dataset 53 for machine learning including the highly reliable third correct label 533 can be generated at low cost.

[0170] Even when the third training data 531 is acquired in a target domain, the first inference model 61 and the second inference model 62 trained to perform an inference task from different perspectives can be used to increase the likelihood of obtaining an appropriate correct answer from common features without being affected by differences in domains. Therefore, not only when the third training data 531 is acquired in the source domain, but also when the third training data 531 is acquired in a target domain different from the source domain, it is possible to generate the third dataset 53 for machine learning, including the highly reliable third correct answer labels 533, at low cost.

[0171] In the model generation device 2 according to this embodiment, by performing the processing in steps S201 to S202, the third dataset 53 including the highly reliable third correct label 533 is used for machine learning, thereby improving the inference performance of the generated trained third inference model 63. Furthermore, in the inference device 3 according to this embodiment, by using the trained third inference model 63 thus generated in the processing in step S302, it is possible to expect the execution of highly accurate inference tasks for the target data 321.

[0172] There are also machine learning methods using the same dataset to train two networks (two output layers) to capture features in different directions by introducing a regularization term that separates the decision boundaries of inference, such as Reference 5 "Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada, "Asymmetric Tri-training for Unsupervised Domain Adaptation," In ICML, 2017" and Reference 6 "Juning Zhang, Chen Liang, C.-C Jay Kuo, "A Fully Convolutional Tri-branch Network (FCTN) for Domain Adaptation," In ICASSP, 2018." However, with this method, the only difference is the direction of feature capture; it is unclear what representation features each network is trained to capture. In addition, it is difficult to determine the weights (hyperparameters) of the regularization term. If the regularization is too weak, the two networks will be trained to capture similar features. If normalization is too strong, the loss function for acquiring the ability to solve an inference task cannot be properly optimized, resulting in two trained networks that are trained to capture different features but have poor inference accuracy. In contrast, according to this embodiment, the feature representation acquired by the second inference model 62 (i.e., the features captured when solving an inference task) can be controlled by adding a disturbance to the first training data 511. This makes it possible to control the direction in which the robustness of the trained second inference model 62 is enhanced. As a result, a highly reliable third correct answer label 533 can be generated by achieving a consensus (match in inference results) between the trained first inference model 61 and the trained second inference model 62.

[0173] §4 Variations Although the embodiments of the present invention have been described above in detail, the above description is merely illustrative of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiment, and descriptions of the same points as those in the above embodiment are omitted where appropriate. The following modifications can be combined as appropriate.

[0174] <4.1> The inference system 100 according to the above embodiment may be applied to any situation in which an inference task is performed on any data. As described above, the data that is the subject of the inference task may be, for example, image data, sound data, numerical data, text data, or other sensing data obtained by various sensors. Furthermore, the data that is the subject of the inference task may be composed of multiple types of data, such as video data that includes sound. The data that is the subject of the inference task may be appropriately selected depending on the inference task. Examples of inference tasks include extracting a region containing a feature from image data, identifying a class of a feature contained in image data, extracting a region containing a feature from sound data, identifying a class of a feature contained in sound data, extracting a region containing a feature from other sensing data, or identifying a class of a feature contained in sensing data. Below, we present modified examples that limit the application scenario.

[0175] (A) A scene of inference based on image data 13 is a diagram illustrating an example of an application scene of the inference system 100 according to the first modification. This modification is an example in which the above embodiment is applied to a scene in which an inference task is performed on image data.

[0176] In this modification, the training data (511, 521, 531) and the target data 321 are composed of image data that may capture the target object RA and are acquired under the respective conditions. The image data may be configured to represent, for example, a still image, a moving image, a 3D image, or other image. The image data may be obtained by a camera SA, may be generated by appropriately processing raw data obtained by the camera SA, or may be generated independently of the camera SA by any image processing. The camera SA may be, for example, a general RGB camera, a depth camera, an infrared camera, or the like. The label generation device 1, the model generation device 2, and the inference device 3 may each be connected to the camera SA via a communication interface (13, 23, 33) or an external interface (14, 24, 34).

[0177] In this modification, the inference task may consist of at least one of extracting a region containing a feature in the image data and identifying a class of the feature contained in the image data. The region containing a feature may be, for example, an area in which the object RA is captured, or an area in which any feature part (e.g., a defect, a body part, etc.) in the object RA is captured. The class of the feature may be, for example, the type of the object RA, the type of the feature part in the object RA (which may include the presence or absence of the feature part), etc. The object RA may be a person or any object.

[0178] As an example, the object RA may be a person. In this case, the region including the feature may be, for example, a region including the entire person, a region including a body part of the person (e.g., a face, arms, legs, joints, etc.), etc. Identifying the class of the feature may be, for example, identifying the attribute of the person, identifying the type of body part, identifying the state of the person, etc. As a specific application scenario, the camera SA may be a surveillance camera installed in a predetermined location (e.g., a street, a train station, an airport, a hall, etc.). In this scenario, identifying the attribute of the person may be, for example, determining whether a specific person is present. Identifying the state of the person may be, for example, determining whether the person shows signs of danger. As yet another specific application scenario, the image data may be medical image data. In this scenario, the region including the feature may be, for example, a lesion region or a region suspected of a lesion. Identifying the class of the feature may be, for example, determining whether a lesion is present or absent, or identifying the type of lesion, etc.

[0179] As another example, the object RA may be a product produced on a production line. The camera SA may be installed to monitor the condition of the product. In this case, the area containing the feature may be, for example, an area in which the product is captured, an area in which a specific location in the product (e.g., a location with a code) is captured, an area in which a defect in the product is captured, or the like. Identifying the class of the feature may be, for example, identifying the type of product, identifying the presence or absence of a defect, identifying the type of defect contained in the product (which may include a type indicating "no defect"), or the like.

[0180] The product may be, for example, a product transported on a production line for electronic devices, electronic components, automotive parts, pharmaceuticals, food, etc. The electronic components may be, for example, a circuit board, a chip capacitor, a liquid crystal display, a relay winding, etc. The automotive parts may be, for example, a connecting rod, a shaft, an engine block, a power window switch, a panel, etc. The pharmaceuticals may be, for example, packaged tablets, unpackaged tablets, etc. The product may be a final product produced after the manufacturing process is completed, an intermediate product produced during the manufacturing process, or an initial product prepared before the manufacturing process. The defects may be, for example, scratches, stains, cracks, dents, burrs, color unevenness, foreign matter contamination, etc.

[0181] Except for these points, the inference system 100 according to this modification may be configured in the same manner as the above embodiment.

[0182] In this modification, the label generation device 1 can generate a third correct label 533 for third training data 531 composed of image data using a processing procedure similar to that of the above embodiment. As in the above embodiment, the third training data 531 may be acquired in the source domain or the target domain. For example, the target domain may differ from the source domain due to changes in conditions such as shooting conditions (e.g., brightness, etc.), the subject, camera settings, camera installation angle, and background from the source domain. Each inference model (61, 62) may be further trained by adversarial learning between each of the discriminative models (67, 68). In adversarial learning, each discriminative model (67, 68) may be trained to identify, for each pixel, which training data the inference results of each inference model (61, 62) correspond to.

[0183] The model generation device 2 can generate a trained third inference model 63 that has acquired the ability to perform an inference task on image data, using a processing procedure similar to that of the above embodiment. The inference device 3 can perform an inference task on target data 321 consisting of image data, using the trained third inference model 63, using a processing procedure similar to that of the above embodiment. Depending on the content of the inference task, the inference device 3 may be interpreted as, for example, an inspection device, an identification device, a monitoring device, etc.

[0184] In addition, the inference device 3 may perform output processing according to the inference task in step S303. In one example, the control unit 31 of the inference device 3 may output the inference result as is. In another example, the control unit 31 may perform any information processing according to the inference result. As a specific example, in the case of determining whether a sign of danger exists, the control unit 31 may output a warning to notify the user if it is determined that a sign of danger exists. As another specific example, in the case of medical images, if a lesion area or a suspected lesion area is extracted, the control unit 31 may display the area on the medical image data along with the type of lesion. As yet another specific example, in the case of visual inspection of products, the production line may include a conveyor device that transports products. The inference device 3 may be connected to the conveyor device or a controller that controls the conveyor device. Based on the result of determining whether or not the product has a defect, the control unit 31 may control the conveyor device to transport defective products and non-defective products on separate lines.

[0185] (Features) According to the first modification, the label generation device 1 can generate highly reliable third correct labels 533 at low cost for third training data 531 consisting of image data. In the model generation device 2, by using the third dataset 53 including the highly reliable third correct labels 533 for machine learning, it is possible to generate a trained third inference model 63 that has acquired the ability to perform inference tasks on image data with high accuracy. Furthermore, by using the trained third inference model 63 thus generated in the inference device 3, it is possible to expect the inference task to be performed with high accuracy on target data 321 consisting of image data.

[0186] (B) Inference based on sound data 14 is a schematic diagram illustrating an example of an application scenario of the inference system 100 according to the second modification. This modification is an example in which the above embodiment is applied to a scenario in which an inference task is performed on sound data.

[0187] In this modification, the training data (511, 521, 531) and the target data 321 are composed of sound data that may be related to the target RB acquired under the respective conditions. The sound data may be obtained by observing the target RB with a microphone SB, may be generated by appropriately processing raw data acquired by the microphone SB, or may be generated independently of the microphone SB through any sound generation process. The type of microphone SB may be appropriately selected depending on the embodiment. The label generation device 1, the model generation device 2, and the inference device 3 may each be connected to the microphone SB via a communication interface (13, 23, 33) or an external interface (14, 24, 34).

[0188] In this variation, the inference task may be configured by at least one of extracting a region containing features in the sound data and identifying a class of features contained in the sound data. The region containing features may be, for example, a range containing a specific sound. Identifying a class of features may be, for example, identifying a type of sound.

[0189] For example, the target RB may be the voice of a speaker. In this case, the range including a specific sound may be, for example, a range including a specific utterance. Identifying the type of sound may be, for example, identifying the speaker or analyzing the content of the utterance.

[0190] As another example, the target RB may be an environmental sound. In this case, the inference task may relate to the state or situation of the environment. Extracting a range including a specific sound may be, for example, extracting sounds related to an accident occurring in the environment. Furthermore, identifying the type of sound may be, for example, determining whether a specific accident is occurring in the environment, determining whether there are signs of an accident occurring, determining the weather, etc.

[0191] As yet another example, the target RB may be the operating sound of a machine. In this case, the inference task may relate to the state of the machine. Extracting a range including a specific sound may be, for example, extracting normal operating sounds of the machine, extracting abnormal or malfunctioning sounds of the machine, etc. Furthermore, identifying the type of sound may be, for example, determining whether the machine is operating normally, determining whether there are signs of a malfunction or abnormality in the machine, etc.

[0192] Except for these points, the inference system 100 according to this modification may be configured in the same manner as the above embodiment.

[0193] In this modification, the label generation device 1 can generate a third correct label 533 for third training data 531 composed of sound data using a processing procedure similar to that of the above embodiment. As in the above embodiment, the third training data 531 may be acquired in the source domain or the target domain. For example, the target domain may differ from the source domain by changing conditions such as sound acquisition conditions, observation target, microphone settings, microphone installation angle, and background sound from the source domain. Each inference model (61, 62) may be further trained by adversarial learning between each of the discriminative models (67, 68). In the adversarial learning, each discriminative model (67, 68) may be trained to identify, for each frequency component, which training data the inference results of each inference model (61, 62) correspond to.

[0194] The model generation device 2 can generate a trained third inference model 63 that has acquired the ability to perform an inference task on sound data using a processing procedure similar to that of the above embodiment. The inference device 3 can use the trained third inference model 63 using a processing procedure similar to that of the above embodiment to perform an inference task on target data 321 consisting of sound data. Depending on the content of the inference task, the inference device 3 may be interpreted as, for example, a detection device, an identification device, a monitoring device, etc.

[0195] In addition, the inference device 3 may execute output processing according to the inference task in step S303. In one example, the control unit 31 of the inference device 3 may output the inference result as is. In another example, the control unit 31 may execute any information processing according to the inference result. As a specific example, in a case where a speaker's voice is recognized, the control unit 31 may determine a response content according to the speaker's utterance content and output the determined response content. Alternatively, the control unit 31 may execute a language search (e.g., a term search, a pop song search, etc.) based on the speaker's utterance content and output the search results. As another specific example, in a case where the state of a machine is inferred from the above-mentioned mechanical sound, if the control unit 31 determines, based on the inference result, that the target machine is broken or has signs of a breakdown, it may execute processing to deal with the breakdown or signs of a breakdown, such as stopping the operation of the machine or outputting a notification informing the user of the breakdown.

[0196] (Features) According to the second modification, the label generation device 1 can generate highly reliable third correct labels 533 at low cost for third training data 531 consisting of sound data. In the model generation device 2, by using the third dataset 53 including the highly reliable third correct labels 533 for machine learning, it is possible to generate a trained third inference model 63 that has acquired the ability to perform inference tasks on sound data with high accuracy. Furthermore, by using the trained third inference model 63 generated in this way in the inference device 3, it is possible to expect the inference task to be performed with high accuracy on target data 321 consisting of sound data.

[0197] (C) Inference based on sensing data 15 is a diagram illustrating an example of an application scenario of the inference system 100 according to the third modification. This modification is an example in which the above embodiment is applied to a scenario in which an inference task is performed on sensing data.

[0198] In this modification, the training data (511, 521, 531) and the target data 321 are composed of sensing data related to the target object RC acquired under the respective conditions. The sensing data may be obtained by observing the target object RC with a sensor SC, or may be generated by appropriately processing raw data obtained by the sensor SC (e.g., extracting features), or may be generated by simulating the operation of the sensor SC. The sensing data may be composed of a single type of data or multiple types of data. The sensor SC may be, for example, a camera, a microphone, an encoder, an environmental sensor, a vital sign sensor, a medical testing device, an in-vehicle sensor, a home security sensor, etc. The label generation device 1, the model generation device 2, and the inference device 3 may be connected to the sensor SC via a communication interface (13, 23, 33) or an external interface (14, 24, 34).

[0199] In this modification, the inference task may be configured by at least one of extracting a region containing a feature in the sensing data and identifying a class of the feature contained in the sensing data. Extracting a region containing a feature may be, for example, extracting a portion related to a specific state or situation of the object RC. Identifying a class of the feature may be, for example, identifying a specific state or situation of the object RC. The sensor SC may be selected appropriately depending on the inference task.

[0200] As an example, the target object RC may be a target person, and the inference task may relate to the target person's condition. In this case, the sensor SC may be, for example, at least one of a microphone, a vital sign sensor, and a medical examination device. Extracting a region including a feature may, for example, extract a component related to a specific condition of the target person. Identifying a class of a feature may, for example, determine whether a specific disease has occurred, determine whether there are signs of a specific disease, identify the type of disease that has occurred, identify the type of health condition, or the like. As an example of a specific application scenario, the target person may be a vehicle driver, and identifying the target person's condition may, for example, identify the level of drowsiness, fatigue, or ease of mind.

[0201] As another example, the object RC may be an industrial machine, and the inference task may relate to the state of the industrial machine. In this case, the sensor SC may be, for example, composed of at least one of a microphone, an encoder, and an environmental sensor. Extracting a region including a feature may, for example, extract a component related to a specific state of the industrial machine. Identifying a class of features may, for example, identify the state of the industrial machine, such as determining whether there is an abnormality in the industrial machine or whether there is a sign of an abnormality occurring in the industrial machine. The sensing data may, for example, be composed of an encoder value, temperature, operating sound, etc. of a motor.

[0202] As another example, the target object RC may be an object outside the vehicle, and the inference task may relate to the state or situation of the object. In this case, the sensor SC may be, for example, a camera or an on-board sensor. Extracting a region including a feature may, for example, extract a portion related to an object outside the vehicle, extract a component related to a specific state or situation of the object, etc. Identifying a feature class may, for example, identify attributes of an object outside the vehicle, identify a congestion situation, identify an accident risk, etc. The object outside the vehicle may, for example, be a road, a traffic light, an obstacle (person, object), etc. Identifying the attributes of an object outside the vehicle may, for example, include determining whether an event such as a person or vehicle suddenly jumping out, a sudden start, a sudden stop, or a lane change has occurred or there is a sign of such an event.

[0203] As another example, the target object RC may be an object present in a specific location, for example, outdoors or a specific indoor location (e.g., inside a greenhouse), and the inference task may relate to the situation of the specific location. In this case, the sensor SC may be composed of, for example, a camera, a microphone, and / or an environmental sensor. Extracting a region including a feature may be, for example, extracting a component related to the specific situation. Identifying a class of features may be, for example, identifying the specific situation. As an example of a specific application scenario, the target object RC may be a plant, and identifying the specific situation may be identifying the plant's cultivation situation.

[0204] As another example, the target object RC may be, for example, an object present in a house, and the inference task may relate to a situation within the house. In this case, the sensor SC may be, for example, a camera, a microphone, an environmental sensor, or a home security sensor. Extracting a region including a feature may, for example, extract a component related to a specific situation within the house. Identifying a class of the feature may, for example, identify a specific situation within the house.

[0205] Except for these points, the inference system 100 according to this modification may be configured in the same manner as the above embodiment.

[0206] In this modification, the label generation device 1 can generate a third correct label 533 for third training data 531 composed of sensing data using a processing procedure similar to that of the above embodiment. As in the above embodiment, the third training data 531 may be acquired in the source domain or the target domain. For example, the target domain may differ from the source domain due to changes in conditions such as sensing conditions, observation target, sensor settings, sensor installation angle, and background from the source domain. Each inference model (61, 62) may be further trained by adversarial learning between each of the discriminative models (67, 68). In the adversarial learning, each discriminative model (67, 68) may be trained to identify, for each frequency component, which training data the inference results of each inference model (61, 62) correspond to.

[0207] The model generation device 2 can generate a trained third inference model 63 that has acquired the ability to perform an inference task on sensing data using a processing procedure similar to that of the above embodiment. The inference device 3 can perform an inference task on target data 321 composed of sensing data using the trained third inference model 63 using a processing procedure similar to that of the above embodiment. Depending on the content of the inference task, the inference device 3 may be interpreted as, for example, a diagnostic device, a detection device, an identification device, a monitoring device, etc.

[0208] In addition, the inference device 3 may perform output processing according to the inference task in step S303. In one example, the control unit 31 of the inference device 3 may output the inference result as is. In another example, the control unit 31 may perform any information processing according to the inference result. As a specific example, in a case where an inference task regarding the condition of the target person is performed, the control unit 31 may output a warning to notify the target person if it is determined that there is an abnormality in their health condition. As another specific example, in a case where an inference task regarding the condition of the driver is performed, the control unit 31 may perform information processing such as notifying the target person of an abnormality in their health condition or prohibiting switching from automatic driving to manual driving if it is determined that the driver's drowsiness or fatigue level is high. As another specific example, in a case where an inference task regarding the situation outside the vehicle is performed, the control unit 31 may determine an operation command for the vehicle according to the identified situation outside the vehicle and output the determined operation command (for example, executing a temporary stop of the vehicle if a person is detected running out into the road).

[0209] (Features) According to the third modification, the label generation device 1 can generate highly reliable third correct labels 533 at low cost for third training data 531 composed of sensing data. In the model generation device 2, by using the third dataset 53 including the highly reliable third correct labels 533 for machine learning, it is possible to generate a trained third inference model 63 that has acquired the ability to perform inference tasks on sensing data with high accuracy. Furthermore, by using the trained third inference model 63 generated in this way in the inference device 3, it is possible to expect the inference task to be performed with high accuracy on target data 321 composed of sensing data.

[0210] <4.2> In the above embodiment, the third inference model 63 may be newly prepared separately from the first inference model 61 and the second inference model 62. Alternatively, the trained third inference model 63 may be generated by performing additional learning or re-learning on the trained second inference model 62. In other words, the third inference model 63 before machine learning may be configured from the trained second inference model 62. In one example, the machine learning of the first inference model 61, the machine learning of the second inference model 62, the generation of the third correct label 533, and the machine learning of the third inference model 63 may be executed as a series of processes. When the label generation device 1 and the model generation device 2 are configured as an integrated computer, these processes may be executed consecutively.

[0211] <4.3> In the above embodiment, the label generation device 1 may generate pseudo-correct labels for training data to which correct labels have been assigned. In this case, the label generation device 1 or another computer may compare the generated pseudo-correct labels with the correct labels assigned to the training data, and confirm whether the assigned correct labels are correct based on the comparison results. If the generated pseudo-correct labels do not match the assigned correct labels (if they deviate by more than a threshold), the label generation device 1 or another computer may output a warning to notify that the assigned correct labels are suspicious.

[0212] <4.4> In the above embodiment, the input format and output format of each model (61 to 63, 65, 67, 68) may be determined appropriately depending on the embodiment. Each model (61 to 63, 65, 67, 68) may be configured to accept input of information other than the above. Each model (61 to 63, 65, 67, 68) may be configured to output information other than the above.

[0213] <4.5> In the above embodiment, the label generation device 1 may generate the third correct label 533 using three or more trained inference models including the trained first inference model 61 and the trained second inference model 62. In this case, the label generation device 1 may generate the third correct label 533 by obtaining consensus among at least some of the three or more trained inference models. Furthermore, by changing the disturbance to be added, multiple different training data groups (each training data group is composed of multiple second datasets 52) may be generated, and multiple different trained second inference models 62 may be generated by using each training data group for machine learning.

[0214] Furthermore, in the processing procedure of the label generating apparatus 1 according to the above embodiment, the processing of step S107 may be omitted. In this case, the output unit 117 may be omitted from the software configuration of the label generating apparatus 1.

[0215] §5 Working Examples In order to verify the effectiveness of the present invention, the following examples and comparative examples were created. However, the present invention is not limited to the following examples. In the following examples and comparative examples, the data that is the subject of the inference task is first-person perspective image data that shows a hand, and the inference task was set to extract the area that shows the hand.

[0216] (1) First experiment First, we prepared multiple data sets using first-person perspective image data with various styles published in the following reference: Reference 7: Y. Li, M. Liu, and J.M. Rehg, "In the eye of beholder: Joint learning of gaze and actions in first person video," In Proceedings of the European Conference on Computer Vision (ECCV), pages 619-635, 2018. We prepared the first data set (EGTEA). Based on reference 8, "Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M.J. Black, I. Laptev, and C. Schmid, "Learning joint reconstruction of hands and manipulated objects," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11807-11816, 2019," we used software (ObMan) to synthetically render simulated hand images onto first-person perspective image data to prepare a second virtual base dataset (ObMan-Ego).The first-person perspective image data used to generate the second data set was taken from reference 9 "D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, "Rescaling egocentric vision. arXiv preprint", arXiv:2006.13256, 2020" and reference 10 "R. Goyal, S. E. Kahou, V. Michalski, J. Materzyn ´ska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic, 'The "something something" video database for learning and evaluating visual common sense', In Proceedings of The data set was obtained from "A. Fathi, A. Farhadi, and J. Rehg, "Understanding egocentric activities," In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5842-5850, 2017." The third data set (GTEA) was prepared from reference 11 "A. Fathi, A. Farhadi, and J. Rehg, "Understanding egocentric activities," In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 407-414, 2011."The fourth data set (EDSH-2) and the fifth data set (EDSH-K) were prepared by splitting two partial data sets recorded in sparse environments in reference 12 "C. Liand and K. Kitani, "Pixel-level hand detection in egocentric videos," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3570-3577, 2013." The sixth data set (UTG) was prepared from reference 13 "M. Cai, K. Kitani, and Y. Sato, "An ego-vision system for hand grasp analysis," IEEE Transactions on Human-Machine Systems, 47(4):524-535, 2017." The seventh data set (YHG) was prepared based on reference 14, "IM Bullock, T. Feix, and AM Dollar, "The Yale Human Grasping Dataset: Grasp, Object, and Task Data in Household and Machine Shop Environments," The International Journal of Robotics Research (IJRR), 34(3):251-255, 2015." The image data (training data) included in each data set was associated with a mask image indicating the area containing the hand as a ground truth label. The image data size for each data set was resized to 256 × 256 pixels. For the reality-based setting (adapting from a real source domain to multiple real target domains), the first data set was selected as the source domain data. For the virtual-based setting (adapting from a virtual source domain to multiple real target domains), the second data set was selected as the source domain data. For each setting, the third to seventh data sets were selected as the target domain data, respectively.

[0217] In our example, we prepared a first trained inference model and a second trained inference model for each of the real-world and virtual-world settings, as in the above-described embodiment. A trained conversion model was generated using a source domain data set and a target domain data set. Ten images were randomly sampled from each target domain data set, and each of the ten sampled images was used for machine learning of the conversion model. A second dataset was generated using the trained conversion model, adapting the style to the target domain. The generated second dataset was then used for machine learning of the second inference model. Adversarial learning was performed on the first and second inference models using a common discriminative model. The discriminative model was configured to identify the origin of each pixel. Using the resulting first and second trained inference models, regions containing hands were extracted from the image data of each target domain, and a mask image showing the overlapping portions of the extracted regions was generated as a pseudo-ground truth label. The generated pseudo-ground truth labels were then used to perform additional learning on the second trained inference model, generating a third trained inference model (the final trained inference model). For each inference model, we employed RefineNet (Reference 15 "G. Lin, A. Milan, C. Shen, and I.D. Reid, "Refinenet: Multipath refinement networks for high-resolution semantic segmentation," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5168-5177, 2017"). For the discrimination model, we employed a three-layer convolutional neural network with a kernel size of 1. In our example, we generated a third inference model trained in a single-target setting, which adapts to each target domain (the third to seventh data sets) individually, and in a multi-target setting, which adapts to all target domains at once.

[0218] In the first comparative example (Source only), an inference model consisting of RefineNet was prepared and trained using a data set from the source domain. This resulted in a trained inference model according to the first comparative example. In the first comparative example, two versions were prepared: one in which a trained inference model was generated without adaptation to the target domain in both a reality-based setting and a virtual-based setting, and the other in which a trained inference model was generated using a data set from the source domain (corresponding to the second dataset in the above embodiment) after adaptation to the style of the target domain using the trained transformation model in the example.

[0219] In the second comparative example (BDL), a trained inference model was generated using the machine learning method proposed in Reference 16 (Y. Li, L. Yuan, and N. Vasconcelos, "Bidirectional learning for domain adaptation of semantic segmentation," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6929-6938, 2019). Reference 16 proposes a framework for alternately training an image translation model and a segmentation model (inference model) for domain adaptation. To ensure a fair comparison, RefineNet was adopted instead of the network proposed in Reference 16 to configure the inference model for the second comparative example. In the third comparative example (UMA), a trained inference model was generated using the machine learning method proposed in Reference 17, "M. Cai, E. Lu, and Y. Sato, "Generalizing hand segmentation in egocentric videos with uncertainty-guided model adaptation," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14380-14389, 2020." Reference 17 also proposes a domain adaptation method using Bayesian CNN. Furthermore, a discriminator that imposes hand shape constraints on the inference model was added to the third comparative example to obtain a trained inference model for the fourth comparative example (UMA+HS). In the real-world setting, a single-target setting was used to obtain trained inference models for the second to fourth comparative examples. Meanwhile, in the virtual-world setting, a single-target setting was used to obtain trained inference models for the second and third comparative examples. Additionally, a trained inference model for the third comparative example, which further employs style adaptation, was also prepared.

[0220] In the first reference example (Ours w / o FSty), the use of the trained transformation model in the examples was omitted. Except for this, the first reference example (Ours w / o FSty) was configured similarly to the examples. That is, in the first reference example, two trained inference models were generated using a data set from the source domain. Then, the two generated trained inference models were used to obtain pseudo-ground truth labels for image data from the target domain. The subsequent processing in the first reference example is similar to the examples. In the second reference example (Ours w / o CPL), the generation of pseudo-ground truth labels in the examples was omitted. That is, in the second reference example, the third inference model before the additional learning in the examples (in other words, the trained second inference model generated by region extraction training and adversarial learning) was obtained as the final inference model. In the first and second reference examples, the final trained inference models were generated in a multi-target setting.

[0221] In the third reference example (Target only), an inference model consisting of RefineNet was prepared and trained using a data set in the target domain. This resulted in a trained inference model for the third reference example. In the third reference example, trained inference models were generated for both single-target and multi-target settings.

[0222] In each of the examples, comparative examples, and reference examples, the Adam optimizer was used as the optimization algorithm. The learning rate of the first inference model was 5×10 -6 and the learning rate of the second inference model is set to 10 -5 In the multi-target setting, target image data was uniformly sampled from the data set of each target domain. The hyperparameter of the loss of adversarial learning for the extraction error loss was set to 0.8. The threshold for the ratio of overlapping parts when generating pseudo-ground truth labels was also set to 0.8.

[0223] Using the final trained inference models obtained in each of the Examples, Comparative Examples, and Reference Examples, the region containing the hand was extracted from the image data of each target domain. The extraction results were then compared with the true value to calculate the average IoU (Intersection over Union). Table 1 below shows the calculation results of the average IoU for each of the Examples, Comparative Examples, and Reference Examples in a real-world setting. Table 2 also shows the calculation results of the average IoU for each of the Examples, Comparative Examples, and Reference Examples in a virtual-world setting.

[0224] [Table 1] [Table 2]

[0225] As shown in Table 1, in the real-base setting, the Example achieved better extraction results than the Comparative Examples, the First Reference Example, and the Second Reference Example. In particular, the Example achieved better extraction results for the sixth data group (UTG) and the seventh data group (YHG). Furthermore, as shown in Table 2, in the virtual-base setting, the Example achieved better extraction results than the Comparative Examples, the First Reference Example, and the Second Reference Example. These results demonstrate that the present invention can generate highly reliable pseudo-ground truth labels, and that using the generated pseudo-ground truth labels in machine learning can improve the inference accuracy of the inference model. Note that, as shown in Table 2, in the virtual-base setting, performance was poor when style adaptation was not performed. This was presumably due to a large domain shift. In contrast, performance was significantly improved when style adaptation was performed. These results demonstrate that adding disturbances using a trained transformation model is effective in cases such as the virtual-base setting.

[0226] (2) Second experiment In the first experiment, it was estimated that the methods of the second and third comparative examples failed due to the large domain shift in the virtual base setting. Therefore, to verify the sensitivity of the pseudo-labeling method to the degree of domain adaptation, the image data of the second data group before conversion was synthesized with the image data of the second data group after conversion to the style of each target domain. Then, using the image data obtained by the synthesis, the performance of the trained inference models of the above-mentioned Example, the first comparative example, and the third comparative example was verified using the same method (average IoU) as in the first experiment.

[0227] FIG. 16 shows the calculation results of the average IoU for each of the working example, comparative example, and reference example in the second experiment. γ indicates the degree of synthesis. γ = 1 indicates that the image data after style conversion was used for training, as in the first experiment. γ = 0 indicates that the image data before style conversion was used for training as is. The smaller γ, the lower the degree of domain adaptation. In other words, the smaller γ, the greater the difference between the image data of the source domain obtained by synthesis and the target domain.

[0228] As shown in FIG. 16, the performance of the first comparative example gradually decreased as the degree of synthesis decreased. Meanwhile, the performance of the third comparative example significantly decreased when the performance of the first comparative example fell below approximately 40%. This was presumably because low-reliability pseudo-labels were generated when the initial performance of the source network was low, thereby reducing the accuracy of the trained inference model. In contrast, the working example was able to suppress performance degradation even when the domains were significantly different. These results demonstrate that the present invention can generate relatively reliable pseudo-ground-truth labels even when there is a significant difference between the source and target domains, and that using the generated pseudo-ground-truth labels in machine learning can improve the inference accuracy of the inference model. [Explanation of symbols]

[0229] 1...Label generation device, 11...control unit, 12...storage unit, 13...communication interface, 14...External interface, 15...input device, 16...output device, 17...drive, 81...label generation program, 91...storage medium, 111...first model acquisition unit, 112...second model acquisition unit, 113...Data acquisition unit, 114...first inference section, 115...second inference section, 116...generation unit, 117...output unit, 2...Model generation device, 21...control unit, 22...storage unit, 23...communication interface, 24...External interface, 25...input device, 26...output device, 27...drive, 82...model generation program, 92...storage medium, 211...data acquisition unit, 212...learning processing unit, 213...storage processing unit, 3... Reasoning device, 31...control unit, 32...storage unit, 33...communication interface, 34...External interface, 35...input device, 36...output device, 37...drive, 83...inference program, 93...storage medium, 311...acquisition unit, 312...inference unit, 313...output unit, 321...Target data, 51...first dataset, 511...first training data, 513...first correct label, 52...Second dataset, 521...second training data, 523...second correct label, 53...First dataset, 531...First training data, 533...First correct label, 61...First inference model, 62...Second inference model, 63...Third inference model, 65...Transformation model, 67·68...Discrimination model, 71...first learning result data, 72...second learning result data, 73...Third learning result data< / url:>

Claims

1. The computer A step of obtaining a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct answer label indicating a correct answer to an inference task for the first training data; A step of obtaining a trained second inference model generated by machine learning using a plurality of second datasets each consisting of a combination of second training data generated by adding disturbances to the first training data and a second correct answer label indicating the correct answer to the inference task for the second training data; obtaining third training data; Obtaining a first inference result by performing the inference task on the acquired third training data using the trained first inference model; Obtaining a second inference result by performing the inference task on the acquired third training data using the trained second inference model; generating a third ground truth label for the third training data based on a match between the first inference result and the second inference result; To execute Label generation method.

2. the third training data is acquired in a target domain different from the source domain; The label generation method of claim 1 .

3. perturbing the first training data comprises transforming the first training data using a trained transformation model; The trained conversion model is generated by machine learning using the first training data and the third training data so as to acquire the ability to convert the style of the first training data into the style of the third training data. The label generation method according to claim 2 .

4. the first inference model and the second inference model are further trained by adversarial learning between the first inference model and a discriminative model; The adversarial learning training the discriminative model using the first training data and the third training data to discriminate whether an inference result of the first inference model is for the first training data or the third training data; training the first inference model using the first training data and the third training data to reduce the discrimination performance of the discriminative model; training the discriminative model using the second training data and the third training data to discriminate whether an inference result of the second inference model is for the second training data or the third training data; and training the second inference model using the second training data and the third training data to reduce the discrimination performance of the discriminative model; It consists of 4. The label generating method according to claim 2 or 3.

5. The computer further executes a step of outputting the generated third correct label. The label generating method according to any one of claims 1 to 4.

6. the inference task is to extract regions containing features; generating the third ground truth label based on the match, Identifying an overlapping portion between the region extracted as the first inference result and the region extracted as the second inference result; and generating the third correct answer label to indicate the overlapping portion as a correct answer to the inference task if the size of the identified overlapping portion exceeds a threshold; It consists of The label generating method according to any one of claims 1 to 5.

7. the inference task is to identify classes of features contained in the data; generating the third correct label based on the match comprises, when a class identified as the first inference result and a class identified as the second inference result match, generating the third correct label to indicate the matched class. The label generating method according to any one of claims 1 to 6.

8. Each of the training data is composed of image data, the inference task comprises at least one of extracting regions in the image data that contain features and identifying classes of features contained in the image data; The label generating method according to any one of claims 1 to 5.

9. Each of the training data is composed of image data, the inference task comprises extracting regions containing features in the image data; the first inference model and the second inference model are further trained by adversarial learning between the first inference model and a discriminative model; The adversarial learning training the discriminative model using the first training data and the third training data to discriminate, for each pixel, whether the inference result of the first inference model is for the first training data or the third training data; training the first inference model using the first training data and the third training data to reduce the discrimination performance of the discriminative model; training the discriminative model to discriminate, for each pixel, whether an inference result of the second inference model is for the second training data or the third training data, using the second training data and the third training data; and training the second inference model using the second training data and the third training data to reduce the discrimination performance of the discriminative model; It consists of 4. The label generating method according to claim 2 or 3.

10. Each of the training data is composed of sound data, the inference task comprises at least one of extracting regions containing features in the sound data and identifying classes of features contained in the sound data; The label generating method according to any one of claims 1 to 5.

11. Each of the training data is composed of sensing data, the inference task comprises at least one of extracting a region containing a feature in the sensing data and identifying a class of the feature contained in the sensing data; The label generating method according to any one of claims 1 to 5.

12. a first model acquisition unit configured to acquire a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct answer label indicating a correct answer to an inference task for the first training data; a second model acquisition unit configured to acquire a trained second inference model generated by machine learning using a plurality of second datasets each consisting of a combination of second training data generated by adding disturbances to the first training data and a second correct answer label indicating a correct answer to the inference task for the second training data; and a data acquisition unit configured to acquire third training data; a first inference unit configured to obtain a first inference result by performing the inference task on the acquired third training data using the trained first inference model; a second inference unit configured to obtain a second inference result by performing the inference task on the acquired third training data using the trained second inference model; a generator configured to generate a third ground truth label for the third training data based on a match between the first inference result and the second inference result; Equipped with Label generator.

13. On the computer, A step of obtaining a trained first inference model generated by machine learning using a plurality of first datasets, each of which is composed of a combination of first training data in a source domain and a first correct answer label indicating a correct answer to an inference task for the first training data; A step of obtaining a trained second inference model generated by machine learning using a plurality of second datasets each consisting of a combination of second training data generated by adding disturbances to the first training data and a second correct answer label indicating the correct answer to the inference task for the second training data; obtaining third training data; Obtaining a first inference result by performing the inference task on the acquired third training data using the trained first inference model; Obtaining a second inference result by performing the inference task on the acquired third training data using the trained second inference model; generating a third ground truth label for the third training data based on a match between the first inference result and the second inference result; In order to execute Label generator.

Images