Model generation method, inference program, and inference device

Abstract

A model generation device according to one aspect of the present invention performs first training of a first model so that the result of an inference task derived by the first model from a first training sample of first data matches a true value. The model generation device performs second training of a second model so that the result of an inference task derived by the second model from a second training sample of second data matches a true value. The model generation device performs, together with the first training and the second training, third training of the first model, the second model, and a fusion model so that the result of an inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the results of the inference tasks matches a true value.

Description

Model generation method, inference program and inference device

[0001] The present invention relates to a model generation method, an inference program, and an inference device.

[0002] In recent years, research has progressed on multimodal models that are configured to accept input of multiple types of data and derive inference results. For example, Non-Patent Document 1 proposes a method of building a fusion model using a convolutional neural network that fuses two types of data and derives inference results from the fused data.

[0003] Hamid Reza Vaezi Joze, et al. "MMTM: Multimodal Transfer Module for CNN Fusion", [online], [Retrieved March 22, 2024], Internet <URL: https: / / arxiv.org / abs / 1911.08670>

[0004] The present inventors have found that the above-mentioned conventional methods have the following problem. That is, in the machine learning of the fusion model, the conventional method may prioritize features suitable for fusing various data, which may result in the loss of features suitable for inferring each individual data. As a result, it may be difficult to improve the inference accuracy of the fusion model.

[0005] In one aspect, the present invention has been made in view of the above circumstances, and its purpose is to provide a technology for improving the inference accuracy of machine learning models.

[0006] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0007] A model generation method according to one aspect of the present invention is an information processing method in which a computer executes steps of controlling machine learning of a first model, a second model, and a fusion model, and outputting the results of the machine learning. The first model is configured to derive a result of an inference task from first data. The second model is configured to derive the result of the inference task from second data of a type different from the first data. The fusion model is configured to derive the result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning includes performing a first training of the first model so that the result of the inference task derived by the first model from a first training sample of the first data conforms to a true value, performing a second training of the second model so that the result of the inference task derived by the second model from a second training sample of the second data conforms to a true value, and performing a third training of the first model, the second model, and the fusion model together with the first training and the second training so that the result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the result of the inference task conforms to a true value.

[0008] In this configuration, training of the fusion model is performed in parallel with training of each model that derives an inference result for each data individually. This allows the fusion model to acquire the ability to derive an inference result from the features of each data, while each individual model (first model, second model) can acquire the ability to extract features suitable for inferring each data (and thereby derive an inference result). Therefore, with this configuration, by training not only the fusion model but also each individual model in parallel, the possibility of losing features suitable for inferring each data can be reduced. As a result, improved inference accuracy can be expected in the generated trained model.

[0009] In the model generation method according to the above aspect, the inference task may be determining whether or not a predetermined event has occurred. A plurality of datasets may be provided, each of which is configured by combining a first sample group of the first data and a second sample of the second data. The plurality of datasets may include one or more positive datasets associated with a true value indicating the occurrence of the predetermined event. For each of the one or more positive datasets, a likelihood of each of a plurality of first samples included in the first sample group may be calculated using a computational model configured to calculate the likelihood of the occurrence of the predetermined event from the first data. In the machine learning, a first sample with the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more positive datasets may be used as the first training sample.

[0010] When an inference task is to determine whether a predetermined event has occurred and data is obtained in the form of a group of samples (e.g., a group of images, a group of sound samples, etc.), features related to the occurrence of the predetermined event may appear only in some of the samples. For example, when diagnosing a disease, lesions related to the disease typically appear in only some of the medical images. Accordingly, the remaining samples in the group may not exhibit the feature (i.e., the state is the same as when the predetermined event has not occurred). In this case, if the remaining samples are used as positive samples for machine learning, samples equivalent to negative samples may be trained as positive samples, which may result in a deterioration in the inference accuracy of the individual and fusion models. In contrast, this configuration selects first samples based on the likelihood calculated using a computational model, allowing first samples that are likely to exhibit features of the occurrence of the predetermined event to be used as first training samples for machine learning. As a result, deterioration in the inference accuracy of the trained model can be suppressed.

[0011] In the model generation method according to the above aspect, the multiple data sets may further include one or more negative data sets associated with a true value indicating that the predetermined event has not occurred. For each of the one or more negative data sets, the likelihood of each of the multiple first samples included in the first sample group may be calculated using the computational model. In the machine learning, the first sample with the highest calculated likelihood among the multiple first samples included in the first sample group for each of the one or more negative data sets may be used as the first training sample. According to this configuration, among the multiple first samples included in the negative data set, the first sample most likely to be positive (the first sample most likely to lead to an error in inference) can be used as the negative sample for machine learning. This can be expected to improve the inference accuracy of the trained model.

[0012] In the model generation method according to the above aspect, the inference task may be determining whether a predetermined event has occurred. A plurality of data sets may be provided, each of which is configured by a combination of a first sample group of the first data and a second sample of the second data. In the machine learning, each of a plurality of first samples included in the first sample group of each of the data sets may be used as the first training sample, and the second sample may be used as the second training sample. If the true value of the inference task associated with the first sample does not match the true value of the inference task associated with the second sample, performing the third training may be omitted.

[0013] As described above, when the inference task is to determine whether a predetermined event has occurred and data is obtained in the form of a sample group, using a sample equivalent to a negative sample as a positive sample in machine learning may result in a deterioration in the inference accuracy of the model. In contrast, with this configuration, if the associated true values ​​do not match, training of the fusion model can be omitted, thereby avoiding training a sample equivalent to a negative sample as a positive sample. As a result, deterioration in the inference accuracy of the trained model can be suppressed.

[0014] Even if the machine learning of one of the first model and the second model is omitted, the machine learning of the individual model and the fusion model can be performed. The fusion model can acquire the ability to derive an inference result, and the other individual model can acquire the ability to extract features suitable for inference of individual data. Therefore, in the model generation method according to each of the above aspects, the machine learning configuration of one of the first model and the second model may be omitted.

[0015] For example, a model generation method according to one aspect of the present invention may be an information processing method in which a computer executes steps of controlling machine learning of a first model and a fusion model and outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. A second model may be provided that is configured to derive a result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; and, together with the first training, training the first model and the fusion model so that a result of the inference task derived by the fusion model from a calculation output of the first model when deriving the result of the inference task and a calculation output of the second model obtained by providing the second model with a second training sample of the second data conforms to a true value. According to this configuration, training of the fusion model is performed together with individual training of the first model. This is expected to improve the inference accuracy of trained models.

[0016] Furthermore, the present invention is not limited to the above-described model generation. One aspect of the present invention may be an inference program that uses each individual model and the fusion model. In one example, at least one of each individual model and the fusion model may be generated by the above-described model generation method.

[0017] For example, an inference program according to one aspect of the present invention may be a program for causing a computer to execute the steps of acquiring a first sample of first data and a second sample of second data of a type different from the first data, deriving a result of an inference task from the acquired first sample and the second sample using a first model, a second model, and a fusion model, and outputting information related to the derived result of the inference task. The step of deriving the result of the inference task may include deriving a first interim result of the inference task from the first sample using the first model, deriving a second interim result of the inference task from the second sample using the second model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result using the fusion model, and deriving the result of the inference task in accordance with the derived first, second, and third interim results. According to this configuration, the robustness of the inference can be increased by ensembling the inference results of each individual model (first model, second model) and the fusion model, which is expected to improve the inference accuracy.

[0018] In the inference program according to the above aspect, the first data may be sensing data obtained by a sensor, and the second data may be data other than the sensing data. With this configuration, improvement in inference accuracy can be expected in situations where sensing data and other types of data other than sensing data are used as the first data and the second data.

[0019] In the inference program according to the above aspect, the first data may be medical image data of a subject, and the second data may be response data to a medical interview given to the subject. The inference task may include determining whether or not the subject has a disease. This configuration can be expected to improve inference accuracy when determining whether or not the subject has a disease.

[0020] In the inference program according to the above aspect, the first data may be image data of an object or sound data related to the object, and the second data may be work history data related to the object. The inference task may include determining whether or not an abnormality has occurred in the object. This configuration can be expected to improve inference accuracy in anomaly detection situations.

[0021] In the inference program according to the above aspect, the first data may be image data of a subject or audio data relating to the subject, and the second data may be text data relating to the subject. The inference task may include inferring emotions of the subject. This configuration can be expected to improve inference accuracy when inferring emotions of the subject.

[0022] In the inference program according to the above aspect, the first data may be first sensing data obtained by a first sensor, and the second data may be second sensing data obtained by a second sensor of a different type from the first sensor. With this configuration, improvement in inference accuracy can be expected in a situation where different types of sensing data are used as the first data and the second data.

[0023] In the inference program according to the above aspect, the first data may be image data of an object in a chemical plant, and the second data may be numerical data relating to an environment of the object in the chemical plant. The inference task may include inferring a state of the object. With this configuration, improved inference accuracy can be expected when inferring the state of the object in a chemical plant.

[0024] In the inference program according to the above aspect, the first data may be image data of an object, and the second data may be point cloud data obtained by measuring the object. The inference task may include inferring the position of the object. This configuration can be expected to improve inference accuracy when inferring the position of the object.

[0025] In the inference program according to the above aspect, the first data may be image data of an equipment, and the second data may be sound data related to the equipment. The inference task may include inferring a state of the equipment. With this configuration, improved inference accuracy can be expected when inferring the state of the equipment.

[0026] In the inference program according to the above aspect, the first data may be image data relating to a crop, and the second data may be meteorological data relating to an environment of the crop. The inference task may include inferring a condition of the crop. This configuration is expected to improve inference accuracy when inferring the condition of the crop.

[0027] Note that the present invention is not limited to the above-described model generation method (information processing method). As another aspect of the model generation method according to each of the above aspects, one aspect of the present invention may be an information processing device (model generation device) that realizes all or part of the above-described configurations, a program, or a storage medium readable by a machine such as a computer on which such a program is stored. A storage medium readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. Similarly, the present invention is not limited to the above-described inference program. As another aspect of the inference program according to each of the above aspects, one aspect of the present invention may be an information processing device (inference device) that realizes all or part of the above-described configurations, an information processing method (inference method), or a storage medium readable by a machine such as a computer on which a program is stored.

[0028] For example, a model generation device according to one aspect of the present invention may include a control unit. The control unit may be configured to execute a step of controlling machine learning of a first model, a second model, and a fusion model, and a step of outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. The second model may be configured to derive a result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; performing a second training of the second model so that a result of the inference task derived by the second model from a second training sample of the second data conforms to a true value; and performing a third training of the first model, the second model, and the fusion model, together with the first training and the second training, so that a result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the result of the inference task conforms to a true value.

[0029] Furthermore, for example, a model generation device according to an aspect of the present invention may include a control unit. The control unit may be configured to execute a step of controlling machine learning of a first model and a fusion model and a step of outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. A second model may be provided that is configured to derive a result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; and, together with the first training, training the first model and the fusion model so that a result of the inference task derived by the fusion model from a calculation output of the first model when deriving the result of the inference task and a calculation output of the second model obtained by providing the second model with a second training sample of the second data conforms to a true value.

[0030] Also, for example, an inference device according to one aspect of the present invention may include a control unit. The control unit may be configured to execute the steps of acquiring first samples of first data and second samples of second data of a type different from the first data, deriving a result of an inference task from the acquired first and second samples using a first model, a second model, and a fusion model, and outputting information related to the derived result of the inference task. The step of deriving the result of the inference task may include deriving a first interim result of the inference task from the first sample using the first model, deriving a second interim result of the inference task from the second sample using the second model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result using the fusion model, and deriving the result of the inference task based on the derived first, second, and third interim results.

[0031] Furthermore, for example, an inference method according to one aspect of the present invention may be an information processing method in which a computer executes the steps of acquiring a first sample of first data and a second sample of second data of a type different from the first data, deriving a result of an inference task from the acquired first sample and the second sample using a first model, a second model, and a fusion model, and outputting information related to the derived result of the inference task. The step of deriving a result of the inference task may include deriving a first interim result of the inference task from the first sample using the first model, deriving a second interim result of the inference task from the second sample using the second model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result using the fusion model, and deriving the result of the inference task in accordance with the derived first, second, and third interim results.

[0032] According to the present invention, it is possible to expect improvement in inference accuracy.

[0033] FIG. 1 schematically shows an example of a scenario in which the present invention is applied. FIG. 2A schematically shows an example of each data configuration according to an embodiment. FIG. 2B schematically shows an example of each data configuration according to an embodiment. FIG. 2C schematically shows an example of each data configuration according to an embodiment. FIG. 3A schematically shows an example of a 1-1 case in which the present invention is applied. FIG. 3B schematically shows an example of a 1-2 case in which the present invention is applied. FIG. 3C schematically shows an example of a 1-3 case in which the present invention is applied. FIG. 3D schematically shows an example of a 1-4 case in which the present invention is applied. FIG. 3E schematically shows an example of a 1-5 case in which the present invention is applied. FIG. 4A schematically shows an example of a 2-1 case in which the present invention is applied. FIG. 4B schematically shows an example of a 2-2 case in which the present invention is applied. FIG. 4C schematically shows an example of a 2-3 case in which the present invention is applied. FIG. 4D schematically shows an example of a 2-4 case in which the present invention is applied. FIG. 4E schematically illustrates an example of Case 2-5 to which the present invention is applied. FIG. 4F schematically illustrates an example of Case 2-6 to which the present invention is applied. FIG. 4G schematically illustrates an example of Case 2-7 to which the present invention is applied. FIG. 4H schematically illustrates an example of Case 2-8 to which the present invention is applied. FIG. 4I schematically illustrates an example of Case 2-9 to which the present invention is applied. FIG. 4J schematically illustrates an example of Case 2-10 to which the present invention is applied. FIG. 5A schematically illustrates an example of Case 3-1 to which the present invention is applied. FIG. 5B schematically illustrates an example of Case 3-2 to which the present invention is applied. FIG. 6A schematically illustrates an example of a first method of selecting samples according to an embodiment. FIG. 6B schematically illustrates an example of a second method of selecting samples according to an embodiment. FIG. 7 schematically illustrates an example of the hardware configuration of a model generation device according to an embodiment. FIG. 8 schematically illustrates an example of the hardware configuration of an inference device according to an embodiment. Fig. 9 schematically shows an example of the software configuration of a model generation device according to an embodiment. Fig. 10 schematically shows an example of the software configuration of an inference device according to an embodiment. Fig. 11 is a flowchart showing an example of the processing procedure of a model generation device according to an embodiment. Fig. 12 is a flowchart showing an example of the processing procedure of an inference device according to an embodiment. Fig. 13 schematically shows an example of a machine learning process according to a modified example.FIG. 14A schematically shows an example of the configuration of a fusion model according to a modified example. FIG. 14B schematically shows an example of the configuration of a fusion model according to a modified example. FIG. 15A shows the model configuration of a first comparative example. FIG. 15B shows the model configuration of a second comparative example. FIG. 15C shows the model configuration of a third comparative example. FIG. 15D shows the model configuration of a fourth comparative example. FIG. 15E shows the model configuration of a sixth comparative example. FIG. 16A shows the model configuration of a first example. FIG. 16B shows the model configuration of a second example. FIG. 16C shows the model configuration of a third example. FIG. 16D shows the model configuration of a fourth example. FIG. 16E shows the model configuration of a fifth example. FIG. 16F shows the model configuration of a sixth example. FIG. 17 shows the results of the first experiment. FIG. 18A shows the results of the second experiment. FIG. 18B shows the calculation results of the ROC curves for each comparative example and each example in the second experiment. Fig. 18C shows the evaluation results of each model of Example 1 in Experiment 2A. Fig. 18D shows heat maps generated for each comparative example and each Example. Fig. 19 shows the results of Experiment 2B.

[0034] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it is specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0035] §1 Application Example Fig. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. The system according to this embodiment comprises a model generation device 1 and an inference device 2.

[0036] The model generation device 1 according to this embodiment is one or more computers configured to generate a trained model by controlling the implementation of machine learning.

[0037] The model generation device 1 controls the machine learning of a first model 51, a second model 52, and a fusion model 53. The first model 51 is configured to derive a result of an inference task (inference result) from first data 30. The second model 52 is configured to derive a result of the inference task from second data 35 of a type different from the first data 30. The fusion model 53 is configured to receive input of the calculation output of the first model 51 and the calculation output of the second model 52, and to derive a result of the inference task from the input calculation output of the first model 51 and the calculation output of the second model 52.

[0038] The machine learning includes performing a first training on the first model 51 so that the result of the inference task derived by the first model 51 from the first training sample 300 of the first data 30 conforms to the true value, performing a second training on the second model 52 so that the result of the inference task derived by the second model 52 from the second training sample 350 of the second data 35 conforms to the true value, and performing a third training on the first model 51, the second model 52 and the fusion model 53 together with the first and second training so that the result of the inference task derived by the fusion model 53 from the calculation output of the first model 51 and the calculation output of the second model 52 when deriving the result of the inference task conforms to the true value.

[0039] In one example, the third training may be performed in parallel with the first training and the second training. Furthermore, the calculation output of the first model 51 when deriving the result of the inference task may be the calculation output of the first model 51 when deriving the result of the inference task from the first training samples 300. The calculation output of the second model 52 when deriving the result of the inference task may be the calculation output of the second model 52 when deriving the result of the inference task from the second training samples 350. By performing this machine learning, a trained first model 51, a trained second model 52, and a trained fusion model 53 are generated. The model generation device 1 outputs the results of this machine learning.

[0040] Meanwhile, the inference device 2 according to this embodiment is one or more computers configured to perform an inference task using a first model 55, a second model 56, and a fusion model 57. The inference device 2 acquires a first sample 305 of the first data 30 and a second sample 355 of the second data 35. The inference device 2 derives a result of the inference task from the acquired first sample 305 and second sample 355 using the first model 55, the second model 56, and the fusion model 57.

[0041] The first model 55 is configured to derive a result of an inference task from the first data 30. In one example, the first model 55 may be a trained first model 51 generated by the model generation device 1. The second model 56 is configured to derive a result of an inference task from second data 35 of a type different from the first data 30. In one example, the second model 56 may be a trained second model 52 generated by the model generation device 1. The fusion model 57 is configured to receive input of the calculation output of the first model 55 and the calculation output of the second model 56, and to derive a result of the inference task from the input calculation output of the first model 55 and the calculation output of the second model 56. In one example, the fusion model 57 may be a trained fusion model 53 generated by the model generation device 1.

[0042] Deriving the result of the inference task includes deriving a first interim result of the inference task from the first sample 305 using the first model 55, deriving a second interim result of the inference task from the second sample 355 using the second model 56, deriving a third interim result of the inference task from the calculation output of the first model 55 obtained in the process of deriving the first interim result and the calculation output of the second model 56 obtained in the process of deriving the second interim result using the fusion model 57, and deriving a result of the inference task based on the derived first, second, and third interim results. This calculation derives the result of the inference task (inference result). The inference device 2 outputs information related to the derived result of the inference task.

[0043] In the model generation device 1 according to this embodiment, training of the fusion model 53 is performed in parallel with training of each model (51, 52) that derives an inference result for each data (30, 35) individually. This allows the fusion model 53 to acquire the ability to derive an inference result from the features of each data (30, 35), while allowing each model (51, 52) to acquire the ability to extract features suitable for inferring each data (30, 35). As a result, it is expected that the possibility of impairing the features suitable for inferring each data (30, 35) can be reduced. Therefore, according to this embodiment, it is expected that the inference accuracy of the generated trained models (51, 52, 53) can be improved.

[0044] Furthermore, by training each model (51, 52) together with the fusion model 53, each model (51, 52) can reflect knowledge of the other's data in its own training. That is, thanks to the presence of the fusion model 53, the first model 51 can be trained to acquire the ability to extract features from the first data 30 while using the features of the second data 35 as clues and derive an inference result from the extracted features. The second model 52 can be trained to acquire the ability to extract features from the second data 35 while using the features of the first data 30 as clues and derive an inference result from the extracted features. As a result of this training, each model (51, 52) can more accurately extract features suitable for inferring each data (30, 35), which is expected to improve the inference accuracy of each model (51, 52).

[0045] Furthermore, in the inference device 2 according to this embodiment, the robustness of the inference can be increased by ensembling the inference results of the first model 55, the second model 56, and the fusion model 57. This is expected to improve the inference accuracy. Note that when the trained first model 51 is used as the first model 55, the trained first model 51 may be provided to the inference device 2 from the model generation device 1 at any timing and by any method. The same applies to the trained second model 52 and the fusion model 53.

[0046] [Inference Task] The inference task may include at least one of regressing and identifying features that appear in data. The inference task may also include prediction. The type of inference task is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the inference task may include diagnosing a disease, detecting an abnormality, detecting a location, distinguishing an emotion, evaluating a state, evaluating a value, distinguishing a type, etc.

[0047] [First Data / Second Data] The first data 30 and the second data 35 are at least a part of the input data (explanatory variables) used for inference. The first data 30 and the second data 35 are not particularly limited and may be selected appropriately depending on the inference task (objective variable).

[0048] The types of the first data 30 and the second data 35 are not particularly limited and may be selected appropriately depending on the embodiment. In one example, the first data 30 and the second data 35 may each be image data, sound data, text data, numerical data, or a combination thereof. The images may be either still images or videos. The first data 30 and the second data 35 may each be composed of raw data, or may be composed of analysis data (analysis results) obtained by analyzing raw data. The analysis may include, for example, extraction of data features, calculation of feature amounts, etc. The feature extraction may include, for example, inference (regression or identification) such as voice analysis and emotion identification. The feature amounts may include, for example, statistics such as maximum value, minimum value, median, mean value, variance, standard deviation, and n-th percentile value.

[0049] The configurations of the first data 30 and the second data 35 are not particularly limited and may be selected appropriately depending on the embodiment. In one example, at least one of the first data 30 and the second data 35 may be sensing data obtained by one or more sensors. The type of sensor may be selected appropriately depending on the embodiment, such as the inference task. The sensor may include, for example, an image sensor, a microphone, an acceleration sensor, an angular acceleration sensor, a gyro sensor, a motion capture device, a pressure gauge, a thermometer, a hygrometer, a blood pressure monitor, a pulse meter, a heart rate monitor, an electrocardiograph, an electromyograph, a clinical thermometer, a skin electrodermal response monitor, an electroencephalograph, a magnetoencephalograph, an activity monitor, a blood glucose monitor, an electrooculography sensor, an eye movement monitor, an odor sensor, a wearable device (such as a smart watch), a position sensor, a microwave sensor, an ultrasonic sensor, a weather sensor, a home security sensor, etc. The image sensor may include, for example, any sensor that acquires data in the form of an image or an image representation, such as an RGB camera, a depth sensor, an infrared sensor, radar, or LiDAR (light detection and ranging). The location sensor may include, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, etc. The home security sensor may include, for example, an image sensor, an infrared sensor, an activity (sound) sensor, a gas (CO ２ The data may include sensors (such as a time sensor), a current sensor, a smart meter (a sensor that measures the power consumption of home appliances, lighting, etc.), etc. The sensors may be arranged appropriately depending on the object being observed. In one example, at least one of the first data 30 and the second data 35 may be data other than sensing data. Data other than sensing data may include, for example, manually generated information such as user responses, inputs, operations, and history, and information generated by computer processing. Data obtained by observing user responses using a sensor such as an image sensor may be considered sensing data or may be considered data other than sensing data. In one example of this embodiment, at least one of the following three configurations may be adopted for the configuration of the first data 30 and the second data 35.

[0050] (1) First Case Example FIG. 2A schematically illustrates an example of the configuration of the first data 30 and the second data 35 in a first case example of this embodiment. As illustrated in FIG. 2A, in one example, the first data 30 may be sensing data SD obtained by a sensor S. As described above, the type of sensor S may be selected as appropriate depending on the embodiment, such as the inference task. Meanwhile, the second data 35 may be data AD other than the sensing data SD. The second data 35 may be selected as appropriate from data other than the sensing data SD depending on the inference scenario.

[0051] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the sensing data SD and a plurality of second training samples 350 of data AD other than the sensing data SD. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, in a situation where the sensing data SD and other types of data AD other than the sensing data SD are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0052] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the sensing data SD and a second sample 355 of data AD other than the sensing data SD. The inference device 2 may derive a result of the inference task from the acquired first sample 305 and second sample 355. According to one example of the present embodiment, in a situation where the sensing data SD and other types of data AD other than the sensing data SD are used as the first data 30 and the second data 35, improvement in inference accuracy can be expected by ensembling the inference results of each model (55, 56, 57).

[0053] (2) Second Case Example Fig. 2B schematically illustrates an example of the configuration of the first data 30 and the second data 35 in a second case example of this embodiment. As illustrated in Fig. 2B, in one example, the first data 30 may be first sensing data SD1 obtained by a first sensor S1, and the second data 35 may be second sensing data SD2 obtained by a second sensor S2. As described above, the type of each sensor (S1, S2) may be appropriately selected depending on the embodiment, such as the inference task.

[0054] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the first sensing data SD1 and a plurality of second training samples 350 of the second sensing data SD2. The model generation device 1 may use the acquired plurality of first training samples 300 and a plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, in a situation where different types of sensing data (SD1, SD2) are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0055] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the first sensing data SD1 and a second sample 355 of the second sensing data SD2. The inference device 2 may derive a result of the inference task from the acquired first sample 305 and second sample 355. According to one example of the present embodiment, in a situation where different types of sensing data (SD1, SD2) are used as the first data 30 and the second data 35, improvement in inference accuracy can be expected by ensembling the inference results of the respective models (55, 56, 57).

[0056] (3) Third Case Example FIG. 2C schematically illustrates an example of the configuration of the first data 30 and the second data 35 in a third case example of this embodiment. As illustrated in FIG. 2C , in one example, the first data 30 may be data AD1 other than sensor data, and the second data 35 may be data AD2 other than sensor data that is different from the data AD1. The type of each data (AD1, AD2) may be appropriately selected from data other than sensor data. The origin of each data (AD1, AD2) may be appropriately selected depending on the inference content, etc.

[0057] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the data AD1 and a plurality of second training samples 350 of the data AD2. The model generation device 1 may use the acquired plurality of first training samples 300 and a plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, in a situation where data (AD1, AD2) other than sensing data and different types of data (AD1, AD2) are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0058] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the data AD1 and a second sample 355 of the data AD2. The inference device 2 may derive a result of the inference task from the acquired first sample 305 and second sample 355. According to one example of the present embodiment, in a situation where different types of data (AD1, AD2) other than sensing data are used as the first data 30 and the second data 35, ensembling the inference results of the models (55, 56, 57) can be expected to improve the inference accuracy.

[0059] As described above, the sensing data (SD, SD1, SD2) and the data other than the sensing data (AD, AD1, AD2) may be used as the respective data (30, 35) as raw data, or may be converted into analysis data and then used as the respective data (30, 35). Furthermore, the configurations of the first data 30 and the second data 35 are not limited to the above three examples, and may be changed as appropriate depending on the embodiment.

[0060] (Relationship of Information Sources) The individuals from which each sample (305, 355) is obtained in the inference stage may or may not be included in the individuals from which each training sample (300, 350) is obtained in the learning stage. That is, the target individuals at the time of inference may or may not match the individuals at the time of learning. The relationship between the information sources (individuals from which samples are obtained) at the time of learning and inference is not particularly limited and may be determined appropriately depending on the embodiment. Furthermore, as long as the trained models (51, 52, 53) generated by the model generation device 1 are applicable to the inference stage, the type of target individuals at the time of inference does not necessarily match the type of individuals from which training samples are obtained at the time of learning. Applicability to the inference stage may mean that the inference accuracy does not deteriorate to the extent that the trained models (51, 52, 53) become unusable.

[0061] [Example of a scenario] The system according to this embodiment may be applied to various scenarios in which any inference task is performed. The application of this embodiment may include at least one of the first, second, and third scenarios described above. Specific application scenarios are exemplified below for each scenario.

[0062] (1) First Case Example The first case example shown in FIG. 2A can be applied to various situations in which sensing data and data other than sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following five situations. However, the application of this embodiment is not limited to the following five examples and may be determined appropriately depending on the embodiment.

[0063] (1-1) Case 1-1 FIG. 3A schematically illustrates an example of Case 1-1 to which this embodiment is applied. Case 1-1 is an example of a situation in which this embodiment is applied to medical diagnosis. As shown in FIG. 3A , in one example, the first data 30 may be medical image data SD10 of a subject, and the second data 35 may be response data AD10 to a medical interview given to the subject. The inference task may include determining whether or not a disease has occurred in the subject.

[0064] The medical image data SD10 is an example of sensing data SD. As long as the medical image data SD10 can be used for image diagnosis, the type of the medical image data SD10 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the medical image data SD10 may be composed of an ultrasound (echo) image, an X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a PET (Positron Emission Tomography) image, a bone scintigraphy image, or a combination thereof. One or more sensors S10 may be used to acquire the medical image data SD10. The type of the sensor S10 may be appropriately selected depending on the type of medical image data SD10. The one or more sensors S10 may be selected from, for example, an ultrasound diagnostic device, an X-ray imaging device, a CT device, an MRI device, a PET device, a scintigraphy device, etc.

[0065] The response data AD10 is an example of data AD other than the sensing data SD. As long as it can be used for diagnosis, the method of acquiring the response data AD10 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. In one example, the response data AD10 may be acquired by the subject inputting answers to a medical questionnaire. In another example, the response data AD10 may be acquired by input by a medical professional such as a doctor or nurse. In yet another example, the response data AD10 may be acquired from medical information such as an electronic medical record. The content of the medical interview may be appropriately determined depending on the embodiment, such as the disease to be inferred.

[0066] As long as it is possible to determine whether or not a disease has occurred, the format of the determination result is not particularly limited and may be determined appropriately depending on the embodiment. In one example, determining whether or not a disease has occurred may be configured by regressing the probability of a specific disease occurring. Determining whether or not a disease has occurred may include identifying the type of disease that has occurred. Furthermore, determining whether or not a disease has occurred may be configured by regressing the probability of disease occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether or not only a specific disease has occurred, or may be configured generally to determine whether or not any disease has occurred. The type of disease to be inferred may be selected depending on the embodiment.

[0067] In the learning stage, the model generating device 1 may acquire a plurality of first training samples 300 of medical image data SD10 and a plurality of second training samples 350 of medical interview response data AD10 for a subject SS10. The subject SS10 is an example of a target person and an example of an individual during learning. The number of subjects SS10 may be determined appropriately depending on the embodiment. The model generating device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when diagnosing the presence or absence of a disease in a subject from the medical image data SD10 and the medical interview response data AD10, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0068] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of medical image data SD10 and a second sample 355 of medical interview response data AD10 for a subject TT10 to be diagnosed. The subject TT10 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (determining the presence or absence of a disease) from the acquired first sample 305 and second sample 355. The inference device 2 may infer the lesion site in the medical image data SD10 by referring to the progress of feature extraction for the medical image data SD10 (first sample 305) in the first model 55 and the fusion model 57. The progress of feature extraction may be referenced by any method. A known method may be adopted as the method for referencing the progress of feature extraction. The inference device 2 may output information indicating the result of determining the presence or absence of a disease to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, information indicating the result of determining whether or not a disease exists may be provided to at least one of the subject TT10 and a medical professional as medical assistance information. According to one example of the present embodiment, when diagnosing whether or not a subject has a disease based on the medical image data SD10 and the medical interview response data AD10, by ensembling the inference results (disease presence / absence determination results) of each model (55, 56, 57), it is possible to expect improvement in inference accuracy.

[0069] The application of this embodiment to a medical diagnosis scenario need not be limited to the form shown in FIG. 3A and may be modified as appropriate depending on the embodiment. In another example, the medical image data SD10 may be replaced with other medical data. For example, the other medical data may include biological data such as blood pressure, pulse, heart rate, electrocardiogram, electromyogram, body temperature, electrodermal response, brain activity (electroencephalography, magnetoencephalography), activity level, blood glucose level, electrooculography, eye movement, and odor. The sensor used to acquire the other medical data may be selected as appropriate depending on the type of the other medical data. In yet another example, this embodiment may be applied to the scenario of Case 2 above by replacing the response data AD10 with the other medical data.

[0070] (1-2) Case 1-2 FIG. 3B schematically illustrates an example of Case 1-2 to which this embodiment is applied. Case 1-2 is an example of a scenario in which this embodiment is applied to detecting an abnormality in an object in a factory. As shown in FIG. 3B, in one example, the first data 30 may be image data SD111 depicting the object or sound data SD112 related to the object. The second data 35 may be work history data AD11 related to the object. The inference task may include determining whether or not an abnormality has occurred in the object.

[0071] The image data SD111 or the sound data SD112 is an example of sensing data SD. As long as it can be used for abnormality detection, the type of each of the image data SD111 and the sound data SD112 is not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S11 may be used to acquire the image data SD111 or the sound data SD112. The type of the sensor S11 may be selected appropriately depending on the embodiment. The sensor S11 may be selected from, for example, an image sensor, a microphone, etc.

[0072] The work history data AD11 is an example of data AD other than the sensing data SD. As long as it can be used for anomaly detection, the method of acquiring the work history data AD11 and its data format are not particularly limited and may be appropriately selected depending on the embodiment. In one example, the work history data AD11 may include the work history of workers in a factory, the operation history of equipment, etc. The work history data may be composed of text data, numerical data, etc. At least a portion of the work history data AD11 may be acquired by input by a worker, or may be acquired automatically from the operation results of equipment, etc.

[0073] As long as it is possible to determine whether an abnormality has occurred, the format of the determination result is not particularly limited and may be determined appropriately depending on the embodiment. In one example, determining whether an abnormality has occurred may be configured by at least one of identifying whether an abnormality has occurred and regressing the probability that a specific abnormality has occurred. Determining whether an abnormality has occurred may include identifying the type of abnormality that has occurred. Furthermore, determining whether an abnormality has occurred may be configured by regressing the probability of abnormality occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured generally to determine whether any abnormality has occurred.

[0074] The object may be selected as appropriate depending on the embodiment. In one example, the object may include at least one of a device constituting factory automation (FA) and a product produced by the FA. The product produced by the FA may include at least one of an intermediate product and a final product. The abnormality may include at least one of a device failure and a product defect.

[0075] When a device constituting an FA is used as the target, the image data SD111 may be acquired to depict a scene of the device operating, etc. The sound data SD112 may be acquired to include the operating sounds, vibration sounds, etc. of the device. When a product produced by the FA is used as the target, the image data SD111 may be acquired to depict the target range of the product (the range in which anomalies are detected). The sound data SD112 may be acquired to include the sounds of the product itself, the operating sounds of devices involved in the production of the product, the vibration sounds of the devices, etc.

[0076] In the learning stage, the model generation device 1 may acquire, for the object SS11, a plurality of first training samples 300 of the image data SD111 or the sound data SD112 and a plurality of second training samples 350 of the work history data AD11. The object SS11 is an example of an individual during learning. The number of objects SS11 from which each training sample (300, 350) is obtained may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when detecting the presence or absence of an abnormality in an object from the image data SD111 or the sound data SD112 and the work history data AD11, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0077] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD111 or the sound data SD112 and a second sample 355 of the work history data AD11 for the target object TT11 for which an abnormality is detected. The target object TT11 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (determining whether or not an abnormality has occurred) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of determining whether or not an abnormality has occurred to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of determining whether or not an abnormality has occurred may be provided to at least one of a worker and a work supervisor present near the target object TT11. The output device may include a terminal of the worker or the supervisor. According to one example of this embodiment, when detecting whether or not an object has an abnormality from image data SD111 or sound data SD112 and work history data AD11, the inference results (determination results of whether or not an abnormality exists) of each model (55, 56, 57) can be ensembled, thereby improving the accuracy of inference.

[0078] Note that the method of applying this embodiment to a scene of abnormality detection in a factory is not limited to the form of FIG. 3B and may be modified as appropriate depending on the embodiment. In another example, both the image data SD111 and the sound data SD112 may be used as the first data 30. In another example, this embodiment may be applied to the scene of Case 2 above by using one of the image data SD111 and the sound data SD112 as the first data 30 and the other as the second data 35. In this example, the work history data AD11 may be omitted.

[0079] (1-3) Case 1-3 FIG. 3C schematically illustrates an example of Case 1-3 to which this embodiment is applied. Case 1-3 is an example of a scenario in which this embodiment is applied to road anomaly detection. As shown in FIG. 3C, in one example, the first data 30 may be image data SD12 depicting a road, and the second data 35 may be text data AD12 relating to road conditions. The inference task may include determining whether an anomaly has occurred on the road.

[0080] The image data SD12 is an example of sensing data SD. As long as it can be used to detect anomalies, the type and acquisition method of the image data SD12 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S12 may be used to acquire the image data SD12. The sensor S12 may be, for example, an image sensor. The method by which the sensor S12 observes the road is not particularly limited and may be selected appropriately depending on the embodiment. The sensor S12 may be placed at a fixed point or on any mobile object (vehicle, drone, satellite, etc.).

[0081] The text data AD12 is an example of data AD other than the sensing data SD. As long as the text data AD12 can be used for anomaly detection, the method of acquiring the text data AD12 is not particularly limited and may be appropriately selected depending on the embodiment. For example, the text data AD12 may include at least one of information provided by a specialized institution such as a road traffic center and information provided by ordinary people via a social networking service (SNS). Furthermore, as long as the text data AD12 includes information about road conditions, the content of the text data AD12 is not particularly limited and may be appropriately determined depending on the embodiment.

[0082] As long as it is possible to determine whether an abnormality has occurred, the format of the determination result is not particularly limited and may be determined appropriately depending on the embodiment. In one example, determining whether an abnormality has occurred may include at least one of identifying whether an abnormality has occurred on the road and regressing the probability that a specific abnormality has occurred. Determining whether an abnormality has occurred may include identifying the type of abnormality that has occurred. Furthermore, determining whether an abnormality has occurred may include regressing the probability of abnormality occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured for general purposes to determine whether any abnormality has occurred. Road abnormalities may include, for example, road damage, fallen objects, dirt on the road surface, accidents, etc. Road damage may include, for example, holes, depressions, collapsed shoulders, etc. If a road abnormality includes a fallen object, the type of the abnormality may include the type of the fallen object. Determining the type of the abnormality may include determining the type of the fallen object.

[0083] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD12 and a plurality of second training samples 350 of the text data AD12 for a target range SS12 of a road. The target range SS12 of the road is an example of an individual during learning. The size and number of the target range SS12 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, improved inference accuracy can be expected in the generated trained models (51, 52, 53) when detecting the presence or absence of an abnormality on a road from the image data SD12 and the text data AD12.

[0084] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD12 and a second sample 355 of the text data AD12 for a target range TT12 of a road for which an abnormality is to be detected. The target range TT12 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (determining whether an abnormality has occurred) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of determining whether an abnormality has occurred on the road to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the information indicating the result of determining whether an abnormality has occurred may be provided to an arbitrary user, such as an administrator who manages the target range TT12 of the road. The output device may include a user terminal. According to one example of the present embodiment, when detecting the presence or absence of an abnormality on a road from the image data SD12 and the text data AD12, improving the accuracy of inference can be expected by ensembling the inference results (determination results of the presence or absence of an abnormality) of each model (55, 56, 57).

[0085] (1-4) Case 1-4 FIG. 3D schematically illustrates an example of Case 1-4 to which this embodiment is applied. Case 1-4 is an example of a scene in which this embodiment is applied to inferring the emotions of a subject. As shown in FIG. 3D, in one example, the first data 30 may be image data SD131 depicting the subject or sound data SD132 related to the subject. The second data 35 may be text data AD13 related to the subject. The inference task may include inferring the emotions of the subject.

[0086] The image data SD131 or the sound data SD132 is an example of sensing data SD. As long as it can be used for emotion inference, the type and acquisition method of the image data SD131 and the sound data SD132 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S13 may be used to acquire the image data SD131 or the sound data SD132. The type of the sensor S13 may be selected appropriately depending on the embodiment. The sensor S13 may be selected from, for example, an image sensor, a microphone, etc. In one example, the image data SD131 may be acquired so as to capture at least a portion of the subject's face. In another example, the sound data SD132 may include the subject's voice, environmental sounds of the subject, etc.

[0087] The text data AD13 is an example of data AD other than the sensing data SD. As long as the text data AD13 can be used for emotion inference, the method of acquiring the text data AD13 is not particularly limited and may be appropriately selected depending on the embodiment. The text data AD13 may be acquired by manual input by the subject, or may be acquired automatically by a method such as voice analysis. In one example, the text data AD13 may include comments by the subject. The comments may include symbolic expressions such as emoticons and stamps.

[0088] The format of the result of inferring the subject's emotion may be determined appropriately depending on the embodiment. In one example, inferring an emotion may include at least one of identifying the type of emotion occurring, regressing the probability of a specific emotion occurring, and regressing the probability of emotion occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to infer only a specific type of emotion, or may be configured generically to infer multiple types of emotions. The type of emotion may be set appropriately, such as joy, anger, sadness, or happiness.

[0089] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD131 or the sound data SD132 and a plurality of second training samples 350 of the text data AD13 for the subject SS13. The subject SS13 is an example of a target person and an example of an individual during learning. The number of subjects SS13 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the subject's emotions from the image data SD131 or the sound data SD132 and the text data AD13, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0090] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD131 or the sound data SD132 and a second sample 355 of the text data AD13 for the subject TT13, whose emotions are to be inferred. The subject TT13 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring emotions) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of the emotion inference to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the first to fourth examples may be used in remote communication situations, such as video conferencing and communication in a virtual space. In this situation, information regarding the result of the emotion inference may be provided to at least one of the subject TT13 himself or the other party in the remote communication. Furthermore, if an avatar is used in the remote communication, the result of the emotion inference may be reflected in the avatar. As a result, the inference device 2 may at least partially automatically control the avatar used in remote communication. The automatic control may consist of at least one of direct control and indirect control by issuing instructions to another computer. According to one example of this embodiment, when inferring a subject's emotions from the image data SD131 or sound data SD132 and text data AD13, improving the inference accuracy can be expected by ensembling the inference results (emotion inference results) of each model (55, 56, 57).

[0091] In the above remote communication scenario, at least one of the computers used in the remote communication may operate as the inference device 2. For example, when the remote communication is performed by direct communication between user terminals, at least one of the user terminals of the users participating in the remote communication (their own user terminal or the user terminal of the communication partner) may operate as the inference device 2. When the remote communication is performed via a server device, at least one of the user terminals and the server device may operate as the inference device 2.

[0092] (1-5) Case 1-5 FIG. 3E schematically illustrates an example of Case 1-5 to which this embodiment is applied. Case 1-5 is an example of a scenario in which this embodiment is applied to inferring the value of an object. As shown in FIG. 3E, in one example, the first data 30 may be image data SD14 depicting the object, and the second data 35 may be attribute data AD14 of the object. The inference task may include predicting a value related to the object.

[0093] Image data SD14 is an example of sensing data SD. As long as it can be used for value prediction, the type and acquisition method of image data SD14 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S14 may be used to acquire the image data SD14. The sensor S14 may be, for example, an image sensor. The image data SD14 may be acquired appropriately so that at least a portion of the object is captured to an extent that value prediction is possible.

[0094] The attribute data AD14 is an example of data AD other than the sensing data SD. As long as it can be used for value prediction, the method of acquiring the attribute data AD14 and the data format thereof are not particularly limited and may be selected appropriately depending on the embodiment. At least a portion of the attribute data AD14 may be generated by manual input by an operator or may be automatically generated by computer processing. The attribute information of the object included in the attribute data AD14 may be selected appropriately depending on the value of the object to be predicted, etc. The attribute data AD14 may include, for example, attribute information such as the price, type, usage history, years of use, and dimensions of the object.

[0095] The format of the value prediction result may be selected appropriately depending on the embodiment. In one example, predicting value may include at least one of identifying a rank of the predicted value and regressing the predicted value. When predicting multiple types of value, the identification and regression may be performed for each type. The type of value to be predicted may be selected appropriately depending on the embodiment. In one example, the predicted value may include values ​​related to the amount of sales, used sales price, insurance premiums, etc. For example, predicting value may include predicting sales. In this case, the attribute data may include the sales price of the object, etc. Also, for example, predicting value may include predicting the used sales price. In this case, the attribute data may include the initial sales price, usage history, etc. of the object. Also, for example, predicting value may include predicting insurance premiums. In this case, the attribute data may include the sales price, usage history, etc. of the object. The object whose value is to be predicted may be selected arbitrarily. The object may include, for example, a vehicle, real estate, etc. If the object is a vehicle, the usage history may include the vehicle's driving history, accident history, etc. If the object is real estate, the usage history may include the real estate's occupancy history, breakdown history, repair history, and the like.

[0096] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD14 and a plurality of second training samples 350 of the attribute data AD14 for the object SS14. The object SS14 is an example of an individual during learning. The number and type of the object SS14 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when predicting the value of an object from the image data SD14 and the attribute data AD14, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0097] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD14 and a second sample 355 of attribute data AD14 for the object TT14 whose value is to be predicted. The object TT14 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (predicting value) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the value prediction result as information regarding the result of the inference task to at least one of an arbitrary output device and a storage area. In one example, the information indicating the value prediction result may be provided to an arbitrary user, such as the owner, manager, or related party (seller, etc.) of the object TT14. The output device may include a user terminal. According to one example of the present embodiment, when predicting the value of an object from the image data SD14 and attribute data AD14, improving the inference accuracy can be expected by ensembling the inference results (value prediction results) of each model (55, 56, 57).

[0098] (2) Second Case The second case shown in FIG. 2B can be applied to various situations in which two or more types of sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following ten situations. However, the application of this embodiment is not limited to the following ten examples and may be determined appropriately depending on the embodiment.

[0099] (2-1) Case 2-1 FIG. 4A schematically illustrates an example of Case 2-1 to which this embodiment is applied. Case 2-1 is an example of a scenario in which this embodiment is applied to inferring the state of an object in a chemical plant. As shown in FIG. 4A , in one example, the first data 30 may be image data SD20 depicting an object in the chemical plant, and the second data 35 may be numerical data SD21 related to the environment of the object in the chemical plant. The inference task may include inferring the state of the object.

[0100] The image data SD20 is an example of the first sensing data SD1. The type and acquisition method of the image data SD20 are not particularly limited as long as they can be used for state inference, and may be selected appropriately depending on the embodiment. One or more first sensors S20 may be used to acquire the image data SD20. The first sensors S20 may be, for example, image sensors. The first sensors S20 may be appropriately positioned to capture images of objects from inside or outside the chemical plant.

[0101] The numerical data SD21 is an example of second sensing data SD2. As long as the numerical data SD21 can be used for state inference, the type and format of the numerical data SD21 are not particularly limited and may be appropriately selected depending on the embodiment. The numerical data SD21 may be composed of information at a single point in time, or may be composed of time-series data including information at multiple points in time. Furthermore, the environment to be observed as the numerical data SD21 may be appropriately selected depending on the state to be inferred, etc. The numerical data SD21 may include information such as pressure, temperature, and humidity. One or more second sensors S21 may be used to acquire the numerical data SD21. The second sensor S21 may be appropriately selected depending on the environment to be observed. The second sensor S21 may be selected from, for example, a pressure gauge, a thermometer, a hygrometer, etc.

[0102] The format of the result of inferring the state may be selected appropriately depending on the embodiment. In one example, inferring the state may include at least one of identifying a category of the inferred state and regressing the degree of the state. When inferring multiple types of states, the identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, the chemical plant may include a fermentation plant, a gas absorption plant, etc. For example, if the chemical plant is a fermentation plant, the image data SD20 may be acquired to capture bubbles generated in the target (fermentation target). The numerical data SD21 may include information such as pressure. Inferring the state may include predicting a yield. The yield is an index indicating the degree to which a specific product or component is efficiently produced in a chemical process or chemical reaction. Note that the target of inference is not limited to this example and may be changed appropriately depending on the embodiment.

[0103] In the learning stage, the model generating device 1 may acquire a plurality of first training samples 300 of image data SD20 and a plurality of second training samples 350 of numerical data SD21 for objects in the chemical plant SS20. The objects in the chemical plant SS20 are an example of individuals during learning. The model generating device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example embodiment, when inferring the state of objects in the chemical plant from the image data SD20 and the numerical data SD21, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0104] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD20 and a second sample 355 of numerical data SD21 for an object within the chemical plant TT20. The object within the chemical plant TT20 is an example of a target individual during inference. The chemical plant TT20 to be inferred may be the same as or different from the chemical plant SS20 during learning. The inference device 2 may derive a result of the inference task (inferring the state of the object) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the object to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the object may be provided to an arbitrary user, such as a manager of the chemical plant TT20. The output device may include a user terminal. According to one example of this embodiment, when inferring the state of an object in a chemical plant from image data SD20 and numerical data SD21, the inference results (state inference results) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of inference.

[0105] (2-2) Case 2-2 FIG. 4B schematically illustrates an example of Case 2-2 to which this embodiment is applied. Case 2-2 is an example of a scenario in which this embodiment is applied to inferring conditions inside a pipe. As shown in FIG. 4B , in one example, the first data 30 may be ultrasonic data SD22 related to the flow velocity inside the pipe, and the second data 35 may be sound data SD23 related to the pipe. The inference task may include inferring conditions inside the pipe.

[0106] The ultrasonic data SD22 is an example of the first sensing data SD1. As long as the ultrasonic data SD22 can be used for state inference, the method of acquiring the ultrasonic data SD22 is not particularly limited and may be appropriately selected depending on the embodiment. One or more first sensors S22 may be used to acquire the ultrasonic data SD22. The first sensor S22 may be, for example, an ultrasonic sensor. In one example, the first sensor S22 may be attached to a pipe with a clamp and configured to observe the flow velocity of an object flowing through the pipe.

[0107] The sound data SD23 is an example of second sensing data SD2. As long as the sound data SD23 can be used for state inference, the method of acquiring the sound data SD23 is not particularly limited and may be selected appropriately depending on the embodiment. One or more second sensors S23 may be used to acquire the sound data SD23. The second sensor S23 may be, for example, a microphone. In one example, the second sensor S23 may be configured to be placed on or near a pipe to observe sounds inside the pipe.

[0108] The format of the result of inferring the state inside the pipe may be selected appropriately depending on the embodiment. In one example, inferring the state may include at least one of identifying a category of the inferred state and regressing the degree of the state. When multiple types of states are inferred, the identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, inferring the state inside the pipe may include determining whether the pipe is clogged or inferring the degree of cloggedness inside the pipe. This may allow the state of an object flowing inside the pipe to be monitored. By controlling the flow rate according to the monitoring results, it is possible to prevent the pipe from being clogged. For example, when the present embodiment is applied to a papermaking process, the state of pulp fibers flowing inside the pipe may be monitored and the flow rate of the pulp fibers may be controlled according to the monitoring results. This may prevent the pipe from being clogged by the pulp fibers.

[0109] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the ultrasound data SD22 and a plurality of second training samples 350 of the sound data SD23 for the pipe SS22. The pipe SS22 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state inside the pipe from the ultrasound data SD22 and the sound data SD23, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0110] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the ultrasound data SD22 and a second sample 355 of the sound data SD23 for the pipe TT22. The pipe TT22 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (inferring the state within the pipe TT22) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state within the pipe TT22 to at least one of an output device and a storage area as information regarding the result of the inference task. In one example, the information regarding the result of the inference task may be provided to an arbitrary user, such as an administrator of the pipe TT22. The output device may include a user terminal. Furthermore, when the inference device 2 is connected to an adjustment device that adjusts the flow rate within the pipe TT22, the inference device 2 may send a command to the adjustment device instructing the flow rate according to the inference result, as an example of a process of outputting information regarding the result of the inference task. For example, if it is inferred that the degree of blockage in the pipe TT22 is high, the inference device 2 may issue a command to the adjustment device to reduce the flow rate in the pipe TT22. On the other hand, if it is inferred that the degree of blockage in the pipe TT22 is low, the inference device 2 may issue a command to the adjustment device to maintain or increase the flow rate in the pipe TT22. Whether the degree of blockage is high or low may be determined by any method, such as threshold comparison. If the flow rate is to be maintained, the inference device 2 may omit sending a command. According to one example of the present embodiment, when inferring the state inside the pipe from the ultrasound data SD22 and the sound data SD23, improvement in inference accuracy can be expected by ensembling the inference results (state inference results) of each model (55, 56, 57).

[0111] (2-3) Case 2-3 FIG. 4C schematically illustrates an example of Case 2-3 to which this embodiment is applied. Case 2-3 is an example of a situation in which this embodiment is applied to inferring the position of an object when driving a robotic device. As shown in FIG. 4C , in one example, the first data 30 may be image data SD24 containing an object, and the second data 35 may be point cloud data SD25 obtained by measuring the object. The inference task may include inferring the position of the object.

[0112] The image data SD24 is an example of the first sensing data SD1. The type and acquisition method of the image data SD24 are not particularly limited as long as they can be used for position inference and may be appropriately selected depending on the embodiment. One or more first sensors S24 may be used to acquire the image data SD24. The first sensor S24 may be, for example, an image sensor. The image sensor constituting the first sensor S24 may be appropriately selected so as not to completely overlap with the second sensor S25. The first sensor S24 may be attached to the robot device or may be disposed externally to the robot device. In one example, the first sensor S24 may be appropriately positioned so that the imaging range includes at least a portion of the working range of the robot device. The working range of the robot device may be appropriately defined according to, for example, the reach of the end effector, the range in which the robot device can move, etc.

[0113] The point cloud data SD25 is an example of second sensing data SD2. As long as the point cloud data SD25 can be used for position inference, the type and acquisition method of the point cloud data SD25 are not particularly limited and may be appropriately selected depending on the embodiment. One or more second sensors S25 may be used to acquire the point cloud data SD25. The second sensor S25 may be appropriately selected from, for example, an ultrasonic sensor, a depth sensor, an infrared sensor, radar, LiDAR, etc. The second sensor S25, like the first sensor S24, may be attached to the robot device or may be located outside the robot device. In one example, the second sensor S25 may be appropriately positioned so that its measurement range includes at least a portion of the imaging range of the first sensor S24. As a result, the point cloud data SD25 may be obtained by measuring at least a portion of the imaging range of the image data SD24.

[0114] The format of the location inference result may be selected appropriately depending on the embodiment. In one example, inferring location may include at least one of identifying presence or absence in the target area, regressing the probability of presence in the target area, regressing the range of presence, and regressing the location of presence (two-dimensional or three-dimensional location information). Inferring location may also include inferring the shape and / or dimensions of the object (including the amount of deformation of the object). Inferring location may also include other inference tasks, such as identifying the type of object.

[0115] The type of robot device is not particularly limited and may be selected appropriately depending on the embodiment. The robot device may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile robot configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, or an orthogonal robot. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a nursing robot, a security robot, or a transport robot. The content of the autonomous processing may be selected appropriately depending on the embodiment. The mobile robot may be, for example, a cleaning robot, the above-mentioned autonomous robot configured to move (including a mobile robot), a vehicle configured to be autonomously driven, an air vehicle capable of autonomous flight (such as a drone), or a ship capable of autonomous navigation (such as a ship or submarine).

[0116] Furthermore, the object whose position is to be inferred need not be particularly limited as long as it can be involved in the operation of the robot device, and may be selected appropriately depending on the embodiment. The object may include, for example, a work target of the robot device, an obstacle, etc. The work target may be, for example, an object to be grasped and carried by the robot device, etc. The obstacle may include an object and a person. The result of the position inference may be used appropriately for controlling the robot device. The object may be, for example, a part of an individual, a subordinate part of an individual such as a mark, or an independent individual.

[0117] In the learning phase, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD24 and a plurality of second training samples 350 of the point cloud data SD25 for the target SS24 that may be involved in the operation of the robot device RS24. The target SS24 is an example of an individual during learning. The type and number of the target SS24 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and a plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of this embodiment, when inferring the position of an object from the image data SD24 and the point cloud data SD25, it is expected that the inference accuracy of the generated trained models (51, 52, 53) can be improved. Note that the robot device RS24 may be omitted in the learning phase.

[0118] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD24 and a second sample 355 of the point cloud data SD25 for the target TT24. The target TT24 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring the position of the target TT24) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the position of the target TT24 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. Furthermore, when connected to the robot device RT24, the inference device 2 may transmit a command to the robot device RT24 instructing the robot device RT24 to perform an operation according to the inference result, as an example of a process of outputting information regarding the result of the inference task. For example, if the robotic device RT24 is an industrial robot and the target TT24 is a work object to be grasped, the inference device 2 may issue a command to the robotic device RT24 to move an end effector to an inferred position and grasp the target TT24 present at the inferred position with the end effector. If inferring the position includes inferring the shape, the inference device 2 may determine a force to grasp the target TT24 according to the inferred shape (e.g., the amount of deformation) of the target TT24 and issue a command to the robotic device RT24 to grasp the object TT24 with the determined force. In this way, the inference device 2 may adjust the grasping force of the robotic device RT24. Also, for example, if the robotic device RT24 is an autonomous robot or a mobile robot and the target TT24 is the objective of the task, the inference device 2 may issue a command to the robotic device RT24 to move near the target TT24 or to face the target TT24. Furthermore, for example, if the robot device RT24 is an industrial robot, an autonomous robot, or a mobile robot, and the target TT24 is an obstacle, the inference device 2 may issue a command to the robot device RT24 to avoid the target TT24 (for example, limiting the driving range, moving along a detour route, etc.). The inference device 2 may directly control the operation of the robot device RT24, or may indirectly control it by issuing an instruction to a controller of the robot device RT24. The robot device RT24 during inference may be the same as or different from the robot device RS24 during learning.According to one example of this embodiment, when inferring the position of an object from image data SD24 and point cloud data SD25, the inference results (position inference results) of each model (55, 56, 57) are ensembled, thereby improving the inference accuracy.

[0119] (2-4) Case 2-4 FIG. 4D schematically illustrates an example of Case 2-4 to which this embodiment is applied. Case 2-4 is an example of a scene in which this embodiment is applied to inferring the state of equipment. As shown in FIG. 4D, in one example, the first data 30 may be image data SD26 depicting the equipment, and the second data 35 may be sound data SD27 related to the equipment. The inference task may include inferring the state of the equipment.

[0120] The image data SD26 is an example of the first sensing data SD1. The type and acquisition method of the image data SD26 are not particularly limited as long as they can be used to infer the state of the equipment, and may be selected appropriately depending on the embodiment. One or more first sensors S26 may be used to acquire the image data SD26. The first sensors S26 may be, for example, image sensors. The first sensors S26 may be appropriately positioned so that their imaging range includes at least a portion of the range in which the state of the equipment is inferred.

[0121] The sound data SD27 is an example of second sensing data SD2. The method of acquiring the sound data SD27 is not particularly limited as long as it can be used to infer the state of the equipment, and may be appropriately selected depending on the embodiment. One or more second sensors S27 may be used to acquire the sound data SD27. The second sensor S27 may be, for example, a microphone. In one example, the second sensor S27 may be appropriately positioned to observe sounds (e.g., vibration sounds, abnormal sounds, etc.) emitted from the equipment.

[0122] The format of the result of inferring the state of the equipment may be selected appropriately depending on the embodiment. In one example, inferring the state may include at least one of identifying a category of the state to be inferred and regressing the degree of the state. When multiple types of states are inferred, the identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, inferring the state of the equipment may include identifying a deterioration level of the equipment or regressing the deterioration degree of the equipment. This may allow the state of the equipment to be monitored. The type of equipment whose state is to be inferred is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the equipment may include structures such as buildings, tunnels, bridges, and water pipes.

[0123] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD26 and a plurality of second training samples 350 of sound data SD27 for the equipment SS26. The equipment SS26 is an example of an individual during learning. The type and number of the equipment SS26 from which each training sample (300, 350) is acquired may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the state of the equipment from the image data SD26 and the sound data SD27, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0124] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD26 and a second sample 355 of sound data SD27 for equipment TT26, the state of which is to be inferred. The equipment TT26 is an example of a target individual during inference. The equipment TT26 during inference may be the same as or different from the equipment SS26 during learning. The inference device 2 may derive the result of the inference task (inferring the state of the equipment TT26) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the equipment TT26 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the equipment TT26 may be provided to an arbitrary user, such as an administrator managing the equipment TT26. The output device may include a user terminal. According to one example of this embodiment, when inferring the state of equipment from image data SD26 and sound data SD27, the inference results (state inference results) of each model (55, 56, 57) are ensembled, thereby making it possible to expect improved inference accuracy.

[0125] (2-5) Case 2-5 FIG. 4E schematically illustrates an example of Case 2-5 to which this embodiment is applied. Case 2-5 is an example of a scenario in which this embodiment is applied to inferring water quality in a facility. As shown in FIG. 4E, in one example, the first data 30 may be image data SD28 depicting water present in the facility, and the second data 35 may be observation data SD29 of the odor of water present in the facility. The inference task may include inferring water quality in the facility.

[0126] The image data SD28 is an example of the first sensing data SD1. The type and acquisition method of the image data SD28 are not particularly limited as long as they can be used to infer water quality, and may be selected appropriately depending on the embodiment. One or more first sensors S28 may be used to acquire the image data SD28. The first sensors S28 may be, for example, image sensors. The first sensors S28 may be appropriately positioned so that their imaging range includes at least a portion of the range in which the quality of water present in the facility is inferred.

[0127] The observation data SD29 is an example of second sensing data SD2. As long as the observation data SD29 can be used to infer water quality, the acquisition method and data format of the observation data SD29 are not particularly limited and may be appropriately selected depending on the embodiment. One or more second sensors S29 may be used to acquire the observation data SD29. The second sensor S29 may be, for example, an odor sensor. In one example, the second sensor S29 may be appropriately positioned within a range where the odor of water present in the facility can be observed.

[0128] The form of the result of inferring water quality may be selected appropriately depending on the embodiment. In one example, inferring water quality may include at least one of identifying a quality level and regressing the degree of quality. Water quality may be defined appropriately according to any standard, including known standards. The facility from which water quality is inferred may be selected appropriately depending on the embodiment. In one example, the facility may include water-related equipment such as a water purification plant or a sewage treatment plant. The facility may also include a natural or artificially created body of water such as a pond or a lake.

[0129] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD28 and a plurality of second training samples 350 of observation data SD29 for a facility SS28. Water present in the facility SS28 is an example of an individual during learning. The type and number of facilities SS28 from which the training samples (300, 350) are acquired may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of this embodiment, when inferring water quality at a facility from the image data SD28 and the observation data SD29, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0130] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD28 and a second sample 355 of observation data SD29 for the facility TT28. The water present in the facility TT28 is an example of a target individual during inference. The facility TT28 during inference may be the same as or different from the facility SS28 during learning. The inference device 2 may derive the result of the inference task (inferring the water quality at the facility TT28) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the water quality at the facility TT28 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the water quality at the facility TT28 may be provided to an arbitrary user, such as an administrator managing the facility TT28. The output device may include a user terminal. If the facility TT28 is a facility such as a water purification plant or a sewage treatment plant, the inference device 2 may generate an operation plan for the facility according to the inference result as information regarding the result of the inference task. For example, if the facility TT28 is a sewage treatment plant and a deterioration in water quality is inferred, the inference device 2 may generate an operation plan to increase the operation rate of the sewage treatment plant. If a maintenance of water quality is inferred, the inference device 2 may generate an operation plan to maintain the operation rate of the sewage treatment plant. The inference device 2 may output instructions to a user or a controller of the facility to operate the facility in accordance with the generated operation plan. According to an example of the present embodiment, when inferring water quality at a facility from image data SD28 and observation data SD29, improving the inference accuracy can be expected by ensembling the inference results (water quality inference results) of each model (55, 56, 57).

[0131] (2-6) Case 2-6 FIG. 4F schematically illustrates an example of Case 2-6 to which this embodiment is applied. Case 2-6 is an example of a scenario in which this embodiment is applied to inferring the state of oil in a fryer. As shown in FIG. 4F , in one example, the first data 30 may be image data SD30 depicting oil in a fryer, and the second data 35 may be observation data SD31 of the smell of oil in the fryer. The inference task may include inferring the state of oil in the fryer.

[0132] The image data SD30 is an example of the first sensing data SD1. The type and acquisition method of the image data SD30 are not particularly limited as long as they can be used to infer the state of the oil, and may be selected appropriately depending on the embodiment. One or more first sensors S30 may be used to acquire the image data SD30. The first sensor S30 may be, for example, an image sensor. The first sensor S30 may be appropriately positioned so that the imaging range includes at least a portion of the oil present in the fryer.

[0133] The observation data SD31 is an example of second sensing data SD2. As long as the observation data SD31 can be used to infer the state of oil, the acquisition method and data format of the observation data SD31 are not particularly limited and may be appropriately selected depending on the embodiment. One or more second sensors S31 may be used to acquire the observation data SD31. The second sensor S31 may be, for example, an odor sensor. In one example, during observation, the second sensor S31 may be appropriately positioned within a range where the odor of oil present in the fryer can be observed.

[0134] The format of the result of inferring the oil condition may be selected appropriately depending on the embodiment. In one example, inferring the oil condition may include at least one of identifying the level of oil quality and regressing the oil quality. The oil quality may be defined appropriately according to any standard, including known standards. In addition, inferring the oil condition may include predicting the time to change the oil. In one example, inferring the oil condition may include determining whether the time to change the oil has arrived or regressing the time to change the oil. This allows the quality of the oil to be monitored. The present Example 2-6 may be implemented in stores that sell oil, such as convenience stores and restaurants.

[0135] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD30 and a plurality of second training samples 350 of observation data SD31 for oil SS30 in a fryer. The oil SS30 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and a plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, when inferring the state of oil in a fryer from the image data SD30 and the observation data SD31, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0136] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD30 and a second sample 355 of the observation data SD31 for the oil TT30 of the fryer. The oil TT30 is an example of a target individual during inference. The fryer during inference may be the same as or different from the fryer during learning. The inference device 2 may derive the result of the inference task (inferring the state of the oil TT30) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the oil TT30 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the oil TT30 may be provided to an arbitrary user, such as an administrator who manages the oil TT30. The output device may include a user's terminal. When the arrival of the replacement time is inferred, the information indicating the result of inferring the state of the oil TT30 may include instruction information for instructing the replacement of the oil TT30. According to one example of this embodiment, when inferring the state of oil in a fryer from image data SD30 and observation data SD31, the inference results (inference results of the oil state) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of the inference.

[0137] The state of the oil can also be observed using other sensing data, such as sound data containing sounds made during use. Therefore, at least one of the image data SD30 and the observation data SD31 may be replaced with other sensing data. For example, the observation data SD31 may be replaced with sound data recorded while the oil is being used. In this case, the second sensor may be, for example, a microphone.

[0138] (2-7) Case 2-7 FIG. 4G schematically illustrates an example of Case 2-7 to which this embodiment is applied. Case 2-7 is an example of a scene in which this embodiment is applied to atmospheric condition inference. As shown in FIG. 4G, in one example, the first data 30 may be image data SD32 depicting the atmosphere, and the second data 35 may be atmospheric meteorological data SD33. The inference task may include inferring atmospheric conditions.

[0139] The image data SD32 is an example of the first sensing data SD1. As long as the image data SD32 reflects the atmospheric conditions and can be used to infer the atmospheric conditions, the type and acquisition method of the image data SD32 are not particularly limited and may be selected appropriately depending on the embodiment. One or more first sensors S32 may be used to acquire the image data SD32. The first sensor S32 may be, for example, an image sensor. The image data SD32 may be, for example, satellite image data, infrared image data, etc.

[0140] The weather data SD33 is an example of the second sensing data SD2. As long as the weather data SD33 can be used to infer atmospheric conditions, the type and acquisition method of the weather data SD33 are not particularly limited and may be selected appropriately depending on the embodiment. One or more second sensors S33 may be used to acquire the weather data SD33. The second sensor S33 may be, for example, a weather sensor. The weather data SD33 may include, for example, weather information such as temperature, humidity, atmospheric pressure, rainfall, wind speed, wind direction, and illuminance. The type of weather information included in the weather data SD33 may be selected appropriately depending on the embodiment.

[0141] The form of the result of inferring the atmospheric condition may be selected as appropriate depending on the embodiment. In one example, inferring the atmospheric condition may include identifying a level of a predetermined indicator of the atmospheric condition and / or regressing the value of the predetermined indicator. The indicator of the atmospheric condition may be defined according to any criteria, including known criteria, as appropriate. Inferring the atmospheric condition may also include identifying or regressing a pollution level of the atmospheric condition, thereby enabling monitoring of the level of air pollution.

[0142] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD32 and a plurality of second training samples 350 of the meteorological data SD33 for the atmospheric corresponding range SS32. The atmospheric corresponding range SS32 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring atmospheric conditions from the image data SD32 and the meteorological data SD33, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0143] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD32 and a second sample 355 of weather data SD33 for the atmospheric target range TT32. The atmospheric target range TT32 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring the state of the target range TT32) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the atmospheric target range TT32 to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the atmospheric target range TT32 may be provided to an arbitrary user via a server device that provides weather information. In this case, the inference device 2 may provide information indicating the inference result to the server device. Alternatively, the server device may operate as the inference device 2. According to one example of this embodiment, when inferring atmospheric conditions from image data SD32 and meteorological data SD33, the inference results (inference results of atmospheric conditions) of each model (55, 56, 57) are ensembled, thereby improving the accuracy of inference.

[0144] (2-8) Case 2-8 FIG. 4H schematically illustrates an example of Case 2-8 to which this embodiment is applied. Case 2-8 is an example of a scenario in which this embodiment is applied to inferring the condition of a crop. As shown in FIG. 4H , in one example, the first data 30 may be image data SD34 related to the crop, and the second data 35 may be weather data SD35 in the environment of the crop. The inference task may include inferring the condition of the crop.

[0145] The image data SD34 is an example of the first sensing data SD1. The type and acquisition method of the image data SD34 are not particularly limited as long as they can be used to infer the condition of the crop, and may be selected appropriately depending on the embodiment. One or more first sensors S34 may be used to acquire the image data SD34. The first sensor S34 may be, for example, an image sensor. The image data SD34 may include general image data as well as satellite image data. The imaging range is not particularly limited and may be determined appropriately as long as the image captures enough of the crop to enable inference of the crop condition. The image data SD34 may include an image of at least a portion of the crop. The image data SD34 may be acquired so as to capture only the target crop, or may be acquired so as to capture multiple crops including the target crop.

[0146] The weather data SD35 is an example of second sensing data SD2. As long as the weather data SD35 can be used to infer the state of the crop, the type and acquisition method of the weather data SD35 are not particularly limited and may be selected appropriately depending on the embodiment. The weather data SD35 may be similar to the weather data SD33 in the above-mentioned Example 2-7. One or more second sensors S35 may be used to acquire the weather data SD35. The second sensors S35 may be similar to the second sensors S33 in the above-mentioned Example 2-7.

[0147] The format of the result of inferring the crop condition may be selected appropriately depending on the embodiment. In one example, inferring the crop condition may include at least one of identifying the level of an arbitrary crop-related indicator and regressing the value of the indicator. The crop-related indicator may be defined appropriately according to any criteria, including known criteria. The crop-related indicator may include, for example, an environmental indicator indicating the degree to which the environment is suitable for growth, a growth indicator indicating the level of crop growth, etc. The environmental indicator may include, for example, an indicator for determining the timing of watering, an indicator for adjusting the growth environment (amount of water, amount of light, temperature, humidity, etc.), etc. This makes it possible to monitor the growth status of the crop. Note that the type of crop is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the crop may include agricultural crops such as common crops, forage crops, green manure crops, horticultural crops, and industrial crops.

[0148] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD34 and a plurality of second training samples 350 of weather data SD35 for the crop SS34. The individual crop SS34 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state of a crop from the image data SD34 and the weather data SD35, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0149] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD34 and a second sample 355 of weather data SD35 for the crop TT34. The target individual of the crop TT34 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (inferring the state of the crop TT34) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of the inference of the state of the crop TT34 to at least one of an output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of the inference of the state of the crop TT34 may be provided to an arbitrary user, such as an administrator managing the crop TT34. The output device may include a user terminal. Furthermore, when connected to a cultivation device configured to adjust the environment of the crop TT34, the inference device 2 may issue an instruction to the cultivation device to adjust the environment of the crop TT34 in accordance with the inference result, as an example of a process of outputting information regarding the result of the inference task. As long as the cultivation device can control the growth environment of the crop TT34, the type of cultivation device is not particularly limited and may be selected appropriately depending on the embodiment. The cultivation device may include, for example, a watering device, a lighting device, a curtain device, a window device, an air conditioning device, etc. The watering device may be, for example, a sprinkler, etc. The lighting device may be, for example, an LED (light emitting diode) light, a fluorescent light, etc. The curtain device may be configured to open and close curtains attached to windows of a building. The window device may be configured to open and close windows of a building. The air conditioning device may be, for example, an air conditioner, etc. The watering device may be used to control the amount of water applied to the crop TT34. The lighting device and the curtain device may be used to control the time and amount of light irradiated onto the crop TT34. The air conditioning device may be used to control the temperature and humidity around the crop TT34. For example, if it is inferred that the state of the crop TT34 is lacking in water, the inference device 2 may issue a command to the watering device to perform watering. This allows at least part of the cultivation of the crop TT34 to be automated.According to one example of this embodiment, when inferring the condition of crops from image data SD34 and meteorological data SD35, the inference results (inference results of the crop condition) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of inference.

[0150] Note that the state of the crops can also be observed using other sensing data. Therefore, at least one of the image data SD34 and the weather data SD35 may be replaced with other sensing data. In one example, either the image data SD34 or the weather data SD35 may be replaced with soil data. Accordingly, either the first sensor S34 or the second sensor S35 may be replaced with a soil sensor.

[0151] (2-9) Case 2-9 Figure 4I schematically illustrates an example of Case 2-9 to which this embodiment is applied. Case 2-9 is an example of a scene in which this embodiment is applied to inferring the state of a subject. As shown in Figure 4I, in one example, the first data 30 may be image data SD36 depicting the subject, and the second data 35 may be biometric data SD37 derived from the subject's living body. The inference task may include inferring the state of the subject.

[0152] The image data SD36 is an example of the first sensing data SD1. The type and acquisition method of the image data SD36 are not particularly limited as long as they can be used to infer the subject's condition, and may be selected appropriately depending on the embodiment. The image data SD36 may include general image data as well as medical image data. The medical image data may be similar to the medical image data SD10 in the above-described Example 1-1. One or more first sensors S36 may be used to acquire the image data SD36. The first sensor S36 may be selected from, for example, an image sensor, an ultrasound diagnostic device, an X-ray device, a CT device, an MRI device, a PET device, a scintigraphy device, etc.

[0153] The biometric data SD37 is an example of the second sensing data SD2. The type and acquisition method of the biometric data SD37 are not particularly limited as long as they can be used to infer the subject's condition, and may be appropriately selected depending on the embodiment. The biometric data SD37 may include, for example, blood pressure, pulse, heart rate, electrocardiogram, electromyogram, body temperature, electrodermal response, brain activity (electroencephalogram, magnetoencephalogram), activity level, blood glucose level, electrooculography, eye movement, and odor. One or more second sensors S37 may be used to acquire the biometric data SD37. The second sensors S37 may be selected from, for example, a blood pressure monitor, a pulse rate monitor, a heart rate monitor, an electrocardiogram, an electromyogram, a thermometer, an electrodermal response meter, an electroencephalogram, a magnetoencephalogram, an activity meter, a blood glucose level meter, an electrooculography sensor, an eye movement meter, and an odor sensor. At least a portion of the second sensors S37 may be a wearable device.

[0154] The format of the result of inferring the subject's condition may be selected appropriately depending on the embodiment. In one example, inferring the subject's condition may include at least one of identifying the level of any indicator related to the subject and regressing the value of the indicator. The indicator related to the condition may be defined appropriately according to any criteria, including known criteria. The subject's condition may include the subject's health condition. As in the above-mentioned Example 1-1, inferring the health condition may include determining whether or not a disease has occurred.

[0155] For example, the image data SD36 may be acquired to capture the inside of the subject's mouth. The biometric data SD37 may include odor observation data. Inferring the subject's condition may include inferring the degree of the subject's periodontal disease. Furthermore, for example, the image data SD36 may include medical image data. The biometric data SD37 may include electrocardiogram data. Inferring the subject's condition may include inferring a sign of myocardial infarction in the subject. Furthermore, for example, the image data SD36 may include a temperature distribution or thermographic image. The biometric data SD37 may include heart rate data. Inferring the subject's condition may include inferring the subject's core body temperature. Furthermore, for example, the image data SD36 may be acquired to capture pupil diameter, gaze point, eye movements such as blinking, etc. The biometric data SD37 may include brain activity data. Inferring the subject's condition may include inferring the subject's attention (wakefulness, etc.).

[0156] In the learning stage, the model generating device 1 may acquire a plurality of first training samples 300 of the image data SD36 and a plurality of second training samples 350 of the biometric data SD37 for the subject SS36. The subject SS36 is an example of a subject and an example of an individual during learning. The number of subjects SS36 may be determined appropriately depending on the embodiment. The model generating device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the subject's condition from the image data SD36 and the biometric data SD37, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0157] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD36 and a second sample 355 of biometric data SD37 for the subject TT36. The subject TT36 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring the state of the subject TT36) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the subject TT36 to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the subject TT36 may be provided to an arbitrary user, such as the subject TT36 himself / herself. The output device may include a user terminal. According to one example of the present embodiment, when inferring the state of the subject from the image data SD36 and the biometric data SD37, improvement in inference accuracy can be expected by ensembling the inference results (inference results of the subject's state) of each model (55, 56, 57).

[0158] (2-10) Case 2-10 FIG. 4J schematically illustrates an example of Case 2-10 to which this embodiment is applied. Case 2-10 is an example of a scenario in which this embodiment is applied to inferring the state of a house. As shown in FIG. 4J , in one example, the first data 30 and the second data 35 may be first observation data SD38 and second observation data SD39 about the interior of the house. The inference task may include inferring the state of the interior of the house.

[0159] The first observation data SD38 is an example of first sensing data SD1, and the second observation data SD39 is an example of second sensing data SD2. As long as they can be used to infer the state inside the house, the types and acquisition methods of the first observation data SD38 and the second observation data SD39 are not particularly limited and may be selected appropriately depending on the embodiment. One or more first sensors S38 may be used to acquire the first observation data SD38, and one or more second sensors S39 may be used to acquire the second observation data SD39. The first sensors S38 and the second sensors S39 may each be selected from, for example, various sensors constituting a home security sensor, an odor sensor, etc. As an example, the first sensor S38 may be a home security sensor, and the second sensor S39 may be an odor sensor. The first sensors S38 and the second sensors S39 may be appropriately positioned to observe the interior of the house.

[0160] The format of the result of inferring the state inside the home may be selected appropriately depending on the embodiment. In one example, inferring the state inside the home may include at least one of identifying whether an abnormality has occurred inside the home and regressing the probability that a specific abnormality has occurred. Inferring the state inside the home may include identifying the type of abnormality occurring inside the home. Inferring the state inside the home may also include regressing the probability of abnormality occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured generally to determine whether any abnormality has occurred. The type of abnormality occurring inside the home may be defined appropriately from the perspective of, for example, security (crime prevention, emergency situations, etc.), monitoring (abnormalities of residents, etc.), etc.

[0161] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the first observation data SD38 and a plurality of second training samples 350 of the second observation data SD39 for the house SS38. The individual of the house SS38 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state inside the house from two types of observation data (the first observation data SD38 and the second observation data SD39), improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0162] In the inference stage, the inference device 2 may acquire a first sample 305 of the first observation data SD38 and a second sample 355 of the second observation data SD39 for the house TT38. The target individual of the house TT38 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (inferring the state inside the house TT38) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state inside the house TT38 to at least one of an output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state inside the house TT38 may be provided to any user, such as a resident of the house TT38, the resident's relative, or a manager. The output device may include a user's terminal. According to one example of this embodiment, when inferring the state inside a house from two types of observation data (first observation data SD38, second observation data SD39), the inference results (inference results of the state inside the house) of each model (55, 56, 57) can be ensembled, thereby improving the accuracy of the inference.

[0163] (Other) Which of the two or more types of sensing data is used as the first data 30 and the second data 35 may be determined appropriately depending on the embodiment, etc. In each of the above cases, the first data 30 and the second data 35 may be interchanged. For example, in the above case 2-1, the image data SD20 may be used as the second data 35, and the numerical data SD21 may be used as the first data 30.

[0164] (3) Third Case The third case shown in FIG. 2C can be applied to various situations in which two or more types of data other than sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following two situations. However, the application of this embodiment is not limited to the following two examples and may be determined appropriately depending on the embodiment.

[0165] (3-1) Case 3-1 FIG. 5A schematically illustrates an example of Case 3-1 to which this embodiment is applied. Case 3-1 is an example of a scenario in which this embodiment is applied to stock price prediction. As shown in FIG. 5A , in one example, the first data 30 may be market data AD50 related to a target of prediction, and the second data 35 may be text data AD51 related to the target. The inference task may include predicting the stock price of the target.

[0166] Market data AD50 is an example of data AD1. As long as it can be used for stock price prediction, the structure and origin of market data AD50 are not particularly limited and may be selected appropriately depending on the embodiment. Market data AD50 may include, for example, current stock prices, trading volumes, market indicators, yields, credit ratings, commodity prices, commodity production volumes, exchange rates, currency trading volumes, interest rates, etc.

[0167] The text data AD51 is an example of the data AD2. As long as the text data AD51 can be used for stock price prediction, the structure and origin of the text data AD51 are not particularly limited and may be appropriately selected depending on the embodiment. The text data AD51 may include, for example, news, SNS data, etc. The SNS data may include, for example, user comments, signatures, emotion identification results, etc.

[0168] The format of the stock price prediction result may be selected appropriately depending on the embodiment. In one example, predicting stock prices may include at least one of identifying whether stock prices will rise or not and regressing future stock prices. The target for stock price prediction is not particularly limited and may be selected appropriately depending on the embodiment. The target may include, for example, a company, an index (such as the Nikkei average), etc.

[0169] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the market data AD50 and a plurality of second training samples 350 of the text data AD51 for the target SS50. The target SS50 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when predicting the stock price of a target from the market data AD50 and the text data AD51, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0170] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of market data AD50 and a second sample 355 of text data AD51 for the target TT50. The target TT50 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (predicting the stock price of the target TT50) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of the prediction of the stock price of the target TT50 to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the information indicating the result of the prediction of the stock price of the target TT50 may be provided to an arbitrary user via a server device. In this case, the inference device 2 may provide information indicating the prediction result to the server device. Alternatively, the server device may operate as the inference device 2. According to one example of this embodiment, when predicting a target stock price from market data AD50 and text data AD51, the inference results (stock price prediction results) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of inference.

[0171] (3-2) Case 3-2 FIG. 5B schematically illustrates an example of Case 3-2 to which this embodiment is applied. Case 3-2 is an example of a situation in which this embodiment is applied to content type classification. As shown in FIG. 5B, in one example, the first data 30 may be content data AD52 that constitutes the content, and the second data 35 may be associated data AD53 related to the content. The inference task may include classifying the type of content.

[0172] Content data AD52 is an example of data AD1. Content data AD52 may be the content itself. Content data AD52 may include, for example, image data, sound data, text data (document data, etc.), etc. Associated data AD53 is an example of data AD2. As long as it is related to the content, the type of associated data AD53 is not particularly limited and may be selected appropriately depending on the embodiment. Associated data AD53 may include, for example, metadata, user writings on the content (text data), etc. Metadata may include, for example, attribute information such as the title, creator name, and creation date.

[0173] The format of the result of classifying the content type may be selected appropriately depending on the embodiment. In one example, classifying the content type may include at least one of identifying a category to which the content belongs and regressing a probability that the content belongs to a target category. Classifying the content type may include classifying whether the content includes specific information such as fake information. Classifying whether the content includes specific information may include either identifying whether the content includes the specific information or regressing a probability that the content includes the specific information.

[0174] In the learning stage, the model generation device 1 may acquire, for the content SS52, a plurality of first training samples 300 of the content data AD52 and a plurality of second training samples 350 of the associated data AD53. The relevant individuals of the content SS52 are an example of individuals during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and a plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when classifying the type of content from the content data AD52 and the associated data AD53, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0175] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the content data AD52 and a second sample 355 of the associated data AD53 for the content TT52. The target individual of the content TT52 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (classifying the type of the content TT52) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of classifying the type of the content TT52 to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, if the content TT52 is classified as including specific information, the inference device 2 may apply a process to delete the specific information from the content TT52 (content data AD52). If the content TT52 is specific information, the content TT52 (content data AD52) itself may be deleted. According to one example of this embodiment, when classifying content types from content data AD52 and associated data AD53, the inference results (type classification results) of each model (55, 56, 57) are ensembled, which is expected to improve inference accuracy.

[0176] (Other) Which of the two or more types of data is used as the first data 30 and the second data 35 may be determined appropriately depending on the embodiment, etc. In each of the above cases, the first data 30 and the second data 35 may be interchanged. For example, in the above case 3-1, the market data AD50 may be used as the second data 35, and the text data AD51 may be used as the first data 30.

[0177] [Model] Each model (51, 52, 53, 55, 56, 57) is configured as a machine learning model. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used in the calculation of the desired inference. The machine learning model may be configured, for example, as a neural network, a regression model, a decision tree model, a support vector machine, or other functional formula (calculation model). The machine learning method may be appropriately selected depending on the machine learning model employed (for example, backpropagation, etc.).

[0178] In one example, each model (51, 52, 53, 55, 56, 57) may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a long short-term memory (LSTM). The neural network may include any type of model such as a diffusion model, a transformer model, or a generative model. The connection weights between each node included in the neural network and the threshold value of each node are examples of calculation parameters.

[0179] The fusion model (53, 56) receives input of the calculation outputs of the first model (51, 55) and the second model (52, 56). The structure of the fusion model (53, 57) is not particularly limited and may be appropriately determined depending on the embodiment, as long as the result of the inference task can be derived from the calculation outputs of the first model (51, 55) and the second model (52, 56). In one example, the fusion model (53, 57) may have a structure that integrates the calculation outputs of the first model (51, 55) and the second model (52, 56) and derives an inference result from the integrated calculation output. The structure of the fusion model (53, 57) may employ a known structure proposed in, for example, Non-Patent Document 1. The fusion model (53, 57) may be configured separately from the first model (51, 55) and the second model (52, 56), or may be configured integrally with at least a part of the first model (51, 55) and the second model (52, 56). Note that the calculation outputs of the first model (51, 55) and the second model (52, 56) input to the fusion model (53, 57) may be at least one of the intermediate outputs and the final outputs of the first model (51, 55) and the second model (52, 56).

[0180] Machine learning involves adjusting (optimizing) values ​​of computational parameters using training samples. Typically, the trained models (51, 52, 53) may be generated by supervised learning using multiple training data sets, each of which is composed of a combination of input samples (training samples) and output samples (teacher signals, labels). However, the method for generating the trained models (51, 52, 53) is not limited to this example and may be modified as appropriate depending on the embodiment.

[0181] In one example of this embodiment, the first model 51 and the second model 52 each individually learn the ability to derive an inference result through the first training and the second training. Furthermore, the fusion model 53 learns the ability to derive an inference result through the third training, while using the calculation outputs of the first model 51 and the second model 52 in the first training and the second training as training samples. The learning of the fusion model 53 through this third training is also reflected in the first model 51 and the second model 52. By executing this third training together with the first training and the second training, each of the first model 51 and the second model 52 can learn the ability to derive an inference result while reflecting knowledge of the other model's data.

[0182] For example, if each model (51, 52, 53) is configured by a neural network, the training method may employ an error backpropagation method. Accordingly, in the first training, the first training sample 300 is provided to the first model 51, and an error calculated therefrom is backpropagated, thereby adjusting the value of the calculation parameter of the first model 51. In the second training, the second training sample 350 is provided to the second model 52, and an error calculated therefrom is backpropagated, thereby adjusting the value of the calculation parameter of the second model 52. In the third training, the first training sample 300 is provided to the first model 51, and the second training sample 350 is provided to the second model 52, and forward calculation processing of each model (51, 52, 53) is performed, thereby deriving the output (inference result) of the fusion model 53. The error of the output of the fusion model 53 may be backpropagated to the first model 51 and the second model 52 via the fusion model 53. By combining the error from this third training with the errors from the first training and the second training, knowledge of the other's data can be reflected in the learning of each of the first model 51 and the second model 52. In one example of this embodiment, as a result of this training, each model (51, 52) can more accurately extract features suitable for inferring each of the data (30, 35), and this can be expected to improve the inference accuracy of each model (51, 52).

[0183] Note that, as long as an inference result can be derived, the input / output format of each model (51, 52, 53, 55, 56, 57) may be appropriately determined depending on the embodiment. In one example, the input data (first data 30, second data 35) may be provided to the first model (51, 55) and the second model (52, 56) as is, or may be provided after preprocessing. Samples of data other than the first data 30 and the second data 35 may also be provided to the first model (51, 55) and the second model (52, 56). Furthermore, in one example, the output of each model (51, 52, 53, 55, 56, 57) may be configured to directly or indirectly indicate the inference result. When the output of each model (51, 52, 53, 55, 56, 57) is configured to indirectly indicate the inference result, the inference result may be obtained by performing any information processing (such as interpretation processing) on ​​the output of each model (51, 52, 53, 55, 56, 57).

[0184] [Method of Selecting Training Samples] The training samples (300, 350) may be collected as appropriate depending on the embodiment. The collection of the training samples (300, 350) may be performed manually or at least partially automatically. Furthermore, the training samples (300, 350) may be collected by the model generation device 1 or by a computer other than the model generation device 1.

[0185] In one example, the inference task may be determining whether or not a predetermined event has occurred, as in the above examples. The predetermined event may be, for example, the above-mentioned disease, abnormality, stock price rise / fall, or a transition to another specific state. Furthermore, at least one of the first data 30 and the second data 35 may be obtained in the form of a sample group (e.g., a group of images, a group of sound samples, etc.). When the inference task is determining whether or not a predetermined event has occurred and the data is obtained in the form of a sample group, features related to the occurrence of the predetermined event may only appear in a portion of the sample group. For example, as in Example 1-1 above, when diagnosing a disease, it is typical that lesions related to the disease appear in only a portion of the images in a group of medical images (e.g., a group of CT images, a group of MRI images, etc.). Accordingly, the remaining samples included in the sample group may not exhibit the features (i.e., may be in the same state as if the predetermined event had not occurred). In this case, samples exhibiting features related to the occurrence of the predetermined event are preferably used as positive samples for machine learning. On the other hand, if the remaining samples that do not represent features are used as positive samples for machine learning, samples equivalent to negative samples may be trained as positive samples, which may result in a deterioration in the inference accuracy of the first model 51, the second model 52, and the fusion model 53.

[0186] This problem is likely to occur in cases where the amount of samples obtained from the same information source is uneven between the first data 30 and the second data 35, such as in the first example. That is, in the first example, the first data 30 may be obtained as a group of samples of the sensing data SD, while the second data 35 may be obtained as a single sample of data AD other than the sensing data SD. For example, in the above-mentioned Example 1-1, a typical example is where, for a single subject at a single point in time, the first data 30 is obtained as a group of medical image samples, and the second data 35 is obtained as a single response sample to a medical interview. The above problem can occur if the group of samples of the first data 30 and the sample of the second data 35 obtained from an information source where a predetermined event has occurred are treated collectively as positive samples. Therefore, in one example of this embodiment, when the inference task includes determining whether a predetermined event has occurred, samples may be selected using one of the following two selection methods.

[0187] (I) First Selection Method FIG. 6A schematically illustrates an example of a first selection method for samples according to this embodiment. In the example of FIG. 6A , multiple data sets DS10 may be provided, each of which is composed of a combination of a first sample group 3010 of the first data 30 and a second sample 351 of the second data 35. The first sample group 3010 of each data set DS10 includes multiple first samples 301. The first sample group 3010 and the second samples 351 of each data set DS10 are obtained from a common information source (individuals). That is, the data set DS10 is formed by associating the first sample group 3010 and the second samples 351 obtained from the same information source. The same individual may be considered to be the same information source, or the same individual may be considered to be from different information sources at different times. The multiple data sets DS10 may be collected by the model generation device 1 or may be provided from a computer other than the model generation device 1. Each sample (301, 351) may be composed of one or more pieces of data (e.g., one or more images).

[0188] (Positive Data Set) Each data set DS10 (at least one of the first sample group 3010 and the second sample 351) may be associated with a truth value of the inference task via a label. The label may be referred to as a teacher signal or ground truth data. The multiple data sets DS10 may include one or more positive data sets PS10 associated with a truth value indicating the occurrence of a predetermined event. For each of the one or more positive data sets PS10, the likelihood of the occurrence of the predetermined event for each of the multiple first samples 301 included in the first sample group 3010 may be calculated using a computational model MD51 configured to calculate the likelihood of the occurrence of the predetermined event from the first data 30.

[0189] As long as the likelihood can be calculated, the configuration of the computation model MD51 is not particularly limited and may be appropriately selected depending on the embodiment. The computation model MD51 may be configured using at least one of a machine learning model and a rule-based model. When a machine learning model is used, the computation model MD51 may be trained in advance. The computation model MD51 may be configured similarly to the first model 51. Calculating the likelihood may be regressing a probability. When the inference task includes regressing a probability (likelihood), the first model 51 may be used for the computation model MD51. On the other hand, the rule-based model is configured to derive an inference result (calculation of likelihood in this embodiment) from a given input in accordance with a rule. The rule may be set appropriately. Note that calculating the likelihood of the first sample 301 may be performed by the model generation device 1 or by a computer other than the model generation device 1. The likelihood of the first sample 301 may be calculated during machine learning of each model (51, 52, 53) or may be calculated in advance.

[0190] In machine learning, among the multiple first samples 301 included in the first sample group 3010 of each of one or more positive data sets PS10, the first sample 301 with the highest calculated likelihood may be used as the first training sample 300. In addition, in machine learning, the corresponding second sample 351 may be used as the second training sample 350 to be used together with the first training sample 300 (first sample 301). That is, the first sample 301 and the second sample 351 that correspond to each other as the data set DS10 (positive data set PS10) may be used as the first training sample 300 and the second training sample 350 to be simultaneously provided to the first model 51 and the second model 52. Note that using the first sample 301 with the highest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the highest calculated likelihood as the first training samples 300 (n is a natural number greater than or equal to 1). The other first samples 301 may not be used for machine learning (i.e., may be excluded from the machine learning target).

[0191] According to one example of this embodiment, by selecting the first sample 301 based on the likelihood calculated using the computational model MD51, it is possible to use the first sample 301 that is highly likely to exhibit characteristics of the occurrence of a predetermined event as a positive sample for machine learning. This reduces the probability of using a sample equivalent to a negative sample as a positive sample, and as a result, it is expected that the deterioration of the inference accuracy of the trained models (51, 52, 53) can be suppressed.

[0192] 6A , the plurality of data sets DS10 may further include one or more negative data sets NS10 associated with a true value indicating that a predetermined event has not occurred. As with the positive data set PS10, for each of the one or more negative data sets NS10, the likelihood of each of the plurality of first samples 301 included in the first sample group 3010 may be calculated using the calculation model MD51.

[0193] In machine learning, among the multiple first samples 301 included in the first sample group 3010 of each of one or more negative data sets NS10, the first sample 301 with the highest calculated likelihood may be used as the first training sample 300. Furthermore, in machine learning, the corresponding second sample 351 may be used as the second training sample 350 used together with the first training sample 300 (first sample 301). That is, the first sample 301 and the second sample 351 corresponding to each other in the data set DS10 (negative data set NS10) may be used as the first training sample 300 and the second training sample 350 to be simultaneously provided to the first model 51 and the second model 52. Note that, as with the positive data set PS10, in the negative data set NS10, using the first sample 301 with the highest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the highest calculated likelihood as the first training samples 300 (n is a natural number greater than or equal to 1).

[0194] According to one example of this embodiment, among the multiple first samples 301 included in the negative data set NS10, the first sample 301 that is most likely to be positive (the first sample that is prone to inference error) can be used as a negative sample for machine learning. This can be expected to improve the inference accuracy of the trained model (51, 52, 53).

[0195] Note that the method of selecting the first sample 301 in the negative data set NS10 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, in machine learning, among the multiple first samples 301 included in each of the first sample groups 3010 of one or more negative data sets NS10, the first sample 301 with the lowest calculated likelihood may be used as the first training sample 300. In this way, by using the first sample 301 with the highest probability of being negative as the negative sample in machine learning, it is possible to train each model (51, 52, 53) to acquire the ability to narrow down to clearly negative cases and discriminate them as negative. Using the first sample 301 with the lowest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the lowest calculated likelihood as the first training samples 300 (n is a natural number greater than or equal to 1). In yet another example, the selection of the first sample 301 in the negative data set NS10 may be omitted, and any first sample 301 included in the first sample group 3010 may be used as the first training sample 300.

[0196] (II) Second Sorting Method Figure 6B schematically shows an example of a second sorting method for samples according to this embodiment. In the example of Figure 6B, similar to Figure 6A above, a plurality of data sets DS10 may be provided, each of which is composed of a combination of a first sample group 3010 of the first data 30 and a second sample 351 of the second data 35. The plurality of data sets DS10 may include one or more positive data sets PS10 and one or more negative data sets NS10.

[0197] 6A, a truth value of the inference task may be associated with each sample (301, 302). In one example, each first sample 301 included in the first sample group 3010 may be assigned a label 302, and each second sample 351 may be assigned a label 352. Each label 302 may be configured to indicate the truth value of the inference task for the associated first sample 301. The label 352 may be configured to indicate the truth value of the inference task for the associated second sample 351.

[0198] In machine learning, each of the multiple first samples 301 included in the first sample group 3010 of each dataset DS10 may be used as a first training sample 300, and the second sample 351 may be used as a second training sample 350. That is, the first sample 301 and the second sample 351 corresponding to each other as the dataset DS10 may be used as the first training sample 300 and the second training sample 350 to be simultaneously given to the first model 51 and the second model 52.

[0199] If the true value of the inference task associated with the first sample 301 matches the true value of the inference task associated with the second sample 351, the first, second, and third training may be performed. On the other hand, if the true value of the inference task associated with the first sample 301 does not match the true value of the inference task associated with the second sample 351, the first and second training may be performed, but the third training may be omitted. That is, if the values ​​of the labels (302, 352) of the first sample 301 and the second sample 351 match, the third training may be performed on the first model 51, the second model 52, and the fusion model 53. On the other hand, if the values ​​of the labels (302, 352) do not match, the third training may not be performed on the first model 51, the second model 52, and the fusion model 53.

[0200] According to one example of this embodiment, whether or not to perform the third training is controlled depending on whether the associated true values ​​(label values) match. This prevents the first sample 301 and the second sample 351, whose true values ​​for the inference task differ from each other, from being used simultaneously in the third training of the fusion model 53. As part of this effect, it is possible to avoid training samples equivalent to negative samples as positive samples. As a result, it is expected that the deterioration of inference accuracy in the trained models (51, 52, 53) can be suppressed.

[0201] 6A and 6B, the first data 30 is obtained as a sample group, and the above-described methods are applied to sorting samples (first samples 301) of the first data 30. However, the data obtained as a sample group does not have to be limited to the first data 30. Instead of the first data 30, or together with the first data 30, second data 35 may be obtained as a sample group. When the second data 35 is obtained as a sample group, the above-described sorting methods may be applied to the samples of the second data 35.

[0202] [Control of Machine Learning] Controlling machine learning may include executing machine learning within the model generation device 1 and issuing instructions to an external computer to execute machine learning. That is, in one example, the model generation device 1 may execute machine learning of the first model 51, the second model 52, and the fusion model 53. In another example, the model generation device 1 may issue instructions to an external computer to execute machine learning of the first model 51, the second model 52, and the fusion model 53. The external computer may execute machine learning of the first model 51, the second model 52, and the fusion model 53 in response to instructions from the model generation device 1. In another example, the model generation device 1 may execute part of the machine learning processing of the first model 51, the second model 52, and the fusion model 53, and the external computer may execute the remaining processing in response to instructions from the model generation device 1. Note that the external computer may be composed of one or more computers. For example, the external computer is a server device. The external computer may be connected to the model generation device 1 via a network or directly connected to the model generation device 1. The type of network is not particularly limited, and may be selected appropriately depending on the embodiment.

[0203] [Outputting Machine Learning Results] Outputting the machine learning results may be configured by outputting any information related to the machine learning results. In one example, outputting the machine learning results may include generating learning result data indicating the machine learning results and storing the generated learning result data in any storage area. In another example, outputting the machine learning results may include outputting a convergence result of training in machine learning. The convergence result may be indicated by, for example, a loss, a number of time steps, a learning curve, etc.

[0204] [System Configuration] In one example, as shown in Figure 1, a model generation device 1 and an inference device 2 may be connected to each other via a network. The type of network may be selected as appropriate from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between each device is not limited to this example and may be selected as appropriate depending on the embodiment. In another example, data may be exchanged using a storage medium.

[0205] 1, the model generation device 1 and the inference device 2 are separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. In another example, the model generation device 1 and the inference device 2 may be configured as a single computer. In yet another example, at least one of the model generation device 1 and the inference device 2 may be configured as multiple computers.

[0206] §2 Configuration Example [Hardware Configuration] (Model Generation Device) Fig. 7 shows a schematic example of the hardware configuration of the model generation device 1 according to this embodiment. The model generation device 1 according to this embodiment is a computer in which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0207] The control unit 11 includes a hardware processor such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, with a hard disk drive or a solid-state drive. The storage unit 12, the RAM, and the ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the generation program 81, a plurality of first training samples 300, a plurality of second training samples 350, first model data 510, second model data 520, and fusion model data 530.

[0208] The generation program 81 is a program for causing the model generation device 1 to execute information processing (see FIG. 11 , described later) related to machine learning for each model (51, 52, 53). The generation program 81 includes a series of instructions for the information processing. The first model data 510 is configured to indicate information about the trained first model 51 generated by machine learning. The second model data 520 is configured to indicate information about the trained second model 52 generated by machine learning. The fusion model data 530 is configured to indicate information about the trained fusion model 53 generated by machine learning. As long as information for executing computational processing of the trained models (51, 52, 53) can be retained, the configurations of the first model data 510, the second model data 520, and the fusion model data 530 are not particularly limited and may be determined appropriately depending on the embodiment. The first model data 510, the second model data 520, and the fusion model data 530 may each be configured to include information indicating values ​​of computational parameters adjusted by machine learning. In some cases, the first model data 510, the second model data 520, and the fusion model data 530 may each be configured to further include information indicating the configuration of each model (51, 52, 53) (e.g., the structure of a neural network, etc.). The storage units of the first model data 510, the second model data 520, and the fusion model data 530 may be selected arbitrarily. In one example, the first model data 510, the second model data 520, and the fusion model data 530 may be held in separate files, or may be held at least partially integrated. In this embodiment, the first model data 510, the second model data 520, and the fusion model data 530 may be generated as a result of executing the generation program 81.

[0209] The communication interface 13 is configured to perform wired or wireless communication via a network. The communication interface 13 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, etc. The model generation device 1 may perform data communication with another computer (e.g., the inference device 2, etc.) via the communication interface 13.

[0210] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the model generation device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device by wire or wirelessly, for example, via a USB (Universal Serial Bus) port, a dedicated port, etc. The input device 14 and the output device 15 may be integrated into one device, for example, a touch panel display, etc.

[0211] The drive 16 is a device for reading various information, such as programs, stored in the storage medium 91. At least one of the generation program 81, the first training samples 300, and the second training samples 350 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (such as stored programs) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The model generation device 1 may acquire at least one of the generation program 81, the first training samples 300, and the second training samples 350 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface.

[0212] Note that, with regard to the specific hardware configuration of the model generation device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. At least one of the first training samples 300 and the second training samples 350 may be stored in an external storage device, such as a network-attached storage (NAS). At least one of the first model data 510, the second model data 520, and the fusion model data 530 may be stored in at least one of the storage medium 91 and the external storage device. The model generation device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be identical. Furthermore, the model generating device 1 may be an information processing device designed specifically for the service to be provided, or may be a general-purpose server device, a general-purpose PC (Personal Computer), a tablet PC, a terminal device, or the like.

[0213] 8 shows a schematic diagram of an example of the hardware configuration of the inference device 2 according to this embodiment. The inference device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, a drive 26, and an external interface 27 are electrically connected.

[0214] The control unit 21 to the drive 26 and the storage medium 92 of the inference device 2 may be configured similarly to the control unit 11 to the drive 16 and the storage medium 91 of the model generation device 1. The control unit 21 (CPU) is an example of a processor resource of the inference device 2. The storage unit 22 (and RAM, ROM) is an example of a memory resource of the inference device 2. In this embodiment, the storage unit 22 stores various information such as an inference program 82, first model data 550, second model data 560, and fusion model data 570.

[0215] The inference program 82 is a program for causing the inference device 2 to execute information processing (see FIG. 12 , described below) related to the performance of an inference task. The inference program 82 includes a series of instructions for the information processing. The first model data 550 is configured to indicate information related to the first model 55. The second model data 560 is configured to indicate information related to the second model 56. The fusion model data 570 is configured to indicate information related to the fusion model 57. The first model data 550, the second model data 560, and the fusion model data 570 may be configured similarly to the first model data 510, the second model data 520, and the fusion model data 530. When the trained first model 51 is used as the first model 55, the first model data 550 may be the first model data 510. When the trained second model 52 is used as the second model 56, the second model data 560 may be the second model data 520. When the trained fusion model 53 is used as the fusion model 57, the fusion model data 570 may be the fusion model data 530. At least one of the first model data 550, the second model data 560, and the fusion model data 570 may be incorporated into the inference program 82. At least one of the inference program 82, the first model data 550, the second model data 560, and the fusion model data 570 may be stored in a storage medium 92 instead of or together with the storage unit 22. The inference device 2 may acquire at least one of the inference program 82, the first model data 550, the second model data 560, and the fusion model data 570 from the storage medium 92.

[0216] The inference device 2 may communicate data with another computer (e.g., the model generation device 1) via the communication interface 23. An operator can operate the inference device 2 by using the input device 24 and the output device 25.

[0217] The external interface 27 is configured to connect to an external device via a wired or wireless connection. The external interface 27 may be, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like. The type and number of external interfaces 27 may be determined appropriately depending on the embodiment. In one example of this embodiment, when a sensor (sensor S, first sensor S1, second sensor S2, etc.) is used to acquire a sample, the inference device 2 may be connected to the sensor via at least one of the communication interface 23 and the external interface 27.

[0218] Note that, with regard to the specific hardware configuration of the inference device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the communication interface 23, input device 24, output device 25, drive 26, and external interface 27 may be omitted. The inference device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The inference device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, etc.

[0219] [Software Configuration] (Model Generation Device) Fig. 9 schematically shows an example of the software configuration of the model generation device 1 according to this embodiment. The control unit 11 of the model generation device 1 executes instructions included in the generation program 81 stored in the storage unit 12 using a CPU. As a result, the model generation device 1 operates as a computer including an acquisition unit 111, a learning processing unit 112, and an output processing unit 113 as software modules. That is, in this embodiment, each software module of the model generation device 1 is realized by the control unit 11 (CPU).

[0220] The acquisition unit 111 is configured to acquire a plurality of first training samples 300 of the first data 30 and a plurality of second training samples 350 of the second data 35. The learning processing unit 112 is configured to control machine learning of the first model 51, the second model 52, and the fusion model 53. The machine learning includes first training for the first model 51, second training for the second model 52, and third training for each model (51, 52, 53). The output processing unit 113 is configured to output the results of the machine learning.

[0221] 10 schematically shows an example of the software configuration of the inference device 2 according to this embodiment. The control unit 21 of the inference device 2 executes instructions included in the inference program 82 stored in the storage unit 22 using a CPU. As a result, the inference device 2 operates as a computer equipped with an acquisition unit 211, an inference unit 212, and an output processing unit 213 as software modules. That is, in this embodiment, similar to the model generation device 1, each software module of the inference device 2 is also realized by the control unit 21 (CPU).

[0222] The acquisition unit 211 is configured to acquire a first sample 305 of the first data 30 and a second sample 355 of the second data 35. By holding the first model data 550, the second model data 560, and the fusion model data 570, the inference unit 212 comprises a first model 55, a second model 56, and a fusion model 57. The inference unit 212 is configured to derive a result of an inference task from the acquired first sample 305 and second sample 355 using the first model 55, the second model 56, and the fusion model 57. Deriving a result of the inference task includes deriving a first interim result of the inference task from the first sample 305 using the first model 55, deriving a second interim result of the inference task from the second sample 355 using the second model 56, deriving a third interim result of the inference task from the calculation output of the first model 55 obtained in the process of deriving the first interim result and the calculation output of the second model 56 obtained in the process of deriving the second interim result using the fusion model 57, and deriving a result of the inference task according to the first interim result, the second interim result, and the third interim result. The output processing unit 213 is configured to output information related to the derived result of the inference task.

[0223] (Other) In this embodiment, an example is described in which each software module of the model generating device 1 and the inference device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the model generating device 1 and the inference device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0224] §3 Operational Example [Model Generation Device] Figure 11 is a flowchart showing an example of the processing procedure of the model generation device 1 according to this embodiment. The following processing procedure is an example of a model generation method executed by a computer. However, the following processing procedure of the model generation device 1 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0225] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 to acquire a plurality of first training samples 300 of the first data 30 and a plurality of second training samples 350 of the second data 35.

[0226] In one example, the data types of the first training sample 300 and the second training sample 350 may be selected from any of the above-mentioned cases 1 to 3. As a specific example, the data types of the first training sample 300 and the second training sample 350 may be selected from any of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2.

[0227] In one example, the inference task may be determining whether a predetermined event has occurred. Accordingly, the control unit 11 may acquire multiple data sets DS10 as shown in FIG. 6A . The multiple data sets DS10 may include one or more positive data sets PS10, and the likelihood of each first sample 301 included in the first sample group 3010 of the positive data set PS10 may be calculated. Furthermore, in one example, the multiple data sets DS10 may include one or more negative data sets NS10, and the likelihood of each first sample 301 included in the first sample group 3010 of the negative data set NS10 may also be calculated.

[0228] In another example, the control unit 11 may acquire multiple data sets DS10 shown in Fig. 6B. Each first sample 301 included in the first sample group 3010 may be assigned a label 302, and each second sample 351 may be assigned a label 352. Each label (302, 352) may be appropriately configured to indicate the truth value of the inference task for each assigned sample (301, 351).

[0229] When machine learning is performed in an external computer, the control unit 11 may acquire the first training samples 300 and the second training samples 350 and provide the acquired first training samples 300 and the second training samples 350 to the external computer. Alternatively, acquiring the first training samples 300 and the second training samples 350 may be configured by issuing an instruction to the external computer to cause the external computer to acquire the first training samples 300 and the second training samples 350. After acquiring the first training samples 300 and the second training samples 350, the control unit 11 proceeds to the next step S102.

[0230] (Step S102) In step S102, the control unit 11 operates as the learning processing unit 112 and controls the machine learning of the first model 51, the second model 52, and the fusion model 53. The acquired multiple first training samples 300 and second training samples 350 are used in the machine learning. The machine learning includes first training for the first model 51 using the first training samples 300, second training for the second model 52 using the second training samples 350, and third training for each model (51, 52, 53) using each training sample (300, 350).

[0231] In one example, the control unit 11 may perform machine learning of each model (51, 52, 53) in the model generation device 1. In another example, the control unit 11 may issue an instruction to an external computer to perform machine learning of each model (51, 52, 53). The external computer may perform machine learning of each model (51, 52, 53) in response to the instruction from the model generation device 1.

[0232] 6A , for each positive data set PS10, the first sample 301 with the highest calculated likelihood may be used as the first training sample 300. Also, for each negative data set NS10, the first sample 301 with the highest or lowest calculated likelihood may be used as the first training sample 300. For each data set (PS10, NS10), the corresponding second sample 351 may be used as the second training sample 350.

[0233] In another example, when multiple data sets DS10 shown in FIG. 6B are acquired, each first sample 301 included in each data set DS10 may be used as the first training sample 300, and each second training sample 350 may be used as the second training sample 350. For each data set DS10, it may be determined whether the true value (value of label 302) associated with each first sample 301 included in the first sample group 3010 matches the true value (value of label 352) associated with each second sample 351. If the true values ​​of the first sample 301 and the second sample 351 match, first training, second training, and third training may be performed. On the other hand, if the true values ​​of the first sample 301 and the second sample 351 do not match, the first training and second training may be performed, but the third training may be omitted.

[0234] As a result of performing the machine learning, trained models (51, 52, 53) that have acquired the ability to perform an inference task within the category of the datasets used (first training sample 300, second training sample 350) can be generated. By selecting any of the above-mentioned first to third cases as the data type of the first training sample 300 and the second training sample 350, trained models (51, 52, 53) that have acquired the ability to perform the inference task of any of the cases can be generated. By selecting any of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2 as the data type of the first training sample 300 and the second training sample 350, trained models (51, 52, 53) that have acquired the ability to perform the inference task of any of the cases can be generated. When the machine learning process is completed, the control unit 11 proceeds to the next step S103.

[0235] (Step S103) In step S103, the control unit 11 operates as the output processing unit 113 to output the results of the machine learning. The output destination and the content of the information to be output may be selected appropriately depending on the embodiment.

[0236] In one example, the control unit 11 may generate model data (510, 520, 530) indicating each trained model (51, 52, 53) as the output process and store the generated model data (510, 520, 530) in a predetermined storage area. In this case, the model data (510, 520, 530) is an example of output information related to the results of machine learning. The predetermined storage area may be, for example, RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, or the like. The external storage device may be, for example, a data server such as a NAS. The external storage device may be, for example, an external storage device. When the machine learning calculation process is performed on an external computer, the model data (510, 520, 530) may be generated by the external computer. In another example, the control unit 11 may output the convergence results of the machine learning training as the output process. The output destination may be, for example, RAM in the control unit 11, the output device 15, an external computer, an external storage device, a storage medium, or a combination thereof. In yet another example, the control unit 11 may output the progress of machine learning training as the output process. In this case, the process of step S103 may be executed together with step S102.

[0237] When the output of the machine learning results is completed, the control unit 11 ends the processing procedure of the model generating device 1 according to this operation example.

[0238] When the inference device 2 uses the trained first model 51 as the first model 55, the generated first model data 510 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method. In one example, the first model data 510 may be provided to the inference device 2 via a network, a storage medium, or the like. In another example, the first model data 510 may be pre-installed in the inference device 2. The same applies to the second model 56 and the fusion model 57. When the trained second model 52 is used as the second model 56, the second model data 520 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method. When the trained fusion model 53 is used as the fusion model 57, the fusion model data 530 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method.

[0239] Furthermore, the control unit 11 may periodically or irregularly repeat the processes of steps S101 to S103 to update or newly generate at least one of the first model data 510, the second model data 520, and the fusion model data 530. During this repetition, at least a portion of the first training samples 300 and the second training samples 350 may be changed, modified, added, deleted, or the like as appropriate. The control unit 11 may then provide at least one of the updated or newly generated first model data 510, the second model data 520, and the fusion model data 530 to the inference device 2 as appropriate. In this way, at least one of the first model data 510 (550), the second model data 520 (560), and the fusion model data 530 (570) held by the inference device 2 may be updated.

[0240] [Inference Device] Figure 12 is a flowchart showing an example of a processing procedure of the inference device 2 according to this embodiment. The following processing procedure is an example of an inference method executed by a computer. However, the following processing procedure of the inference device 2 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0241] (Step S201) In step S201, the control unit 21 operates as the acquisition unit 211 and acquires the first sample 305 of the first data 30 and the second sample 355 of the second data 35.

[0242] In one example, the data type of the first sample 305 and the second sample 355 may be selected from any of the above-described first to third cases. As a specific example, the data type of the first sample 305 and the second sample 355 may be selected from any of the above-described cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2. In one example, the control unit 21 may acquire at least one of the first sample 305 and the second sample 355 directly or indirectly from a sensor. After acquiring the first sample 305 and the second sample 355, the control unit 21 proceeds to the next step S202.

[0243] (Step S202) In step S202, the control unit 21 operates as the inference unit 212 and derives the result of the inference task from the acquired first sample 305 and second sample 355 using the first model 55, the second model 56 and the fusion model 57.

[0244] The control unit 21 provides the first sample 305 to the first model 55 and the second sample 355 to the second model 56, and executes arithmetic processing of the first model 55, the second model 56, and the fusion model 57. The arithmetic processing may be executed appropriately depending on the embodiment, such as the configuration of each model (55, 56, 57). For example, if each model (55, 56, 57) is configured by a neural network, the control unit 21 may execute forward arithmetic processing of each model (55, 56, 57). As a result of executing this arithmetic processing, the control unit 21 can obtain a first interim result from the first model 55, a second interim result from the second model 56, and a third interim result from the fusion model 57.

[0245] The control unit 21 derives a final result of the inference task by ensembling the first interim result, the second interim result, and the third interim result. The ensembling method may be selected appropriately depending on the embodiment. A known method may be adopted as the ensembling method. For example, if the inference task is classification, the control unit 21 may calculate a simple average or weighted average of the values ​​derived as each interim result. Alternatively, the control unit 21 may calculate a simple sum or weighted sum of the values ​​derived as each interim result. When a weighted average or weighted sum is adopted, the weight of each interim result may be set arbitrarily. The control unit 21 may then determine the classification result by comparing the calculated value with a threshold. Furthermore, for example, if the inference task is regression, the control unit 21 may calculate a simple average or weighted average of the values ​​derived as each interim result. The control unit 21 may acquire the calculated value as the final result.

[0246] In one example, the first model 55 may be the trained first model 51. The second model 56 may be the trained second model 52. The fusion model 57 may be the trained fusion model 53.

[0247] In one example, any one of the above-mentioned first to third cases may be selected as the data type of the first sample 305 and the second sample 355. Any one of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2 may be selected as the data type of the first sample 305 and the second sample 355. Each model (55, 56, 57) may be configured to have the ability to perform an inference task of any one of the cases. This makes it possible to derive a result of the inference task corresponding to any one of the cases. After deriving the result of the inference task, the control unit 21 proceeds to the next step S203.

[0248] (Step S203) In step S203, the control unit 21 operates as the output processing unit 213 and outputs information relating to the result of the inference task.

[0249] The output destination and the content of the information to be output may be selected appropriately depending on the embodiment. In one example, the control unit 21 may simply output the result of the inference task derived in step S202. In another example, the control unit 21 may execute arbitrary information processing based on the result of the derived inference task. The control unit 21 may then output the result of the information processing as information related to the result of the inference task. The output of the result of the information processing may include outputting a specific message according to the inference result, controlling the operation of the controlled device according to the inference result, etc. The output destination may be, for example, RAM, the storage unit 22, the output device 25, another computer, the controlled device, etc. Depending on whether any of the above cases is selected as the data type of the first sample 305 and the second sample 355, the output destination and the content of the information to be output may be determined according to the above case.

[0250] When the output of information is completed, the control unit 21 ends the processing procedure of the inference device 2 according to this operation example. The control unit 21 may execute the series of processes from step S201 to step S203 at any timing, such as a user operation or when a condition is satisfied. The control unit 21 may also execute the series of processes from step S201 to step S203 in real time, or may execute them as a post-event inference process.

[0251] [Features] In this embodiment, in step S102, training of the fusion model 53 is performed in parallel with training of each model (51, 52) that derives an inference result individually for each data (30, 35). This allows the fusion model 53 to acquire the ability to derive an inference result from the features of each data (30, 35), while allowing each model (51, 52) to acquire the ability to extract features suitable for inferring each data (30, 35). As a result, it is expected that the possibility of impairing the features suitable for inferring each data (30, 35) can be reduced. Therefore, according to this embodiment, it is expected that the inference accuracy of the generated trained models (51, 52, 53) can be improved.

[0252] Furthermore, in this embodiment, in step S202, the robustness of the inference can be improved by ensembling the inference results of the first model 55, the second model 56, and the fusion model 57. This can be expected to improve the inference accuracy. The inference accuracy can also be expected to be improved by using the trained first model 51, the trained second model 52, and the trained fusion model 53 for the first model 55, the trained second model 56, and the trained fusion model 57, respectively.

[0253] §4 Modifications Although the embodiments of the present invention have been described above in detail, the above description is merely an example of the present invention in every respect. The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradictions arise. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, below, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of similar points to those in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

[0254] <4.1> Even if the machine learning of one of the first model 51 and the second model 52 is omitted, it is possible to implement the machine learning of the other model and the fusion model 53. Even if the machine learning of one model is omitted, it is possible to cause the fusion model 53 to acquire the ability to derive an inference result, and to cause the other model to acquire the ability to extract features suitable for inference. Therefore, in the above embodiment, the configuration related to the machine learning of one of the first model 51 and the second model 52 may be omitted.

[0255] FIG. 13 schematically illustrates an example of a machine learning process according to a modified example. FIG. 13 illustrates an example of a scene in which the configuration related to the machine learning of the second model 52 is omitted. In this modified example, the model generation device 1 may control the machine learning of the first model 51 and the fusion model 53 and output the results of the machine learning. The first model 51 may be configured to derive an inference result from the first data 30, as in the above embodiment. A second model 52 may be provided that is configured to derive a result of an inference task from second data 35 of a type different from the first data 30. The fusion model 53 may be configured to derive a result of an inference task from the calculation outputs of the first model 51 and the second model 52. The machine learning may include performing a first training on the first model 51 so that the result of an inference task derived by the first model 51 from the first training samples 300 of the first data 30 conforms to a true value, and, together with the first training, performing training (fusion training) on ​​the first model 51 and the fusion model 53 so that the result of the inference task derived by the fusion model 53 from the calculation output of the first model 51 when deriving the result of the inference task and the calculation output of the second model 52 obtained by providing the second model 52 with the second training samples 350 of the second data 35 conforms to a true value. With this modification as well, improvement in the inference accuracy of the trained models (51, 53) can be expected.

[0256] Note that the target for which the machine learning configuration is omitted does not have to be the second model 52. Instead of the second model 52, the machine learning configuration of the first model 51 may be omitted. In this case, the model generation device 1 may control the machine learning of the second model 52 and the fusion model 53 and output the machine learning results. The first model 51 may be provided as appropriate. The machine learning may include performing second training on the second model 52 so that the result of the inference task derived by the second model 52 from the second training samples 350 of the second data 35 conforms to the true value, and, together with the second training, performing training (fusion training) on ​​the second model 52 and the fusion model 53 so that the result of the inference task derived by the fusion model 53 from the calculation output of the first model 51 obtained by providing the first training samples 300 of the first data 30 to the first model 51 and the calculation output of the second model 52 when deriving the result of the inference task conforms to the true value. This modification is also expected to improve the inference accuracy of the trained models (52, 53).

[0257] <4.2> In the above embodiment, an example has been described in which the result of an inference task is derived from two types of data, the first data 30 and the second data 35. However, the number of types of data used for inference is not limited to two, and may be three or more. Deriving the result of an inference task from the first data 30 and the second data 35 may be configured by deriving the result of the inference task from three or more types of data including the first data 30 and the second data 35. Accordingly, in one example, the fusion model (53, 57) may be configured to accept input of three or more types of data. In another example, the fusion model (53, 57) may be provided for each combination of two or more types of data.

[0258] (First Modification) Fig. 14A schematically illustrates an example of the configuration of a fusion model (53, 57) according to a modification. In the example of Fig. 14A, a third model (61, 65) is provided that is configured to derive a result of an inference task from third data of a type different from the first data 30 and the second data 35. The fusion model (53, 57) may be configured to further receive inputs of the calculation outputs of the first model (51, 55), the second model (52, 56), and the third model (61, 65), and to derive a result of the inference task from the input calculation outputs of the first model (51, 55), the second model (52, 56), and the third model (61, 65).

[0259] In the learning stage, the machine learning may further include training the third model so that a result of an inference task derived by the third model 61 from a third training sample of the third data conforms to a true value. In the third training, the first model 51, the second model 52, the third model 61, and the fusion model 53 may be trained so that a result of an inference task derived by the fusion model 53 from the calculation outputs of the first model 51, the second model 52, and the third model 61 when deriving a result of the inference task conforms to a true value. Alternatively, as in <4.1> above, the machine learning configuration of the third model 61 may be omitted.

[0260] Meanwhile, in the inference stage, the inference device 2 may further acquire a third sample of the third data. The inference device 2 may further derive an interim result of the inference task from the acquired third sample using a third model 65. The fusion model 57 may receive the calculation outputs of the first model 55, the second model 56, and the third model 65 obtained in the process of deriving the interim result. The inference device 2 may derive a final result of the inference task from the interim results of the first model 55, the second model 56, the third model 65, and the fusion model 57. In one example, the trained third model 61 may be used for the third model 65.

[0261] According to this modification, improvement in inference accuracy can be expected in situations where the result of an inference task is derived from three or more types of data. The model to be added does not have to be limited to the third model (61, 65). Fourth and subsequent models may also be present. The fusion model (53, 57) may be configured to further accept input of the calculation outputs of the fourth and subsequent models and derive the result of the inference task from the input calculation outputs.

[0262] (Second Modification) FIG. 14B schematically illustrates another example of the configuration of a fusion model according to a modification. In the example of FIG. 14B , similar to FIG. 14A , a third model (61, 65) is provided that is configured to derive the result of an inference task from third data of a type different from the first data 30 and the second data 35. A fusion model (531, 571) is provided between the first model (51, 55) and the second model (52, 56). A fusion model (532, 572) is provided between the second model (52, 56) and the third model (61, 65). The fusion models (531, 571) and (532, 572) are examples of fusion models (53, 57). A fusion model may also be provided between the first model (51, 55) and the third model (61, 65).

[0263] In the learning stage, the fusion model 531 may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications. The fusion model 532 may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications, except that the first model 51 is replaced with the third model 61. When a fusion model is also provided between the first model 51 and the third model 61, the fusion model may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications, except that the second model 52 is replaced with the third model 61.

[0264] Meanwhile, in the inference stage, the inference device 2 may further acquire a third sample of the third data. The inference device 2 may further derive a provisional result of the inference task from the acquired third sample using a third model 65. The inference device 2 may provide the calculation outputs of the first model 55 and the second model 56 obtained in the process of deriving the provisional result to the fusion model 571 and execute the calculation processing of the fusion model 571 to acquire the provisional result of the inference task from the fusion model 571. Similarly, the inference device 2 may provide the calculation outputs of the second model 56 and the third model 65 obtained in the process of deriving the provisional result to the fusion model 572 and execute the calculation processing of the fusion model 572 to acquire the provisional result of the inference task from the fusion model 572. The inference device 2 may derive a final result of the inference task from the provisional results of the first model 55, the second model 56, the third model 65, and each fusion model (571, 572). When a fusion model is also provided between the first model 55 and the third model 65, the inference device 2 may obtain an interim result of the inference task from the fusion model by providing the calculation outputs of the first model 55 and the third model 65 obtained in the process of deriving the interim result to the fusion model and performing calculation processing on the fusion model. The inference device 2 may further use this interim result to derive the final result of the inference task. In one example, the trained third model 61 may be used for the third model 65. The trained fusion model 531 may be used for the fusion model 571. The trained fusion model 532 may be used for the fusion model 572.

[0265] According to this modification, improved inference accuracy can be expected in situations where the results of an inference task are derived from three or more types of data. The model to be added does not have to be limited to the third model (61, 65). A fourth model and subsequent models may also exist. Fusion models (53, 57) may also be provided between the fourth model and subsequent models. Furthermore, in the example of FIG. 14B, fusion models (531, 532, 571, 572) are arranged for each combination of two types of data. However, the relationship between the fusion models (53, 57) and the data types is not limited to this example. Similar to the example of FIG. 14A, fusion models (53, 57) may be arranged for each of three or more types of data.

[0266] §5 Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

[0267] [Preliminary Preparation] As preliminary preparation, we prepared two types of datasets. The first dataset was the Stanford University pulmonary embolism dataset (PE dataset) used in Reference 1 (Yuyin Zhou, et al. "RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR", [online], [searched March 22, 2024], Internet URL: https: / / arxiv.org / abs / 2111.11665). This PE dataset consisted of a group of CT images and clinical data extracted from the electronic medical records of the same patients. The clinical data was organized in a table format and included questionnaire responses. Specifically, the clinical data included 1,505 attributes, such as age, gender, and pulse rate. Using the method proposed in Reference 2 (Avanti Shrikumar, et al. "Learning Important Features Through Propagating Activation Differences", [online], [searched March 22, 2024], Internet URL: https: / / arxiv.org / abs / 1704.02685), 250 attributes useful for identifying pulmonary embolism were selected, and the table data consisting of the selected 250 attributes was used for machine learning. Each CT image was resized from 256 × 256 to 224 × 224. The size of each CT image was then converted to 192 × 192 by cropping the center. The intensity of each CT image was normalized to the range [-1, 1]. The PE dataset contained a total of 1,837 combinations of CT image groups and clinical data (table data), of which 1,454 combinations were used for training, 190 combinations for validation, and 193 combinations for testing. Each combination consisted of 100 CT images. Five samples (192 × 192 × 24) were obtained by dividing the 100 CT images into 24 consecutive CT images.In the fifth sample (group), the four CT images were expanded to 24 CT images.

[0268] The second dataset was the NACC dataset used in reference 3 (Duane L Beekly, et al. "The National Alzheimer's Coordinating Center (NACC) Database: an Alzheimer disease database", [online], [searched March 22, 2024], internet <URL: https: / / pubmed.ncbi.nlm.nih.gov / 15592144 / >). In this NACC dataset, cognitive impairment levels were classified into three categories: 0 (normal), 1 (mild), and 2 (severe). The NACC dataset consisted of brain MRI images and questionnaire response data (table format). The MRI images included T1, T2, and FLAIR images. Among these, T1 images were selected and bone removal, resizing, and intensity normalization were performed using imaging software (FreeSurfer). After this preprocessing, 1,252 MRI images (3D volumes) measuring 182 × 218 × 182 voxels were obtained. The volume intensities were normalized to the range [-8, 8]. Meanwhile, preprocessing using zero padding, feature selection, and normalization was applied to the response data. After preprocessing, the response data (table data) consisted of 99 attributes, including cognitive questionnaire information, gender, age, etc. This resulted in 1,252 combinations of MRI images and response data (table data). Of these, 752 combinations were used for training, 249 combinations for validation, and 251 for testing. The NACC dataset was employed for machine learning of multiclass classification tasks, including classification of cognitive impairment levels.

[0269] [Model Configuration] (First Comparative Example) FIG. 15A shows the model configuration of the first comparative example. The model of the first comparative example was configured to classify the presence or absence of each disease from each image. The model structure of the first comparative example adopted the PENet architecture proposed in Reference 4 (Shih-Cheng Huang, et al. "PENet - a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging", [online], [searched March 22, 2024], Internet <URL: https: / / www.nature.com / articles / s41746-020-0266-y>).

[0270] (Second Comparative Example) FIG. 15B shows the model configuration of the second comparative example. The model of the second comparative example was configured to classify the presence or absence of each disease from tabular data. The model structure of the second comparative example employed the Tabular Transformer proposed in Reference 5 (Yury Gorishniy, et al. "Revisiting Deep Learning Models for Tabular Data", [online], [searched March 22, 2024], Internet <URL: https: / / arxiv.org / abs / 2106.11959>).

[0271] (Third Comparative Example) Figure 15C shows the model configuration of the third comparative example. The model of the third comparative example was configured to classify the presence or absence of each disease from images and table data. The model structure of the third comparative example employed the MMTM architecture proposed in Non-Patent Document 1. The PENet architecture, similar to that of the first comparative example, was employed for the part that extracts image features. The Tabular Transformer, similar to that of the second comparative example, was employed for the part that extracts table data features.

[0272] (Fourth Comparative Example) FIG. 15D shows the model configuration of the fourth comparative example. Similar to the third comparative example, the model of the fourth comparative example was configured to classify the presence or absence of each disease from image and table data. The model structure of the fourth comparative example employed the CLIP architecture proposed in Reference 6 (Paul Hager, et al. "Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data", [online], [searched March 22, 2024], Internet <URL: https: / / arxiv.org / abs / 2303.14080>). The ENet architecture, similar to that of the first comparative example, was employed for extracting image features. The Tabular Transformer, similar to that of the second comparative example, was employed for extracting table data features. As shown in FIG. 15D, the model of the fourth comparative example was configured to fuse the features of each data set via a communication unit during the learning phase. On the other hand, in the inference stage, the model of the fourth comparative example was configured to obtain a final inference result by ensembling the inference results (first output and second output) derived from the parts that extract features of various data, without using a communication part. The ensemble was performed by simple averaging.

[0273] (Fifth Comparative Example) The model of the fifth comparative example was constructed using the models of the first comparative example and the second comparative example. Specifically, the model of the fifth comparative example was configured to derive an inference result by ensembling the inference results of the models of the first comparative example and the second comparative example. As a result, the model of the fifth comparative example was also configured to classify the presence or absence of each disease from image and table data, similar to the third comparative example, etc. The ensemble was performed using a simple average.

[0274] (Sixth Comparative Example) Figure 15E shows the model configuration of the sixth comparative example. The model of the sixth comparative example was prepared to master the classification task of the NACC dataset. The model of the sixth comparative example was configured to classify cognitive impairment levels from MRI images and tabular data. A convolutional neural network was used to extract image features. A tabular transformer similar to the second comparative example was used to extract tabular data features.

[0275] (First Example) FIG. 16A shows the model configuration of the first example. The model of the first example was configured to include a first model, a second model, and a fusion model. The first model employed the same PENet architecture as the first comparative example. The second model employed the same Tabular Transformer as the second comparative example. The fusion model employed the same MMTM architecture as the third comparative example. In the inference phase, the model of the first example was configured to obtain a final inference result by ensembling the inference results of the first model, the second model, and the fusion model. The ensemble was performed by simple averaging. The main differences between the first example and the third comparative example were that the first model and the second model each had a section that output the inference result, and that in the learning phase, machine learning was performed on the fusion model while individual machine learning was performed on the first model and the second model.

[0276] Second Example: Figure 16B shows the model configuration of the second example. The model of the second example was configured by replacing the fusion model in the model configuration of the first example with a concatenation-based fusion model including a Transformer block. In all other respects, the model of the second example was configured in the same way as the first example.

[0277] (Third Embodiment) Fig. 16C shows the model configuration of the third embodiment. The model of the third embodiment is configured by replacing the fusion model in the model configuration of the first embodiment with the TokenFusion-based fusion model proposed in Reference 7 (Yikai Wang, et al. "Multimodal Token Fusion for Vision Transformers", [online], [searched on March 22, Reiwa 6], Internet <URL: https: / / arxiv.org / abs / 2204.08721>). In other respects, the model of the third embodiment is configured in the same manner as the first embodiment.

[0278] (Fourth Embodiment) Fig. 16D shows the model configuration of the fourth embodiment. The model of the fourth embodiment is configured by replacing the fusion model in the model configuration of the first embodiment with the Co-attention-based fusion model proposed in Reference 8 (Pei Li, et al. "Multimodal Fusion with Co-attention Mechanism", [online], [searched on March 22, Reiwa 6], Internet <URL: https: / / ieeexplore.ieee.org / document / <9190483>). In other respects, the model of the fourth embodiment is configured in the same manner as the first embodiment.

[0279] (Fifth Example) FIG. 16E shows the model configuration of the fifth example. The model of the fifth example was prepared to master the classification task of the NACC dataset. In the model of the fifth example, a first model was constructed by adding a fully connected (linear) layer to the part of the model of the sixth comparative example that extracts image features. A second model was constructed by adding a fully connected (linear) layer to the part of the model of the sixth comparative example that extracts table data features. As a result, the model of the fifth example was configured to include a first model, a second model, and a fusion model. In the inference phase, the model of the fifth example, like the first example, was configured to ensemble the inference results of the first model, the second model, and the fusion model to obtain a final inference result. The ensemble was performed by simple averaging. The main differences between the fifth example and the sixth comparative example were that the first model and the second model each had a part that outputted the inference results, and that in the learning phase, machine learning was performed on the fusion model while individual machine learning was performed on the first model and the second model.

[0280] (Sixth Example) Figure 16F shows the model configuration of the sixth example. Like the fifth example, the model of the sixth example was prepared to learn the classification task of the NACC dataset. The model of the sixth example was constructed by replacing the fusion model in the model configuration of the fifth example with the same fusion model as the second example. In all other respects, the model of the sixth example was constructed in the same way as the fifth example.

[0281] (Other) In each figure, "Transformer" refers to the Transformer proposed in Reference 5 (Comparative Example 2). "PENet" refers to the PENet encoder proposed in Reference 4 (Comparative Example 1). "Transformer block" refers to the basic configuration of a Transformer consisting of multi-head attention and a feedforward network. "Linear" refers to a fully connected (linear) layer. "AP" and "Avgpool" refer to average pooling. "Maxpooling" and "Maxpool" refer to maximum value pooling. "TokenFusion" refers to the TokenFusion block. "Co-attention" refers to the Co-attention block. "Conv" refers to convolution. "Conv3D" refers to 3D convolution. "Reshape" refers to the Reshape function. "Flatten" refers to flattening (reducing to one dimension). "Concat" refers to channel-wise concatenation. "BN" refers to batch normalization. "ReLu" refers to the ReLu function. "CLS Emb" indicates embedding of class tokens.

[0282] [First Experiment] Using the PE dataset and the model of Example 1, the first selection method (FIG. 6A) and the second selection method (FIG. 6B) were compared. Specifically, both the first selection method and the second selection method were employed under two conditions to obtain trained models.

[0283] First, in the first selection method, the likelihood of each of the five samples was calculated using the pre-trained PENet obtained in Reference 1. Then, the sample with the highest likelihood, both positive and negative, was used for machine learning, and the other samples were excluded from machine learning. On the other hand, in the second selection method, a label was assigned to each of the five samples, and training of the fusion model (third training) was performed if the labels of the CT image sample and the table data matched. If the labels of the CT image sample and the table data did not match, training of the fusion model was omitted.

[0284] In the first condition ("without pre-training") for each selection method, only the first model (PENet) was pre-trained on the Kinetics-600 dataset (see Joao Carreira, et al. "A Short Note about Kinetics-600", [online], [searched March 22, 2024], Internet URL: https: / / arxiv.org / abs / 1808.01340). In the second condition ("with pre-training"), both the first model (PENet) and the second model (Tabular Transformer) were pre-trained using the PE dataset.

[0285] After pre-training under each of the above conditions, we used each selection method and performed machine learning using 1,454 combinations to obtain a trained model. The learning rate was 10 -4 The number of epochs was set to 1,000. The optimization algorithm used was AdamW from Reference 10 (Ilya Loshchilov, et al. "Decoupled Weight Decay Regularization", [online], [searched March 22, 2024], Internet URL: https: / / arxiv.org / abs / 1711.05101). The loss function L was defined by the following equation 1.

[0286] L ｃ denotes the cross-entropy loss. i , z t and z f denotes the output of the first model, the second model, and the fusion model. i , y t and y f denotes the labels for the first model, the second model, and the fusion model. The regularization parameters are λ = 0.1, λ = 0.1, and λ 3 = 0.8. 190 validation combinations were used to adjust the values ​​of each hyperparameter. The resulting trained models (first model, second model, and fusion model) were then used to perform a classification task on 193 test combinations, obtaining classification results for each combination. Accuracy evaluation values ​​were calculated from the resulting classification results. The evaluation values ​​used were the Area Under the Precision-Recall Curve (AUPRC), Area Under the ROC Curve (AUROC), Accuracy (ACC), Matthews Correlation Coefficient (MCC), and Recall.

[0287] [Second Experiment] (A) Experiment 2A Using the PE dataset, trained models for the first to fifth comparative examples and the first to fourth examples were generated. CT image samples were used to train the model for the first comparative example. Table data was used to train the model for the second comparative example. CT image samples and table data were used to train the models for the third and fourth comparative examples. The trained model for the fifth comparative example was constructed from the trained models for the first and second comparative examples. The second condition ("with prior learning") of the first selection method in the first experiment was used to select the CT image samples for the first to fourth examples, the third and fourth comparative examples. The other conditions were set the same as in the first experiment. As a result, trained models for the first to fifth comparative examples and the first to fourth examples were obtained. Using the obtained trained models, a classification task was performed on 193 test combinations, and classification results for each combination were obtained. Accuracy evaluation values ​​were calculated from the obtained classification results.

[0288] ROC (Receiver Operating Characteristic) curves were calculated for the first to third comparative examples, the fifth comparative example, and the first example. Furthermore, to evaluate the accuracy of each part of the first example, (i) the first model only, (ii) the second model only, (iii) the fusion model only, (iv) the ensemble of the first and second models, and (v) the ensemble of the first, second, and fusion models (first example) were used to calculate the evaluation values ​​related to accuracy. AUPRC, AUROC, ACC, MCC, and Recall were used as the evaluation values ​​to be calculated.

[0289] Furthermore, to visualize the classification process in the first, third, and fourth comparative examples and the first example, we designed a 3D Grad-CAM model and fitted it to each model. For the Grad-CAM model, we adopted the architecture proposed in Reference 11 (Ramprasaath R. Selvaraju, et al. "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], [searched March 22, 2024], Internet URL: https: / / arxiv.org / abs / 1610.02391) after improving it to be applicable to multiple inputs and multiple outputs. This resulted in a Grad-CAM heatmap.

[0290] (B) Experiment 2B Using the NACC dataset, trained models for the first comparative example, the second comparative example, the fifth comparative example, the sixth comparative example, the fifth example, and the sixth example were generated. Specifically, for the fifth and sixth examples, pre-training of the first and second models was performed using the NACC dataset, similar to the second condition of the first experiment. After pre-training, trained models were obtained by machine learning using 752 combinations. In the first comparative example, a trained model was obtained by machine learning using MRI images. In the second comparative example, a trained model was obtained by machine learning using table data. Note that the number of parameters for the models of the first and second comparative examples was changed from that in Experiment 2A to fit the NACC dataset. In the fifth and sixth comparative examples, trained models were obtained by machine learning using MRI images and table data. The learning rate was 10 -3 The number of epochs was set to 100. The optimization algorithm used was Adam. 249 validation combinations were used to adjust the values ​​of each hyperparameter. The resulting trained model was then used to perform a classification task on 251 test combinations, obtaining classification results for each combination. Accuracy evaluation values ​​were calculated from the obtained classification results. The evaluation values ​​used were ACC, MCC, Recall, Jaccard coefficient, and F1 score.

[0291] [Experimental Results] Figure 17 shows the results of Experiment 1. Figure 18A shows the results of Experiment 2A. Figure 18B shows the calculation results of ROC curves for each Comparative Example and each Example in Experiment 2A. Figure 18C shows the evaluation results for each model of Example 1 in Experiment 2A. Figure 18D shows heat maps generated by the models of each Comparative Example and Example. Figure 19 shows the results of Experiment 2B.

[0292] 17, it was found that both the first and second selection methods were able to obtain good trained models. Furthermore, from the viewpoint of improving the accuracy of the trained model, it was found that the first selection method was superior under the condition with pre-learning, and the second selection method was superior under the condition without pre-learning.

[0293] Furthermore, in Experiment 2A, the results shown in Figures 18A and 18B showed that each Example outperformed each Comparative Example in all indices except Recall. In Experiment 2B, the results shown in Figure 19 showed that each Example outperformed each Comparative Example in all indices. These results verified that the training method and inference method of the above-described embodiment can be expected to improve the accuracy of the trained model. Furthermore, when comparing the results of the first model shown in Figure 18C with the results of the first Comparative Example shown in Figure 18A, the performance of the first model of Example 1 exceeded that of the model of the first Comparative Example in terms of the indices AUPRC, AUROC, and MCC. Furthermore, as shown in Figure 18D, the generated heat map showed that Example 1 more accurately captured the location of pulmonary embolism than the comparative examples. These results suggest that by training the first and second models together with the fusion model, the first and second models can incorporate knowledge of the other model's data into their own training, thereby potentially improving inference accuracy.

[0294] The present specification includes the following disclosure: [Supplementary Note 1] A computer (1) executes the steps of: controlling machine learning of a first model (51), a second model (52), and a fusion model (53); and outputting the results of the machine learning, wherein the first model (51) is configured to derive a result of an inference task from first data (30), the second model (52) is configured to derive the result of the inference task from second data (35) of a type different from the first data (30), and the fusion model (53) is configured to derive the result of the inference task from a calculation output of the first model (51) and a calculation output of the second model (52), and the machine learning includes: performing a first training on the first model (51) so that the result of the inference task derived by the first model (51) from a first training sample (300) of the first data (30) conforms to a true value; A model generation method comprising: performing second training on the second model (52) so that a result of the inference task derived by the second model (52) from a second training sample (350) of the second data (35) conforms to a true value; and performing third training on the first model (51), the second model (52), and the fusion model (53) together with the first training and the second training so that a result of the inference task derived by the fusion model (53) from the calculation output of the first model (51) and the calculation output of the second model (52) when deriving the result of the inference task conforms to a true value.[Supplementary Note 2] The model generation method according to Supplementary Note 1, wherein the inference task is to determine whether or not a predetermined event has occurred, and a plurality of datasets are provided, each of which is composed of a combination of a first sample group of the first data (30) and a second sample of the second data (35), and the plurality of datasets include one or more positive datasets associated with a true value indicating the occurrence of the predetermined event, and for each of the one or more positive datasets, a likelihood of each of a plurality of first samples included in the first sample group is calculated using a computational model configured to calculate the likelihood of the occurrence of the predetermined event from the first data (30), and in the machine learning, a first sample with the highest calculated likelihood is used as the first training sample (300) from among the plurality of first samples included in the first sample group of each of the one or more positive datasets. [Supplementary Note 3] The model generation method described in Supplementary Note 2, wherein the plurality of data sets further include one or more negative data sets associated with a true value indicating no occurrence of the predetermined event, and for each of the one or more negative data sets, a likelihood of each of a plurality of first samples included in the first sample group is calculated using the computational model, and in the machine learning, a first sample with the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more negative data sets is used as the first training sample (300). [Supplementary Note 4] The model generation method according to Supplementary Note 1, wherein the inference task is to determine whether or not a predetermined event has occurred, a plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data (30) and a second sample of the second data (35), and in the machine learning, each of the plurality of first samples included in the first sample group of each of the data sets is used as the first training sample (300), and the second sample is used as the second training sample (350), and if the true value of the inference task associated with the first sample does not match the true value of the inference task associated with the second sample, performing the third training is omitted.[Supplementary Note 5] An inference program (82) for causing a computer (2) to execute the steps of: acquiring a first sample (305) of first data (30) and a second sample (355) of second data (35) of a type different from the first data (30); deriving a result of an inference task from the acquired first sample (305) and second sample (355) using a first model (55), a second model (56), and a fusion model (57); and outputting information related to the derived result of the inference task, wherein the step of deriving a result of the inference task includes: deriving a first interim result of the inference task from the first sample (305) using the first model (55); and deriving a second interim result of the inference task from the second sample (355) using the second model (56). An inference program (82) comprising: using the fusion model (57) to derive a third interim result of the inference task from a calculation output of the first model (55) obtained in the process of deriving the first interim result and a calculation output of the second model (56) obtained in the process of deriving the second interim result; and deriving a result of the inference task according to the derived first interim result, the second interim result, and the third interim result. [Supplementary Note 6] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is sensing data obtained by a sensor, and the second data (35) is other than the sensing data. [Supplementary Note 7] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is medical image data of a subject, the second data (35) is response data to a medical interview given to the subject, and the inference task includes determining whether or not the subject has a disease. [Appendix 8] An inference program (82) according to Appendix 5, wherein the first data (30) is image data of an object or sound data relating to the object, the second data (35) is work history data relating to the object, and the inference task includes determining whether or not an abnormality has occurred in the object.[Supplementary Note 9] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is image data depicting a subject or audio data relating to the subject, the second data (35) is text data relating to the subject, and the inference task includes inferring emotions of the subject. [Supplementary Note 10] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is first sensed data by a first sensor, and the second data (35) is second sensed data by a second sensor of a different type from the first sensor. [Supplementary Note 11] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is image data depicting a subject in a chemical plant, the second data (35) is numerical data relating to the environment of the subject in the chemical plant, and the inference task includes inferring a state of the subject. [Supplementary Note 12] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is image data of an object, the second data (35) is point cloud data obtained by measuring the object, and the inference task includes inferring a position of the object. [Supplementary Note 13] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is image data of an equipment, the second data (35) is sound data related to the equipment, and the inference task includes inferring a state of the equipment. [Supplementary Note 14] The inference program (82) according to Supplementary Note 5, wherein the first data (30) is image data of a crop, the second data (35) is weather data in an environment of the crop, and the inference task includes inferring a state of the crop.[Supplementary Note 15] An inference device including a control unit, wherein the control unit is configured to execute the steps of: acquiring a first sample (305) of first data (30) and a second sample (355) of second data (35) of a type different from the first data (30); deriving a result of an inference task from the acquired first sample (305) and the second sample (355) using a first model (55), a second model (56), and a fusion model (57); and outputting information related to the derived result of the inference task, wherein the step of deriving a result of the inference task includes: deriving a first interim result of the inference task from the first sample (305) using the first model (55); and deriving a second interim result of the inference task from the second sample (355) using the second model (56). using the fusion model (57) to derive a third interim result of the inference task from a calculation output of the first model (55) obtained in the process of deriving the first interim result and a calculation output of the second model (56) obtained in the process of deriving the second interim result; and deriving a result of the inference task in accordance with the derived first interim result, the second interim result, and the third interim result.[Supplementary Note 16] A computer (1) executes the steps of controlling machine learning of a first model (51) and a fusion model (53); and outputting the results of the machine learning, wherein the first model (51) is configured to derive a result of an inference task from first data (30), and a second model (52) configured to derive the result of the inference task from second data (35) of a type different from the first data (30) is provided, the fusion model (53) is configured to derive the result of the inference task from a calculation output of the first model (51) and a calculation output of the second model (52), and the machine learning includes: performing a first training on the first model (51) so that the result of the inference task derived by the first model (51) from a first training sample (300) of the first data (30) conforms to a true value; A model generation method comprising: together with the first training, training the first model (51) and the fusion model (53) so that the result of the inference task derived by the fusion model (53) from the calculation output of the first model (51) when deriving the result of the inference task and the calculation output of the second model (52) obtained by providing the second model (52) with second training samples (350) of the second data (35) conforms to a true value.

[0295] DESCRIPTION OF SYMBOLS 1...Model generation device, 11...Control unit, 12...Storage unit, 13...Communication interface, 14...Input device, 15...Output device, 16...Drive, 81...Generation program, 91...Storage medium, 111...Acquisition unit, 112...Learning processing unit, 113...Output processing unit, 2...Inference device, 21...Control unit, 22...Storage unit, 23...Communication interface, 24...Input device, 25...Output device, 26...Drive, 27...External interface, 82...Inference program, 92...Storage medium, 30...First data, 300...First training sample, 305...First sample, 35...Second data, 350...Second training sample, 355...Second sample, 51, 55...First model, 52, 56...Second model, 53, 57...Fusion model

Claims

1. A computer executes the steps of controlling machine learning of a first model, a second model, and a fusion model, and outputting a result of the machine learning, wherein the first model is configured to derive a result of an inference task from first data, the second model is configured to derive a result of the inference task from second data of a type different from the first data, and the fusion model is configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model, and the machine learning includes: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value, performing a second training of the second model so that a result of the inference task derived by the second model from a second training sample of the second data conforms to a true value, and performing a third training of the first model, the second model, and the fusion model together with the first training and the second training so that a result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the result of the inference task conforms to a true value. Model generation method.

2. The model generation method of claim 1, wherein the inference task is to determine whether or not a predetermined event has occurred, and wherein a plurality of datasets are provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data, and the plurality of datasets include one or more positive datasets associated with a true value indicating the occurrence of the predetermined event, and for each of the one or more positive datasets, a likelihood of each of a plurality of first samples included in the first sample group is calculated using a computational model configured to calculate the likelihood of the occurrence of the predetermined event from the first data, and in the machine learning, a first sample with the highest calculated likelihood is used as the first training sample among the plurality of first samples included in the first sample group of each of the one or more positive datasets.

3. The model generation method of claim 2, wherein the plurality of datasets further include one or more negative datasets associated with a true value indicating that the predetermined event has not occurred, and for each of the one or more negative datasets, the likelihood of each of a plurality of first samples included in the first sample group is calculated using the computational model, and in the machine learning, the first sample with the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more negative datasets is used as the first training sample.

4. The model generation method of claim 1, wherein the inference task is to determine whether or not a predetermined event has occurred, a plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data, and in the machine learning, each of the plurality of first samples included in the first sample group of each of the data sets is used as the first training sample, and the second sample is used as the second training sample, and if the true value of the inference task associated with the first sample does not match the true value of the inference task associated with the second sample, performing the third training is omitted.

5. An inference program for causing a computer to execute the steps of: acquiring a first sample of first data and a second sample of second data of a type different from the first data; deriving a result of an inference task from the acquired first sample and second sample using a first model, a second model, and a fusion model; and outputting information related to the derived result of the inference task, wherein the step of deriving a result of the inference task includes: deriving a first interim result of the inference task from the first sample using the first model; deriving a second interim result of the inference task from the second sample using the second model; deriving a third interim result of the inference task using the fusion model from the calculation output of the first model obtained in the process of deriving the first interim result and the calculation output of the second model obtained in the process of deriving the second interim result; and deriving the result of the inference task in accordance with the derived first, second, and third interim results.

6. The inference program according to claim 5, wherein the first data is sensing data obtained by a sensor, and the second data is other than the sensing data.

7. An inference program as described in claim 5, wherein the first data is medical image data of the subject, the second data is response data to a medical interview given to the subject, and the inference task includes determining whether or not a disease has occurred in the subject.

8. An inference program as described in claim 5, wherein the first data is image data depicting an object or sound data relating to the object, the second data is work history data relating to the object, and the inference task includes determining whether or not an abnormality has occurred in the object.

9. The inference program of claim 5, wherein the first data is image data of a subject or audio data relating to the subject, the second data is text data relating to the subject, and the inference task includes inferring emotions of the subject.

10. The inference program according to claim 5, wherein the first data is first sensing data obtained by a first sensor, and the second data is second sensing data obtained by a second sensor of a different type from the first sensor.

11. The inference program of claim 5, wherein the first data is image data of an object within a chemical plant, the second data is numerical data relating to the environment of the object within the chemical plant, and the inference task includes inferring the state of the object.

12. An inference program as described in claim 5, wherein the first data is image data depicting an object, the second data is point cloud data obtained by measuring the object, and the inference task includes inferring the position of the object.

13. The inference program according to claim 5, wherein the first data is image data showing an equipment, the second data is sound data relating to the equipment, and the inference task includes inferring the state of the equipment.

14. The inference program of claim 5, wherein the first data is image data relating to a crop, the second data is weather data in an environment of the crop, and the inference task includes inferring a state of the crop.

15. An inference device including a control unit, wherein the control unit is configured to execute the steps of: acquiring a first sample of first data and a second sample of second data of a type different from the first data; deriving a result of an inference task from the acquired first sample and second sample using a first model, a second model, and a fusion model; and outputting information related to the derived result of the inference task, wherein the step of deriving the result of the inference task includes: deriving a first interim result of the inference task from the first sample using the first model; deriving a second interim result of the inference task from the second sample using the second model; deriving a third interim result of the inference task using the fusion model from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result; and deriving the result of the inference task in accordance with the derived first, second, and third interim results.

16. A model generation method, comprising: a computer executing the steps of controlling machine learning of a first model and a fusion model; and outputting a result of the machine learning, wherein the first model is configured to derive a result of an inference task from first data; a second model configured to derive a result of the inference task from second data of a type different from the first data is provided; the fusion model is configured to derive the result of the inference task from a calculation output of the first model and a calculation output of the second model; and the machine learning includes: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; and training the first model and the fusion model together with the first training so that a result of the inference task derived by the fusion model from a calculation output of the first model when deriving the result of the inference task conforms to a true value.

Images