Model generation method, inference program, and inference device

Abstract

To provide a technique which improves inference accuracy of a machine learning model.SOLUTION: A model generation device according to an aspect of the present invention performs first training on a first model so that a result of an inference task derived from a first training sample of first data by the first model matches a true value. The model generation device performs second training on a second model so that a result of the inference task derived from a second training sample of second data by the second model matches a true value. Together with the first training and the second training, the model generation device performs third training on the first model, the second model, and a fusion model so that a result of the inference task derived from an operation output of the first model and an operation output of the second model at the time of deriving the results of the inference task, by the fusion model matches a true value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a model generation method, an inference program, and an inference device. [Background technology]

[0002] In recent years, research has progressed on multimodal models that are configured to accept input of multiple types of data and derive inference results. For example, Non-Patent Document 1 proposes a method of building a fusion model using a convolutional neural network that fuses two types of data and derives inference results from the fused data. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Hamid Reza Vaezi Joze, et al. "MMTM: Multimodal Transfer Module for CNN Fusion", [online], [Retrieved March 22, 2024], Internet<URL:https: / / arxiv.org / abs / 1911.08670> Summary of the Invention [Problem to be solved by the invention]

[0004] The present inventors have found that the above-mentioned conventional methods have the following problem. That is, in the machine learning of the fusion model, the conventional method may prioritize features suitable for fusing various data, which may result in the loss of features suitable for inferring each individual data. As a result, it may be difficult to improve the inference accuracy of the fusion model.

[0005] In one aspect, the present invention has been made in view of the above circumstances, and its purpose is to provide a technology for improving the inference accuracy of machine learning models. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0007] A model generation method according to one aspect of the present invention is an information processing method in which a computer executes steps of controlling machine learning of a first model, a second model, and a fusion model, and outputting the results of the machine learning. The first model is configured to derive a result of an inference task from first data. The second model is configured to derive the result of the inference task from second data of a type different from the first data. The fusion model is configured to derive the result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning includes: performing a first training of the first model so that the result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; performing a second training of the second model so that the result of the inference task derived by the second model from a second training sample of the second data conforms to a true value; and performing a third training of the first model, the second model, and the fusion model together with the first training and the second training so that the result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the result of the inference task conforms to a true value.

[0008] In this configuration, training of the fusion model is performed in parallel with training of each model that derives inference results for each data individually. This allows the fusion model to have the ability to derive inference results from the characteristics of each data. While the fusion model acquires the feature, each individual model (first model, second model) can acquire the ability to extract features suitable for inferring each data (and thereby derive an inference result). Therefore, with this configuration, by training not only the fusion model but also each individual model in parallel, it is possible to reduce the possibility of losing features suitable for inferring each data. Therefore, it is expected that the inference accuracy of the generated trained model will be improved.

[0009] In the model generation method according to the above aspect, the inference task may be determining whether or not a predetermined event has occurred. A plurality of data sets may be provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data. The plurality of data sets may include one or more positive data sets associated with a true value indicating the occurrence of the predetermined event. For each of the one or more positive data sets, a likelihood of each of a plurality of first samples included in the first sample group may be calculated using a computational model configured to calculate the likelihood of the occurrence of the predetermined event from the first data. In the machine learning, a first sample with the highest calculated likelihood may be used as the first training sample among the plurality of first samples included in the first sample group of each of the one or more positive data sets.

[0010] When an inference task is to determine whether a predetermined event has occurred and data is obtained in the form of a group of samples (e.g., a group of images, a group of sound samples, etc.), it is possible that features related to the occurrence of the predetermined event appear only in some of the group of samples. For example, when diagnosing a disease, lesions related to the disease typically appear in only some of the images in a group of medical images. Accordingly, it is possible that the features do not appear in the remaining samples in the group of samples (i.e., the state is the same as when the predetermined event has not occurred). In this case, if the remaining samples are used as positive samples for machine learning, samples equivalent to negative samples will be trained as positive samples, which may result in a deterioration in the inference accuracy of the individual model and the fusion model. In contrast, with this configuration, by selecting first samples based on the likelihood calculated using a computational model, it is possible to use first samples that are highly likely to exhibit features of the occurrence of the predetermined event as first training samples for machine learning. As a result, it is possible to suppress a deterioration in the inference accuracy of the trained model.

[0011] In the model generation method according to the above aspect, the plurality of data sets may further include one or more negative data sets associated with a true value indicating that the predetermined event has not occurred. For each of the one or more negative data sets, the likelihood of each of a plurality of first samples included in the first sample group may be calculated using the computational model. In the machine learning, among the plurality of first samples included in the first sample group of each of the one or more negative data sets, the first sample with the highest calculated likelihood may be used as the first training sample. According to this configuration, among the plurality of first samples included in the negative data set, the first sample most likely to be positive (a first sample prone to inference error) can be used as the negative sample for machine learning. This can be expected to improve the inference accuracy of the trained model.

[0012] In the model generation method according to the above aspect, the inference task may be determining whether or not a predetermined event has occurred. A plurality of data sets may be provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data. In the machine learning, each of a plurality of first samples included in the first sample group of each of the data sets may be used as the first training sample, and the second sample may be used as the second training sample. If the true value of the inference task associated with the first sample does not match the true value of the inference task associated with the second sample, The third training may be omitted.

[0013] As described above, when the inference task is to determine whether a predetermined event has occurred and data is obtained in the form of a sample group, using a sample equivalent to a negative sample as a positive sample in machine learning may result in a deterioration in the inference accuracy of the model. In contrast, with this configuration, if the associated true values ​​do not match, training of the fusion model can be omitted, thereby avoiding training a sample equivalent to a negative sample as a positive sample. As a result, deterioration in the inference accuracy of the trained model can be suppressed.

[0014] Even if the machine learning of one of the first model and the second model is omitted, the machine learning of the individual model and the fusion model can be implemented. The fusion model can acquire the ability to derive an inference result, and the other individual model can acquire the ability to extract features suitable for inference of individual data. Therefore, in the model generation method according to each of the above aspects, the configuration related to the machine learning of one of the first model and the second model may be omitted.

[0015] For example, a model generation method according to one aspect of the present invention may be an information processing method in which a computer executes steps of controlling machine learning of a first model and a fusion model and outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. A second model may be provided that is configured to derive a result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; and, together with the first training, training the first model and the fusion model so that a result of the inference task derived by the fusion model from a calculation output of the first model when deriving the result of the inference task and a calculation output of the second model obtained by providing the second model with a second training sample of the second data conforms to a true value. According to this configuration, training of the fusion model is performed together with individual training of the first model. This is expected to improve the inference accuracy of trained models.

[0016] Furthermore, the present invention is not limited to the above-described model generation. One aspect of the present invention may be an inference program that uses each individual model and the fusion model. In one example, at least one of each individual model and the fusion model may be generated by the above-described model generation method.

[0017] For example, an inference program according to one aspect of the present invention may be a program for causing a computer to execute the steps of acquiring a first sample of first data and a second sample of second data of a type different from the first data, deriving a result of an inference task from the acquired first sample and the second sample using a first model, a second model, and a fusion model, and outputting information related to the derived result of the inference task. The step of deriving the result of the inference task may include deriving a first provisional result of the inference task from the first sample using the first model, deriving a second provisional result of the inference task from the second sample using the second model, deriving a third provisional result of the inference task from a calculation output of the first model obtained in the process of deriving the first provisional result and a calculation output of the second model obtained in the process of deriving the second provisional result using the fusion model, and deriving the result of the inference task according to the derived first, second, and third provisional results. According to this configuration, the inference results of each individual model (first model, second model) and the fusion model are analyzed. By sampling, the robustness of the inference can be increased, which is expected to improve the accuracy of the inference.

[0018] In the inference program according to the above aspect, the first data may be sensing data obtained by a sensor, and the second data may be data other than the sensing data. With this configuration, improvement in inference accuracy can be expected in situations where sensing data and other types of data other than sensing data are used as the first data and the second data.

[0019] In the inference program according to the above aspect, the first data may be medical image data of a subject, and the second data may be response data to a medical interview given to the subject. The inference task may include determining whether or not the subject has a disease. This configuration can be expected to improve inference accuracy when determining whether or not the subject has a disease.

[0020] In the inference program according to the above aspect, the first data may be image data of an object or sound data related to the object, and the second data may be work history data related to the object. The inference task may include determining whether or not an abnormality has occurred in the object. This configuration can be expected to improve inference accuracy in anomaly detection situations.

[0021] In the inference program according to the above aspect, the first data may be image data of a subject or audio data relating to the subject, and the second data may be text data relating to the subject. The inference task may include inferring emotions of the subject. This configuration can be expected to improve inference accuracy when inferring emotions of the subject.

[0022] In the inference program according to the above aspect, the first data may be first sensing data obtained by a first sensor, and the second data may be second sensing data obtained by a second sensor of a different type from the first sensor. With this configuration, improvement in inference accuracy can be expected in a situation where different types of sensing data are used as the first data and the second data.

[0023] In the inference program according to the above aspect, the first data may be image data of an object in a chemical plant, and the second data may be numerical data relating to the environment of the object in the chemical plant. The inference task may include inferring a state of the object. This configuration can be expected to improve inference accuracy when inferring the state of the object in a chemical plant.

[0024] In the inference program according to the above aspect, the first data may be image data of an object, and the second data may be point cloud data obtained by measuring the object. The inference task may include inferring the position of the object. This configuration can be expected to improve inference accuracy when inferring the position of the object.

[0025] In the inference program according to the above aspect, the first data may be image data of an equipment, and the second data may be sound data related to the equipment. The inference task may include inferring a state of the equipment. With this configuration, improvement in inference accuracy can be expected when inferring the state of the equipment.

[0026] In the inference program according to the above aspect, the first data may be image data relating to a crop, and the second data may be meteorological data in an environment of the crop. The inference task may include inferring the state of the crop. With this configuration, improvement in inference accuracy can be expected in inferring the state of the crop.

[0027] Note that the present invention is not limited to the above-described model generation method (information processing method). As another aspect of the model generation method according to each of the above aspects, one aspect of the present invention may be an information processing device (model generation device) that realizes all or part of the above-described configurations, a program, or a storage medium readable by a machine such as a computer on which such a program is stored. A storage medium readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action. Similarly, the present invention is not limited to the above-described inference program. As another aspect of the inference program according to each of the above aspects, one aspect of the present invention may be an information processing device (inference device) that realizes all or part of the above-described configurations, an information processing method (inference method), or a storage medium readable by a machine such as a computer on which a program is stored.

[0028] For example, a model generation device according to one aspect of the present invention may include a control unit. The control unit may be configured to execute the steps of controlling machine learning of a first model, a second model, and a fusion model, and outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. The second model may be configured to derive the result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive the result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include: performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value; performing a second training of the second model so that a result of the inference task derived by the second model from a second training sample of the second data conforms to a true value; and performing a third training of the first model, the second model, and the fusion model together with the first training and the second training so that a result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving the result of the inference task conforms to a true value.

[0029] Furthermore, for example, a model generation device according to an aspect of the present invention may include a control unit. The control unit may be configured to execute a step of controlling machine learning of a first model and a fusion model and a step of outputting a result of the machine learning. The first model may be configured to derive a result of an inference task from first data. A second model may be provided that is configured to derive a result of the inference task from second data of a type different from the first data. The fusion model may be configured to derive a result of the inference task from a calculation output of the first model and a calculation output of the second model. The machine learning may include performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data conforms to a true value, and, together with the first training, training the first model and the fusion model so that a result of the inference task derived by the fusion model from a calculation output of the first model when deriving the result of the inference task and a calculation output of the second model obtained by providing the second model with a second training sample of the second data conforms to a true value.

[0030] Also, for example, an inference device according to one aspect of the present invention may include a control unit, which performs the steps of acquiring a first sample of first data and a second sample of second data of a type different from the first data, and deriving a result of an inference task from the acquired first sample and second sample using a first model, a second model, and a fusion model; and outputting information about the derived result of the inference task. The step of deriving the result of the inference task may include deriving a first interim result of the inference task from the first sample using the first model, deriving a second interim result of the inference task from the second sample using the second model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result using the fusion model, and deriving the result of the inference task in accordance with the derived first interim result, the second interim result, and the third interim result.

[0031] Furthermore, for example, an inference method according to one aspect of the present invention may be an information processing method executed by a computer, the information processing method including the steps of acquiring a first sample of first data and a second sample of second data of a type different from the first data, deriving a result of an inference task from the acquired first sample and the second sample using a first model, a second model, and a fusion model, and outputting information related to the derived result of the inference task. The step of deriving a result of the inference task may include deriving a first interim result of the inference task from the first sample using the first model, deriving a second interim result of the inference task from the second sample using the second model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result using the fusion model, and deriving the result of the inference task based on the derived first, second, and third interim results. [Effects of the Invention]

[0032] According to the present invention, it is possible to expect improvement in inference accuracy. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2A] FIG. 2A schematically illustrates an example of each data configuration according to the embodiment. [Figure 2B] FIG. 2B schematically shows an example of each data structure according to the embodiment. [Figure 2C] FIG. 2C schematically illustrates an example of each data configuration according to the embodiment. [Figure 3A] FIG. 3A is a schematic diagram showing an example of Case 1-1 to which the present invention is applied. [Figure 3B] FIG. 3B schematically shows an example of the first and second cases to which the present invention is applied. [Figure 3C] FIG. 3C is a schematic diagram showing an example of the first to third cases to which the present invention is applied. [Figure 3D] FIG. 3D is a schematic diagram showing an example of the first to fourth cases to which the present invention is applied. [Figure 3E] FIG. 3E is a schematic diagram showing an example of Cases 1-5 to which the present invention is applied. [Figure 4A] FIG. 4A schematically shows an example of Case 2-1 to which the present invention is applied. [Figure 4B] FIG. 4B schematically shows an example of Case 2-2 to which the present invention is applied. [Figure 4C] FIG. 4C is a schematic diagram showing an example of the second and third cases to which the present invention is applied. [Figure 4D] FIG. 4D is a schematic diagram showing an example of the second to fourth cases to which the present invention is applied. [Figure 4E] FIG. 4E is a schematic diagram showing an example of Case 2-5 to which the present invention is applied. [Figure 4F] FIG. 4F schematically shows an example of Case 2-6 to which the present invention is applied. [Figure 4G] FIG. 4G schematically shows an example of Case 2-7 to which the present invention is applied. [Figure 4H] FIG. 4H is a schematic diagram showing an example of Case 2-8 to which the present invention is applied. [Figure 4I]FIG. 4I schematically shows an example of Case 2-9 to which the present invention is applied. [Figure 4J] FIG. 4J is a schematic diagram showing an example of Case 2-10 to which the present invention is applied. [Figure 5A] FIG. 5A schematically shows an example of Case 3-1 to which the present invention is applied. [Figure 5B] FIG. 5B schematically shows an example of Case 3-2 to which the present invention is applied. [Figure 6A] FIG. 6A schematically illustrates an example of a first sorting method for samples according to an embodiment. [Figure 6B] FIG. 6B schematically illustrates an example of a second method for sorting samples according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a hardware configuration of a model generating device according to an embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a hardware configuration of an inference device according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a software configuration of a model generating device according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the software configuration of the inference device according to the embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a processing procedure of the model generating device according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a processing procedure of the inference device according to the embodiment. [Figure 13] FIG. 13 shows a schematic diagram of an example of a machine learning process according to a modified example. [Figure 14A] FIG. 14A schematically illustrates an example of the configuration of a fusion model according to a modified example. [Figure 14B] FIG. 14B schematically illustrates an example of the configuration of a fusion model according to a modified example. [Figure 15A] FIG. 15A shows a model configuration of a first comparative example. [Figure 15B] FIG. 15B shows the model configuration of the second comparative example. [Figure 15C] FIG. 15C shows the model configuration of the third comparative example. [Figure 15D] FIG. 15D shows the model configuration of the fourth comparative example. [Figure 15E] FIG. 15E shows the model configuration of the sixth comparative example. [Figure 16A] FIG. 16A shows the model configuration of the first embodiment. [Figure 16B] FIG. 16B shows the model configuration of the second embodiment. [Figure 16C] FIG. 16C shows the model configuration of the third embodiment. [Figure 16D] FIG. 16D shows the model configuration of the fourth embodiment. [Figure 16E] FIG. 16E shows the model configuration of the fifth embodiment. [Figure 16F] FIG. 16F shows the model configuration of the sixth embodiment. [Figure 17] FIG. 17 shows the results of the first experiment. [Figure 18A] FIG. 18A shows the results of experiment 2A. [Figure 18B] FIG. 18B shows the calculation results of the ROC curves for each comparative example and each example in Experiment 2A. [Figure 18C] FIG. 18C shows the evaluation results of each model of the first embodiment in Experiment 2A. [Figure 18D] FIG. 18D shows the heat maps generated for each comparative example and each example. [Figure 19] FIG. 19 shows the results of experiment 2B. DETAILED DESCRIPTION OF THE INVENTION

[0034] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it is specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0035] §1 Application Examples 1 is a diagram showing an example of a situation in which the present invention is applied. The system according to this embodiment comprises a model generation device 1 and an inference device 2.

[0036] The model generation device 1 according to this embodiment is one or more computers configured to generate a trained model by controlling the implementation of machine learning.

[0037] The model generation device 1 controls the machine learning of a first model 51, a second model 52, and a fusion model 53. The first model 51 is configured to derive a result of an inference task (inference result) from first data 30. The second model 52 is configured to derive a result of the inference task from second data 35 of a type different from the first data 30. The fusion model 53 is configured to receive input of the calculation output of the first model 51 and the calculation output of the second model 52, and to derive a result of the inference task from the input calculation output of the first model 51 and the calculation output of the second model 52.

[0038] The machine learning includes performing a first training on the first model 51 so that the result of the inference task derived by the first model 51 from the first training sample 300 of the first data 30 conforms to the true value, performing a second training on the second model 52 so that the result of the inference task derived by the second model 52 from the second training sample 350 of the second data 35 conforms to the true value, and performing a third training on the first model 51, the second model 52 and the fusion model 53 together with the first and second training so that the result of the inference task derived by the fusion model 53 from the calculation output of the first model 51 and the calculation output of the second model 52 when deriving the result of the inference task conforms to the true value.

[0039] In one example, the third training may be performed in parallel with the first training and the second training. Furthermore, the calculation output of the first model 51 when deriving the result of the inference task may be the calculation output of the first model 51 when deriving the result of the inference task from the first training sample 300. The calculation output of the second model 52 when deriving the result of the inference task may be the calculation output of the second model 52 when deriving the result of the inference task from the second training sample 350. By performing this machine learning, a trained first model 51, a trained second model 52, and a trained fusion model 53 are generated. The model generation device 1 outputs the results of this machine learning.

[0040] On the other hand, the inference device 2 according to this embodiment is one or more computers configured to perform an inference task using a first model 55, a second model 56, and a fusion model 57. The inference device 2 acquires a first sample 305 of the first data 30 and a second sample 355 of the second data 35. The inference device 2 derives a result of the inference task from the acquired first sample 305 and second sample 355 using the first model 55, the second model 56, and the fusion model 57.

[0041] The first model 55 is configured to derive a result of an inference task from the first data 30. In one example, the first model 55 may be a trained first model 51 generated by the model generation device 1. The second model 56 is configured to derive a result of an inference task from second data 35 of a type different from the first data 30. In one example, the second model 56 may be a trained second model 52 generated by the model generation device 1. The fusion model 57 is configured to receive input of the calculation output of the first model 55 and the calculation output of the second model 56, and to derive a result of the inference task from the input calculation output of the first model 55 and the calculation output of the second model 56. In one example, the fusion model 57 may be a trained fusion model 53 generated by the model generation device 1.

[0042] Deriving the result of the inference task includes deriving a first interim result of the inference task from the first sample 305 using the first model 55, deriving a second interim result of the inference task from the second sample 355 using the second model 56, and using the fusion model 57 to calculate the calculation output of the first model 55 obtained in the process of deriving the first interim result and the process of deriving the second interim result. The process includes deriving a third interim result of the inference task from the calculation output of the second model 56 obtained in the step, and deriving a result of the inference task based on the derived first interim result, second interim result, and third interim result. This calculation derives the result of the inference task (inference result). The inference device 2 outputs information related to the derived result of the inference task.

[0043] In the model generation device 1 according to this embodiment, training of the fusion model 53 is performed in parallel with training of each model (51, 52) that derives an inference result for each data (30, 35) individually. This allows the fusion model 53 to acquire the ability to derive an inference result from the features of each data (30, 35), while allowing each model (51, 52) to acquire the ability to extract features suitable for inferring each data (30, 35). As a result, it is expected that the possibility of losing the features suitable for inferring each data (30, 35) can be reduced. Therefore, according to this embodiment, it is expected that the inference accuracy of the generated trained models (51, 52, 53) can be improved.

[0044] Furthermore, by training each model (51, 52) together with the fusion model 53, each model (51, 52) can reflect knowledge of the other model's data in its own training. That is, thanks to the presence of the fusion model 53, the first model 51 can be trained to acquire the ability to extract features from the first data 30 while using the features of the second data 35 as clues and to derive inference results from the extracted features. The second model 52 can be trained to acquire the ability to extract features from the second data 35 while using the features of the first data 30 as clues and to derive inference results from the extracted features. As a result of this training, each model (51, 52) can more accurately extract features suitable for inferring from each data (30, 35), which is expected to improve the inference accuracy of each model (51, 52).

[0045] Furthermore, in the inference device 2 according to this embodiment, the robustness of the inference can be improved by ensembling the inference results of the first model 55, the second model 56, and the fusion model 57. This is expected to improve the inference accuracy. Note that when the trained first model 51 is used as the first model 55, the trained first model 51 may be provided to the inference device 2 from the model generation device 1 at any timing and by any method. The same applies to the trained second model 52 and the fusion model 53.

[0046] [Inference task] The inference task may include at least one of regressing and identifying features that appear in the data. The inference task may also include prediction. The type of inference task is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the inference task may include diagnosing a disease, detecting an abnormality, detecting a location, determining an emotion, evaluating a state, evaluating a value, determining a type, etc.

[0047] [1st data / 2nd data] The first data 30 and the second data 35 are at least a part of the input data (explanatory variables) used for inference. The first data 30 and the second data 35 are not particularly limited and may be appropriately selected depending on the inference task (objective variables).

[0048] The types of the first data 30 and the second data 35 are not particularly limited and may be selected appropriately depending on the embodiment. In one example, the first data 30 and the second data 35 may each be image data, sound data, text data, numerical data, or a combination of these. The images may be composed of either still images or moving images. The first data 30 and the second data 35 may each be composed of raw data, or may be composed of analysis data (analysis results) obtained by analyzing raw data. The analysis may include, for example, extraction of data features, calculation of feature amounts, etc. Extraction of features may include, for example, analysis of voice, recognition of emotions, etc. The feature may include other inferences (regression or discrimination), etc. The feature may include statistics such as maximum, minimum, median, mean, variance, standard deviation, n-th percentile, etc.

[0049] The configuration of the first data 30 and the second data 35 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, at least one of the first data 30 and the second data 35 may be sensing data obtained by one or more sensors. The type of sensor may be selected appropriately depending on the embodiment, such as the inference task. The sensor may include, for example, an image sensor, a microphone, an acceleration sensor, an angular acceleration sensor, a gyro sensor, a motion capture device, a pressure gauge, a thermometer, a hygrometer, a blood pressure monitor, a pulse meter, a heart rate monitor, an electrocardiograph, an electromyograph, a clinical thermometer, a skin electrodermal response monitor, an electroencephalograph, a magnetoencephalograph, an activity monitor, a blood glucose monitor, an electrooculography sensor, an eye movement monitor, an odor sensor, a wearable device (such as a smart watch), a position sensor, a microwave sensor, an ultrasonic sensor, a weather sensor, a home security sensor, etc. The image sensor may include, for example, any sensor that acquires data in the form of an image or an image representation, such as an RGB camera, a depth sensor, an infrared sensor, radar, or LiDAR (light detection and ranging). The position sensor may be, for example, a GPS (Global Positioning System) sensor or a GNS The home security sensor may include, for example, an image sensor, an infrared sensor, an activity (sound) sensor, a gas (CO2, etc.) sensor, a current sensor, a smart meter (a sensor that measures the power consumption of home appliances, lighting, etc.), etc. The sensors may be appropriately arranged depending on the object to be observed. In addition, in one example, at least one of the first data 30 and the second data 35 may be data other than sensing data. Data other than sensing data may include, for example, information generated manually such as a user's response, input, operation, history, etc., and information generated by computer processing. Data obtained by observing a user's response using a sensor such as an image sensor may be considered sensing data or may be considered data other than sensing data. In one example of this embodiment, at least one of the following three configurations may be adopted for the first data 30 and the second data 35.

[0050] (1) First Case 2A schematically illustrates an example of the configuration of the first data 30 and the second data 35 in a first example of this embodiment. As illustrated in FIG. 2A, in one example, the first data 30 may be sensing data SD obtained by a sensor S. As described above, the type of sensor S may be selected as appropriate depending on the embodiment, such as the inference task. Meanwhile, the second data 35 may be data AD other than the sensing data SD. The second data 35 may be selected as appropriate from data other than the sensing data SD depending on the inference situation.

[0051] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the sensing data SD and a plurality of second training samples 350 of data AD other than the sensing data SD. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, in a situation where the sensing data SD and other types of data AD other than the sensing data SD are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0052] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the sensing data SD and a second sample 355 of data AD other than the sensing data SD. The inference device 2 may derive the result of the inference task from the acquired first sample 305 and second sample 355. According to one example of this embodiment, in a situation where the sensing data SD and other types of data AD other than the sensing data SD are used as the first data 30 and the second data 35, the inference results of each model (55, 56, 57) are ensembled to improve the inference accuracy. You can expect this.

[0053] (2) Second Case 2B schematically illustrates an example of the configuration of the first data 30 and the second data 35 in a second example of this embodiment. As illustrated in FIG. 2B, in one example, the first data 30 may be first sensing data SD1 obtained by a first sensor S1, and the second data 35 may be second sensing data SD2 obtained by a second sensor S2. As described above, the type of each sensor (S1, S2) may be appropriately selected depending on the embodiment, such as the inference task.

[0054] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the first sensing data SD1 and a plurality of second training samples 350 of the second sensing data SD2. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, in a situation where different types of sensing data (SD1, SD2) are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0055] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the first sensing data SD1 and a second sample 355 of the second sensing data SD2. The inference device 2 may derive the result of the inference task from the acquired first sample 305 and second sample 355. According to one example of the present embodiment, in a situation where different types of sensing data (SD1, SD2) are used as the first data 30 and the second data 35, improvement in inference accuracy can be expected by ensembling the inference results of each model (55, 56, 57).

[0056] (3) Third Case 2C is a schematic diagram illustrating an example of the configuration of the first data 30 and the second data 35 in a third example of this embodiment. As shown in FIG. 2C, in one example, the first data 30 may be data AD1 other than sensing data obtained by a sensor, and the second data 35 may be data AD2 other than sensing data that is different from the data AD1. The type of each data (AD1, AD2) may be appropriately selected from data other than sensing data. The origin of each data (AD1, AD2) may be appropriately selected depending on the content of the inference, etc.

[0057] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the data AD1 and a plurality of second training samples 350 of the data AD2. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, in a situation where data (AD1, AD2) other than sensing data and different types of data (AD1, AD2) are used as the first data 30 and the second data 35, improvement in inference accuracy of the generated trained models (51, 52, 53) can be expected.

[0058] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the data AD1 and a second sample 355 of the data AD2. The inference device 2 may derive the result of the inference task from the acquired first sample 305 and second sample 355. According to one example of the present embodiment, in a situation where data (AD1, AD2) other than sensing data and different types of data are used as the first data 30 and the second data 35, an improvement in inference accuracy can be expected by ensembling the inference results of each model (55, 56, 57).

[0059] As described above, the sensing data (SD, SD1, SD2) and data other than the sensing data (AD, AD1, AD2) may be used as raw data (30, 35) or may be converted into analysis data and then used as the data (30, 35). The configurations of the first data 30 and the second data 35 are not limited to the above three examples and may be changed as appropriate depending on the embodiment.

[0060] (Relationship of information sources) The individuals from which each sample (305, 355) is obtained in the inference stage may or may not be included in the individuals from which each training sample (300, 350) is obtained in the learning stage. That is, the target individuals at the time of inference may or may not match the individuals from which the training samples are obtained in the learning stage. The relationship between the information sources (individuals from which samples are obtained) at the time of learning and inference is not particularly limited and may be determined appropriately depending on the embodiment. Furthermore, as long as the trained models (51, 52, 53) generated by the model generation device 1 can be applied to the inference stage, the type of target individuals at the time of inference does not necessarily match the type of individuals from which training samples are obtained in the learning stage. Applicability to the inference stage may mean that the inference accuracy does not deteriorate to the extent that the trained models (51, 52, 53) become unusable.

[0061] [Scene example] The system according to this embodiment may be applied to various situations where any inference task is performed. The application of this embodiment may include at least one of the first, second, and third cases described above. Specific application situations are exemplified below for each case.

[0062] (1) First Case The first example shown in FIG. 2A can be applied to various situations in which sensing data and data other than sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following five situations. However, the application of this embodiment is not limited to the following five examples and may be determined appropriately depending on the embodiment.

[0063] (1-1) Case 1-1 FIG. 3A schematically illustrates an example of Case 1-1 to which this embodiment is applied. Case 1-1 is an example of a situation in which this embodiment is applied to medical diagnosis. As shown in FIG. 3A, in one example, the first data 30 may be medical image data SD10 of a subject, and the second data 35 may be response data AD10 of a medical interview given to the subject. The inference task may include determining whether or not a disease has occurred in the subject.

[0064] The medical image data SD10 is an example of sensing data SD. As long as it can be used for image diagnosis, the type of medical image data SD10 is not particularly limited and may be appropriately selected depending on the embodiment. In one example, the medical image data SD10 may be an ultrasound (echo) image, an X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a PET (Positron Emission Tomography) image, a bone scintigraphy image, or The medical image data SD10 may be configured by a combination of these. One or more sensors S10 may be used to acquire the medical image data SD10. The type of the sensor S10 may be selected appropriately depending on the type of the medical image data SD10. The one or more sensors S10 may be selected from, for example, an ultrasound diagnostic device, an X-ray device, a CT device, an MRI device, a PET device, a scintigraphy device, etc.

[0065] The response data AD10 is an example of data AD other than the sensing data SD. As long as it can be used for diagnosis, the method of acquiring the response data AD10 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. In one example, the response data AD10 may be acquired by the subject inputting answers to a questionnaire. In another example, the response data The answer data AD10 may be obtained by input from a medical professional such as a doctor or nurse. In another example, the answer data AD10 may be obtained from medical information such as an electronic medical record. The content of the medical interview may be determined appropriately depending on the embodiment of the disease to be inferred, etc.

[0066] As long as it is possible to determine whether or not a disease has occurred, the format of the determination result is not particularly limited and may be determined appropriately depending on the embodiment. For example, determining whether or not a disease has occurred may be configured by regressing the probability of a specific disease occurring. Determining whether or not a disease has occurred may include identifying the type of disease that has occurred. Furthermore, determining whether or not a disease has occurred may be configured by regressing the probability of disease occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether or not only a specific disease has occurred, or may be configured generally to determine whether or not any disease has occurred. The type of disease to be inferred may be selected depending on the embodiment.

[0067] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of medical image data SD10 and a plurality of second training samples 350 of medical interview response data AD10 for a subject SS10. The subject SS10 is an example of a target person and an example of an individual during learning. The number of subjects SS10 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when diagnosing the presence or absence of a disease in a subject from the medical image data SD10 and the medical interview response data AD10, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0068] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of medical image data SD10 and a second sample 355 of interview response data AD10 for the subject TT10 to be diagnosed. The subject TT10 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (determining the presence or absence of a disease) from the acquired first sample 305 and second sample 355. The inference device 2 may infer the lesion site in the medical image data SD10 by referring to the progress of feature extraction for the medical image data SD10 (first sample 305) in the first model 55 and the fusion model 57. The progress of feature extraction may be referenced by any method. A known method may be adopted as the method for referencing the progress of feature extraction. The inference device 2 may output information indicating the result of determining the presence or absence of a disease as information regarding the result of the inference task to at least one of an arbitrary output device and a memory area. In one example, the information indicating the result of determining the presence or absence of a disease may be provided to at least one of the subject TT10 and a medical professional as information for medical assistance. According to one example of this embodiment, when diagnosing the presence or absence of a disease in a subject from medical image data SD10 and medical interview response data AD10, the inference results (disease presence / absence determination results) of each model (55, 56, 57) can be ensembled to expect improved inference accuracy.

[0069] The application of this embodiment to a medical diagnosis scenario need not be limited to the form shown in FIG. 3A and may be modified as appropriate depending on the embodiment. In another example, the medical image data SD10 may be replaced with other medical data. For example, the other medical data may include biological data such as blood pressure, pulse, heart rate, electrocardiogram, electromyogram, body temperature, electrodermal response, brain activity (electroencephalogram, magnetoencephalogram), activity level, blood glucose level, electrooculography, eye movement, and odor. The sensor used to acquire the other medical data may be selected as appropriate depending on the type of the other medical data. In yet another example, this embodiment may be applied to the scenario of Case 2 above by replacing the response data AD10 with the other medical data.

[0070] (1-2) Case 1-2 FIG. 3B schematically illustrates an example of Case 1-2 to which this embodiment is applied. Case 1-2 is an example of a situation in which this embodiment is applied to detecting an abnormality in an object in a factory. As shown in FIG. 3B, in one example, the first data 30 may be image data SD111 depicting the object or sound data SD112 related to the object. The second data 35 may be work history data AD11 related to the object. The inference task may include determining whether or not an abnormality has occurred in the object.

[0071] The image data SD111 or the sound data SD112 is an example of sensing data SD. As long as it can be used for abnormality detection, the type of each of the image data SD111 and the sound data SD112 is not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S11 may be used to acquire the image data SD111 or the sound data SD112. The type of the sensor S11 may be selected appropriately depending on the embodiment. The sensor S11 may be selected from, for example, an image sensor, a microphone, etc.

[0072] The work history data AD11 is an example of data AD other than the sensing data SD. As long as it can be used for anomaly detection, the method of acquiring the work history data AD11 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. In one example, the work history data AD11 may include the work history of workers in a factory, the operation history of equipment, etc. The work history data may be composed of text data, numerical data, etc. At least a portion of the work history data AD11 may be acquired by input by a worker, or may be acquired automatically from the operation results of equipment, etc.

[0073] As long as it is possible to determine whether an abnormality has occurred, the format of the determination result is not particularly limited and may be determined appropriately depending on the embodiment. In one example, determining whether an abnormality has occurred may be configured by at least one of identifying whether an abnormality has occurred or regressing the probability that a specific abnormality has occurred. Determining whether an abnormality has occurred may include identifying the type of abnormality that has occurred. Furthermore, determining whether an abnormality has occurred may be configured by regressing the probability of abnormality occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured generally to determine whether any abnormality has occurred.

[0074] The object may be selected appropriately depending on the embodiment. In one example, the object may include at least one of a device constituting FA (Factory Automation) and a product produced by the FA. The product produced by the FA may include at least one of an intermediate product and a final product. The abnormality may include at least one of a device failure and a product defect.

[0075] When a device constituting an FA is used as the target, the image data SD111 may be acquired to show a scene of the device operating, etc. The sound data SD112 may be acquired to include the operating sounds, vibration sounds, etc. of the device. When a product produced by the FA is used as the target, the image data SD111 may be acquired to show the target range of the product (the range in which anomalies are detected). The sound data SD112 may be acquired to include the sounds of the product itself, the operating sounds of devices involved in the production of the product, the vibration sounds of the devices, etc.

[0076] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD111 or the sound data SD112 and a plurality of second training samples 350 of the work history data AD11 for the object SS11. The object SS11 is an example of an individual during learning. The number of objects SS11 from which each training sample (300, 350) is obtained may be determined appropriately depending on the embodiment. The model generation device 1 processes the acquired plurality of first training samples 300 and plurality of second training samples 350 into a first model 51, a second model 52, and a fusion model. The trained models (51, 52, 53) may be used for machine learning of the trained models 51, 52, 53. According to one example of this embodiment, when detecting the presence or absence of an abnormality in an object from the image data SD111 or the sound data SD112 and the work history data AD11, it is expected that the inference accuracy of the trained models (51, 52, 53) generated will be improved.

[0077] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD111 or the sound data SD112 and a second sample 355 of the work history data AD11 for the object TT11 for which an abnormality is detected. The object TT11 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (determining whether or not an abnormality has occurred) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of determining whether or not an abnormality has occurred to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of determining whether or not an abnormality has occurred may be provided to at least one of a worker and a work supervisor who are present near the object TT11. The output device may include a terminal of the worker or the supervisor. According to one example of this embodiment, when detecting whether or not an object has an abnormality from image data SD111 or sound data SD112 and work history data AD11, the inference results (determination results of whether or not an abnormality exists) of each model (55, 56, 57) are ensembled, and improvement in inference accuracy can be expected.

[0078] Note that the method of applying this embodiment to a scene of anomaly detection in a factory is not limited to the form of FIG. 3B and may be modified as appropriate depending on the embodiment. In another example, both the image data SD111 and the sound data SD112 may be used as the first data 30. In another example, this embodiment may be applied to the scene of Case 2 above by using one of the image data SD111 and the sound data SD112 as the first data 30 and the other as the second data 35. In this example, the work history data AD11 may be omitted.

[0079] (1-3) Case 1-3 FIG. 3C schematically illustrates an example of Case 1-3 to which this embodiment is applied. Case 1-3 is an example of a scenario in which this embodiment is applied to detecting an abnormality on a road. As shown in FIG. 3C, in one example, the first data 30 may be image data SD12 showing a road, and the second data 35 may be text data AD12 related to the road condition. The inference task may include determining whether an abnormality has occurred on the road.

[0080] The image data SD12 is an example of sensing data SD. As long as it can be used to detect anomalies, the type and acquisition method of the image data SD12 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S12 may be used to acquire the image data SD12. The sensor S12 may be, for example, an image sensor. The method by which the sensor S12 observes the road is not particularly limited and may be selected appropriately depending on the embodiment. The sensor S12 may be placed at a fixed point or on any moving body (vehicle, drone, satellite, etc.).

[0081] The text data AD12 is an example of data AD other than the sensing data SD. As long as it can be used for anomaly detection, the method of acquiring the text data AD12 is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the text data AD12 may include at least one of information provided by a specialized institution such as a road traffic center and information provided by ordinary people such as SNS (Social Networking Service). As long as the text data AD12 contains information about the situation, the content of the text data AD12 is not particularly limited and may be determined appropriately depending on the embodiment.

[0082] As long as it is possible to determine whether or not an abnormality has occurred, the format of the determination result is not particularly limited. The method may be appropriately determined depending on the embodiment. In one example, determining whether an abnormality has occurred may include at least one of identifying whether an abnormality has occurred on the road and regressing the probability that a specific abnormality has occurred. Determining whether an abnormality has occurred may include identifying the type of abnormality that has occurred. Determining whether an abnormality has occurred may also include regressing the probability of the abnormality occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured generally to determine whether any abnormality has occurred. Road abnormalities may include, for example, road damage, fallen objects, dirt on the road surface, accidents, etc. Road damage may include, for example, holes, depressions, collapsed shoulders, etc. If the road abnormality includes a fallen object, the type of the abnormality may include the type of the fallen object. Determining the type of the abnormality may include determining the type of the fallen object.

[0083] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD12 and a plurality of second training samples 350 of the text data AD12 for a target range SS12 of a road. The target range SS12 of the road is an example of an individual during learning. The size and number of the target range SS12 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, in a scene where the presence or absence of an abnormality on a road is detected from the image data SD12 and the text data AD12, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0084] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD12 and a second sample 355 of the text data AD12 for a target range TT12 of a road where an abnormality is to be detected. The target range TT12 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (determining whether or not an abnormality has occurred) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of determining whether or not an abnormality has occurred on the road to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, the information indicating the result of determining whether or not an abnormality has occurred may be provided to an arbitrary user, such as an administrator who manages the target range TT12 of the road. The output device may include a user terminal. According to one example of the present embodiment, when detecting the presence or absence of an abnormality on a road from the image data SD12 and the text data AD12, improvement in inference accuracy can be expected by ensembling the inference results (determination results of the presence or absence of an abnormality) of each model (55, 56, 57).

[0085] (1-4) Case 1-4 FIG. 3D schematically illustrates an example of Case 1-4 to which this embodiment is applied. Case 1-4 is an example of a scene in which this embodiment is applied to inferring a subject's emotions. As shown in FIG. 3D, in one example, the first data 30 may be image data SD131 depicting the subject or sound data SD132 related to the subject. The second data 35 may be text data AD13 related to the subject. The inference task may include inferring the subject's emotions.

[0086] The image data SD131 or the sound data SD132 is an example of sensing data SD. As long as it can be used for emotion inference, the type and acquisition method of the image data SD131 and the sound data SD132 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S13 may be used to acquire the image data SD131 or the sound data SD132. The type of the sensor S13 may be selected appropriately depending on the embodiment. The sensor S13 may be selected from, for example, an image sensor, a microphone, etc. In one example, the image data SD131 may be acquired so as to capture at least a portion of the subject's face. In another example, the sound data SD132 may include the subject's voice, environmental sounds of the subject, etc.

[0087] The text data AD13 is an example of data AD other than the sensing data SD. As long as it can be used for emotion inference, the method of acquiring the text data AD13 is not particularly limited and may be appropriately selected depending on the embodiment. The text data AD13 may be acquired by manual input by the subject, or may be acquired automatically by a method such as voice analysis. In one example, the text data AD13 may include comments by the subject. The comments may include symbolic expressions such as emoticons and stamps.

[0088] The format of the result of inferring the subject's emotion may be determined appropriately depending on the embodiment. In one example, inferring the emotion may include at least one of identifying the type of emotion occurring, regressing the probability of a specific emotion occurring, and regressing the probability of emotion occurrence for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to infer only a specific type of emotion, or may be configured generally to infer multiple types of emotions. The type of emotion may be set appropriately, such as joy, anger, sadness, fun, etc.

[0089] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD131 or sound data SD132 and a plurality of second training samples 350 of text data AD13 for the subject SS13. The subject SS13 is an example of a target person and an example of an individual during learning. The number of subjects SS13 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the subject's emotions from the image data SD131 or sound data SD132 and the text data AD13, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0090] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD131 or sound data SD132 and a second sample 355 of text data AD13 for the subject TT13, whose emotions are to be inferred. The subject TT13 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring emotions) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of the emotion inference to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the first to fourth examples may be used in remote communication situations, such as video conferencing and communication in a virtual space. In this situation, information regarding the result of the emotion inference may be provided to at least one of the subject TT13 himself or the remote communication partner. Furthermore, if an avatar is used in remote communication, the result of the emotion inference may be reflected in the avatar. As a result, the inference device 2 may at least partially automatically control the avatar used in remote communication. The automatic control may be configured by at least one of direct control and indirect control by giving instructions to another computer. According to one example of this embodiment, in a situation where a subject's emotion is inferred from image data SD131 or sound data SD132 and text data AD13, by ensembling the inference results (emotion inference results) of each model (55, 56, 57), it is possible to expect improvement in inference accuracy.

[0091] In the above remote communication scenario, at least one of the computers used in the remote communication may operate as the inference device 2. For example, when the remote communication is performed by direct communication between user terminals, at least one of the user terminals of the users participating in the remote communication (their own user terminal or the user terminal of the communication partner) may operate as the inference device 2. When remote communication is performed via the internet, at least one of the user terminals and the server device may operate as the inference device 2.

[0092] (1-5) Case 1-5 FIG. 3E schematically illustrates an example of Case 1-5 to which this embodiment is applied. Case 1-5 is an example of a situation in which this embodiment is applied to inferring the value of an object. As shown in FIG. 3E, in one example, the first data 30 may be image data SD14 depicting the object, and the second data 35 may be attribute data AD14 of the object. The inference task may include predicting the value of the object.

[0093] The image data SD14 is an example of sensing data SD. As long as it can be used for value prediction, the type and acquisition method of the image data SD14 are not particularly limited and may be selected appropriately depending on the embodiment. One or more sensors S14 may be used to acquire the image data SD14. The sensor S14 may be, for example, an image sensor. The image data SD14 may be acquired appropriately so as to capture at least a portion of the object to an extent that value prediction is possible.

[0094] The attribute data AD14 is an example of data AD other than the sensing data SD. As long as it can be used for value prediction, the method of acquiring the attribute data AD14 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. At least a portion of the attribute data AD14 may be generated by manual input by an operator or may be automatically generated by computer processing. The attribute information of the object included in the attribute data AD14 may be appropriately selected depending on the value of the object to be predicted, etc. The attribute data AD14 may include, for example, attribute information such as the price, type, usage history, years of use, and dimensions of the object.

[0095] The format of the value prediction result may be selected appropriately depending on the embodiment. In one example, predicting value may include at least one of identifying a rank of the predicted value and regressing the predicted value. When predicting multiple types of value, the identification and regression may be performed for each type. The type of value to be predicted may be selected appropriately depending on the embodiment. In one example, the predicted value may include values ​​related to the amount of sales, used sales price, insurance premiums, etc. For example, predicting value may include predicting sales. In this case, the attribute data may include the sales price of the object, etc. Also, for example, predicting value may include predicting the used sales price. In this case, the attribute data may include the initial sales price, usage history, etc. of the object. Also, for example, predicting value may include predicting insurance premiums. In this case, the attribute data may include the sales price, usage history, etc. of the object. The object whose value is to be predicted may be selected arbitrarily. The object may include, for example, a vehicle, real estate, etc. If the object is a vehicle, the usage history may include the vehicle's driving history, accident history, etc. If the object is real estate, the usage history may include the real estate's occupancy history, breakdown history, repair history, and the like.

[0096] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD14 and a plurality of second training samples 350 of attribute data AD14 for the object SS14. The object SS14 is an example of an individual during learning. The number and type of the object SS14 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when predicting the value of an object from the image data SD14 and the attribute data AD14, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0097] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD14 and a second sample 355 of attribute data AD14 for the object TT14 whose value is to be predicted. The object TT14 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (predicting value) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the value prediction result as information regarding the result of the inference task to at least one of an arbitrary output device and a storage area. In one example, the information indicating the value prediction result may be provided to an arbitrary user, such as the owner, manager, or related party (seller, etc.) of the object TT14. The output device may include a user terminal. According to one example of the present embodiment, when predicting the value of an object from the image data SD14 and the attribute data AD14, improvement in inference accuracy can be expected by ensembling the inference results (value prediction results) of each model (55, 56, 57).

[0098] (2) Second Case The second example shown in FIG. 2B can be applied to various situations in which two or more types of sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following ten situations. However, the application of this embodiment is not limited to the following ten examples and may be determined appropriately depending on the embodiment.

[0099] (2-1) Case 2-1 FIG. 4A schematically illustrates an example of Case 2-1 to which this embodiment is applied. Case 2-1 is an example of a scenario in which this embodiment is applied to inferring the state of an object in a chemical plant. As shown in FIG. 4A, in one example, the first data 30 may be image data SD20 depicting an object in the chemical plant, and the second data 35 may be numerical data SD21 related to the environment of the object in the chemical plant. The inference task may include inferring the state of the object.

[0100] The image data SD20 is an example of first sensing data SD1. The type and acquisition method of the image data SD20 are not particularly limited as long as they can be used for state inference, and may be selected appropriately depending on the embodiment. One or more first sensors S20 may be used to acquire the image data SD20. The first sensor S20 may be, for example, an image sensor. The first sensor S20 may be appropriately positioned to capture an image of an object from inside or outside the chemical plant.

[0101] The numerical data SD21 is an example of second sensing data SD2. As long as it can be used for state inference, the type and format of the numerical data SD21 are not particularly limited and may be appropriately selected depending on the embodiment. The numerical data SD21 may be composed of information at a single point in time, or may be composed of time-series data including information at multiple points in time. Furthermore, the environment to be observed as the numerical data SD21 may be appropriately selected depending on the state to be inferred, etc. The numerical data SD21 may include information such as pressure, temperature, and humidity. One or more second sensors S21 may be used to acquire the numerical data SD21. The second sensor S21 may be appropriately selected depending on the environment to be observed. The second sensor S21 may be selected from, for example, a pressure gauge, a thermometer, a hygrometer, etc.

[0102] The format of the result of inferring the state may be selected appropriately depending on the embodiment. In one example, inferring the state may include at least one of identifying the category of the state to be inferred and regressing the degree of the state. When inferring multiple types of states, identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, the chemical plant may include a fermentation plant, a gas absorption plant, etc. For example, if the chemical plant is a fermentation plant, the image data SD20 may be acquired so as to capture bubbles occurring in the object (fermentation object). The numerical data SD21 may include information such as pressure. Inferring the state may include predicting a yield. The yield is an index indicating the degree to which a particular product or component is efficiently produced in a chemical process or chemical reaction. Note that the subject of inference is not limited to these examples and may be changed as appropriate depending on the embodiment.

[0103] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD20 and a plurality of second training samples 350 of numerical data SD21 for objects in the chemical plant SS20. The objects in the chemical plant SS20 are an example of individuals during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state of objects in the chemical plant from the image data SD20 and the numerical data SD21, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0104] Furthermore, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD20 and a second sample 355 of numerical data SD21 for an object in the chemical plant TT20. The object in the chemical plant TT20 is an example of a target individual during inference. The chemical plant TT20 to be inferred may be the same as or different from the chemical plant SS20 during learning. The inference device 2 may derive a result of the inference task (inferring the state of the object) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the object to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the object may be provided to an arbitrary user, such as a manager of the chemical plant TT20. The output device may include a user terminal. According to one example of this embodiment, when inferring the state of an object in a chemical plant from image data SD20 and numerical data SD21, the inference results (state inference results) of each model (55, 56, 57) are ensembled, which is expected to improve the inference accuracy.

[0105] (2-2) Case 2-2 FIG. 4B schematically illustrates an example of Case 2-2 to which this embodiment is applied. Case 2-2 is an example of a scenario in which this embodiment is applied to inferring conditions inside a pipe. As shown in FIG. 4B , in one example, the first data 30 may be ultrasonic data SD22 related to the flow velocity inside the pipe, and the second data 35 may be sound data SD23 related to the pipe. The inference task may include inferring conditions inside the pipe.

[0106] The ultrasonic data SD22 is an example of the first sensing data SD1. As long as the ultrasonic data SD22 can be used for state inference, the method of acquiring the ultrasonic data SD22 is not particularly limited and may be appropriately selected depending on the embodiment. One or more first sensors S22 may be used to acquire the ultrasonic data SD22. The first sensor S22 may be, for example, an ultrasonic sensor. In one example, the first sensor S22 may be configured to be attached to a pipe with a clamp and to observe the flow speed of an object flowing through the pipe.

[0107] The sound data SD23 is an example of second sensing data SD2. As long as the sound data SD23 can be used for state inference, the method of acquiring the sound data SD23 is not particularly limited and may be selected appropriately depending on the embodiment. One or more second sensors S23 may be used to acquire the sound data SD23. The second sensor S23 may be, for example, a microphone. In one example, the second sensor S23 may be configured to be placed on or near a pipe to observe sounds inside the pipe.

[0108] The format of the result of inferring the state in the pipe may be selected appropriately depending on the embodiment. In an example, inferring a state may include at least one of identifying a category of the state to be inferred and regressing the degree of the state. When multiple types of states are inferred, the identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, inferring a state in a pipe may include determining whether the pipe is clogged or inferring the degree of cloggedness in the pipe. This may monitor the state of an object flowing through the pipe. By controlling the flow rate based on the monitoring results, it is possible to prevent the pipe from becoming clogged. For example, when applying this embodiment to a papermaking process, it may monitor the state of pulp fibers flowing through the pipe and control the flow rate of the pulp fibers based on the monitoring results. This may prevent the pipe from becoming clogged with pulp fibers.

[0109] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the ultrasound data SD22 and a plurality of second training samples 350 of the sound data SD23 for the pipe SS22. The pipe SS22 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state inside the pipe from the ultrasound data SD22 and the sound data SD23, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0110] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of the ultrasound data SD22 and a second sample 355 of the sound data SD23 for the pipe TT22. The pipe TT22 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (inferring the state within the pipe TT22) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state within the pipe TT22 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, information regarding the result of the inference task may be provided to an arbitrary user, such as an administrator of the pipe TT22. The output device may include a user terminal. Furthermore, when the inference device 2 is connected to an adjustment device that adjusts the flow rate within the pipe TT22, the inference device 2 may send a command to the adjustment device instructing the flow rate according to the inference result, as an example of a process of outputting information regarding the result of the inference task. For example, if it is inferred that the level of clogging within the pipe TT22 is high, the inference device 2 may issue a command to the adjustment device to reduce the flow rate within the pipe TT22. On the other hand, if it is estimated that the degree of blockage in the pipe TT22 is low, the inference device 2 may issue a command to the adjustment device to maintain or increase the flow rate in the pipe TT22. Whether the degree of blockage is high or low may be determined by any method, such as threshold comparison. If the flow rate is to be maintained, the inference device 2 may omit sending a command. According to one example of this embodiment, when inferring the state inside the pipe from the ultrasonic data SD22 and the sound data SD23, improvement in inference accuracy can be expected by ensembling the inference results (state inference results) of each model (55, 56, 57).

[0111] (2-3) Case 2-3 FIG. 4C schematically illustrates an example of Case 2-3 to which this embodiment is applied. Case 2-3 is an example of a situation in which this embodiment is applied to inferring the position of an object when driving a robotic device. As shown in FIG. 4C , in one example, the first data 30 may be image data SD24 containing the object, and the second data 35 may be point cloud data SD25 obtained by measuring the object. The inference task may include inferring the position of the object.

[0112] The image data SD24 is an example of the first sensing data SD1. As long as it can be used for position inference, the type and acquisition method of the image data SD24 are not particularly limited and may be appropriately selected depending on the embodiment. To acquire the image data SD24, one or more first sensors S24 may be used. The first sensor S24 may be, for example, an image sensor. The image sensor constituting the first sensor S24 may be appropriately selected so as not to completely overlap with the second sensor S25. The first sensor S24 may be attached to the robot device or may be disposed external to the robot device. In one example, the first sensor S24 may be appropriately disposed so that the imaging range includes at least a part of the working range of the robot device. The working range of the robot device may be appropriately defined according to, for example, the reachable range of the end effector, the movable range of the robot device, etc.

[0113] The point cloud data SD25 is an example of second sensing data SD2. As long as it can be used for position inference, the type and acquisition method of the point cloud data SD25 are not particularly limited and may be selected appropriately depending on the embodiment. To acquire the point cloud data SD25, one or more second sensors S25 may be used. The second sensor S25 may be selected appropriately from, for example, an ultrasonic sensor, a depth sensor, an infrared sensor, a radar, a LiDAR, etc. The second sensor S25 is a first sensor. Like the second sensor S24, the second sensor S25 may be attached to the robotic device or may be disposed outside the robotic device. In one example, the second sensor S25 may be appropriately disposed so that the measurement range includes at least a portion of the imaging range of the first sensor S24. In this way, the point cloud data SD25 may be obtained by measuring at least a portion of the imaging range of the image data SD24.

[0114] The format of the location inference result may be selected appropriately depending on the embodiment. In one example, inferring location may include at least one of identifying presence or absence in the target area, regressing the probability of presence in the target area, regressing the range of presence, and regressing the location of presence (two-dimensional or three-dimensional location information). Inferring location may also include inferring the shape and / or dimensions of the object (including the amount of deformation of the object). Inferring location may also include other inference tasks, such as identifying the type of object.

[0115] The type of robot device is not particularly limited and may be selected appropriately depending on the embodiment. The robot device may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile robot configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, or an orthogonal robot. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a nursing care robot, a security robot, or a transport robot. The content of the autonomous processing may be selected appropriately depending on the embodiment. The mobile robot may be, for example, a cleaning robot, the above-mentioned autonomous robot (including a mobile robot) configured to move, a vehicle configured to be able to drive automatically, an air vehicle capable of flying automatically (such as a drone), or a ship capable of navigating automatically (such as a ship or submarine).

[0116] Furthermore, the object whose position is to be inferred need not be particularly limited as long as it can be involved in the operation of the robot device, and may be selected appropriately depending on the embodiment. The object may include, for example, a work target of the robot device, an obstacle, etc. The work target may be, for example, an object to be grasped and carried by the robot device, etc. The obstacle may include an object and a person. The result of the position inference may be used appropriately for controlling the robot device. The object may be, for example, a part of an individual, a subordinate part of an individual such as a mark, or an independent individual.

[0117] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the image data SD24 and a plurality of second training samples 350 of the point cloud data SD25 for the target SS24 that may be involved in the operation of the robot device RS24. The target SS24 is an example of an individual during learning. The type and number of the target SS24 may be determined appropriately depending on the embodiment. The model generation device 1 uses the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of this embodiment, when the position of an object is inferred from the image data SD24 and the point cloud data SD25, it is possible to expect an improvement in the inference accuracy of the generated trained models (51, 52, 53). Note that the robot device RS24 may be omitted in the learning stage.

[0118] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of the image data SD24 and a second sample 355 of the point cloud data SD25 for the object TT24. The object TT24 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring the position of the object TT24) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the position of the object TT24 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. Furthermore, when connected to the robot device RT24, the inference device 2 may send a command to the robot device RT24 instructing an operation according to the inference result as an example of a process of outputting information regarding the result of the inference task. For example, if the robot device RT24 is an industrial robot and the object TT24 is a work object to be grasped, the inference device 2 may issue a command to the robot device RT24 to move an end effector to the inferred position and grasp the object TT24 located at the inferred position with the end effector. When inferring the position includes inferring the shape, the inference device 2 may determine the force with which to grip the object TT24 according to the inferred shape of the object TT24 (e.g., the amount of deformation) and issue a command to the robot device RT24 to grip the object TT24 with the determined force. This allows the inference device 2 to adjust the gripping force of the robot device RT24. For example, if the robot device RT24 is an autonomous robot or a mobile robot and the object TT24 is the objective of the task, the inference device 2 may issue a command to the robot device RT24 to move near the object TT24 or to face the object TT24. For example, if the robot device RT24 is an industrial robot, an autonomous robot, or a mobile robot and the object TT24 is an obstacle, the inference device 2 may issue a command to the robot device RT24 to avoid the object TT24 (e.g., limit the driving range, move along a detour, etc.). The inference device 2 may directly control the operation of the robot device RT24 or indirectly control it by issuing a command to a controller of the robot device RT24. The robot device RT24 during inference may be the same as or different from the robot device RS24 during learning.According to one example of this embodiment, when inferring the position of an object from image data SD24 and point cloud data SD25, the inference results (position inference results) of each model (55, 56, 57) are ensembled, which is expected to improve the inference accuracy.

[0119] (2-4) Case 2-4 FIG. 4D schematically illustrates an example of Case 2-4 to which this embodiment is applied. Case 2-4 is an example of a situation in which this embodiment is applied to inferring the state of equipment. As shown in FIG. 4D, in one example, the first data 30 may be image data SD26 depicting the equipment, and the second data 35 may be sound data SD27 related to the equipment. The inference task may include inferring the state of the equipment.

[0120] The image data SD26 is an example of first sensing data SD1. The type and acquisition method of the image data SD26 are not particularly limited as long as they can be used to infer the state of the equipment, and may be selected appropriately depending on the embodiment. One or more first sensors S26 may be used to acquire the image data SD26. The first sensor S26 may be, for example, an image sensor. The first sensor S26 may be appropriately positioned so that the imaging range includes at least a part of the range in which the state of the equipment is inferred.

[0121] The sound data SD27 is an example of the second sensing data SD2. As long as the sound data SD27 can be used to infer the state of the equipment, the method of acquiring the sound data SD27 is not particularly limited. The number of second sensors S27 may be appropriately selected depending on the situation. One or more second sensors S27 may be used to acquire the sound data SD27. The second sensors S27 may be, for example, microphones. In one example, the second sensors S27 may be appropriately arranged to observe sounds (for example, vibration sounds, abnormal sounds, etc.) emitted from the equipment.

[0122] The format of the result of inferring the state of the equipment may be selected appropriately depending on the embodiment. In one example, inferring the state may include at least one of identifying a category of the state to be inferred and regressing the degree of the state. When multiple types of states are inferred, the identification and regression may be performed for each type. The type of state to be inferred may be selected appropriately depending on the embodiment. In one example, inferring the state of the equipment may include identifying a deterioration level of the equipment or regressing the deterioration degree of the equipment. This may allow the state of the equipment to be monitored. The type of equipment whose state is to be inferred is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the equipment may include structures such as buildings, tunnels, bridges, and water pipes.

[0123] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD26 and a plurality of second training samples 350 of sound data SD27 for the equipment SS26. The equipment SS26 is an example of an individual during learning. The type and number of equipment SS26 from which each training sample (300, 350) is acquired may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the state of the equipment from the image data SD26 and the sound data SD27, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0124] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD26 and a second sample 355 of sound data SD27 for the equipment TT26, the state of which is to be inferred. The equipment TT26 is an example of a target individual during inference. The equipment TT26 during inference may be the same as or different from the equipment SS26 during learning. The inference device 2 may derive the result of the inference task (inferring the state of the equipment TT26) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the equipment TT26 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the equipment TT26 may be provided to an arbitrary user, such as an administrator who manages the equipment TT26. The output device may include a user's terminal. According to one example of this embodiment, when inferring the state of equipment from image data SD26 and sound data SD27, the inference results (state inference results) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of inference.

[0125] (2-5) Case 2-5 FIG. 4E schematically illustrates an example of Case 2-5 to which this embodiment is applied. Case 2-5 is an example of a scenario in which this embodiment is applied to inferring water quality in a facility. As shown in FIG. 4E, in one example, the first data 30 may be image data SD28 of water present in the facility, and the second data 35 may be observation data SD29 of the odor of water present in the facility. The inference task may include inferring water quality in the facility.

[0126] The image data SD28 is an example of the first sensing data SD1. The type and acquisition method of the image data SD28 are not particularly limited as long as they can be used to infer water quality, and may be selected appropriately depending on the embodiment. One or more first sensors S28 may be used to acquire the image data SD28. The first sensor S28 may be, for example, an image sensor. The first sensor S28 may be appropriately positioned so that its imaging range includes at least a part of the range in which the quality of water present in the facility is inferred.

[0127] The observation data SD29 is an example of second sensing data SD2. As long as it can be used to infer water quality, the method of acquiring the observation data SD29 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. One or more second sensors S29 may be used to acquire the observation data SD29. The second sensor S29 may be, for example, an odor sensor. In one example, the second sensor S29 may be appropriately placed within a range where the odor of water present in the facility can be observed.

[0128] The form of the result of inferring water quality may be selected appropriately depending on the embodiment. In one example, inferring water quality may include at least one of identifying a quality level and regressing the degree of quality. Water quality may be defined appropriately according to any standard, including known standards. The facility from which water quality is inferred may be selected appropriately depending on the embodiment. In one example, the facility may include water-related equipment such as a water purification plant or a sewage treatment plant. The facility may also include a natural or artificially created body of water such as a pond or a lake.

[0129] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD28 and a plurality of second training samples 350 of observation data SD29 for the facility SS28. Water present in the facility SS28 is an example of an individual during learning. The type and number of facilities SS28 from which the training samples (300, 350) are acquired may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, when inferring water quality at a facility from the image data SD28 and the observation data SD29, improved inference accuracy can be expected in the generated trained models (51, 52, 53).

[0130] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD28 and a second sample 355 of observation data SD29 for the facility TT28. The water present in the facility TT28 is an example of a target individual during inference. The facility TT28 during inference may be the same as or different from the facility SS28 during learning. The inference device 2 may derive a result of the inference task (inferring the water quality at the facility TT28) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the water quality at the facility TT28 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the water quality at the facility TT28 may be provided to an arbitrary user, such as an administrator who manages the facility TT28. The output device may include a user's terminal. If the facility TT28 is a facility such as a water purification plant or a sewage treatment plant, the inference device 2 may generate an operation plan for the facility according to the inference result as information regarding the result of the inference task. For example, if the facility TT28 is a sewage treatment plant and a deterioration in water quality is inferred, the inference device 2 may generate an operation plan to increase the availability of the sewage treatment plant. If a maintenance of water quality is inferred, the inference device 2 may generate an operation plan to maintain the availability of the sewage treatment plant. The inference device 2 may output instructions to a user or a controller of the equipment to operate the equipment in accordance with the generated operation plan. According to one example of the present embodiment, when inferring water quality at a facility from image data SD28 and observation data SD29, improving the accuracy of inference can be expected by ensembling the inference results (water quality inference results) of each model (55, 56, 57).

[0131] (2-6) Case 2-6 FIG. 4F is a schematic diagram showing an example of Case 2-6 to which this embodiment is applied. Case 2-6 is an example of a scene in which this embodiment is applied to inferring the state of oil in a fryer. As shown, in one example, the first data 30 may be image data SD30 of oil in a fryer, and the second data 35 may be observation data SD31 of the smell of the oil in the fryer. The inference task may include inferring the condition of the oil in the fryer.

[0132] The image data SD30 is an example of the first sensing data SD1. The type and acquisition method of the image data SD30 are not particularly limited as long as they can be used to infer the state of the oil, and may be selected appropriately depending on the embodiment. One or more first sensors S30 may be used to acquire the image data SD30. The first sensor S30 may be, for example, an image sensor. The first sensor S30 may be appropriately positioned so that the imaging range includes at least a portion of the oil present in the fryer.

[0133] The observation data SD31 is an example of second sensing data SD2. As long as it can be used to infer the state of oil, the method of acquiring the observation data SD31 and the data format thereof are not particularly limited and may be appropriately selected depending on the embodiment. One or more second sensors S31 may be used to acquire the observation data SD31. The second sensor S31 may be, for example, an odor sensor. In one example, during observation, the second sensor S31 may be appropriately positioned within a range where the odor of oil present in the fryer can be observed.

[0134] The format of the result of inferring the oil condition may be selected appropriately depending on the embodiment. In one example, inferring the oil condition may include at least one of identifying the level of oil quality and regressing the oil quality. The oil quality may be defined appropriately according to any standard, including known standards. Inferring the oil condition may also include predicting the time to change the oil. In one example, inferring the oil condition may include determining whether the time to change the oil has arrived or regressing the time to change the oil. This allows the quality of the oil to be monitored. The present Example 2-6 may be implemented in stores that sell oil, such as convenience stores and restaurants.

[0135] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD30 and a plurality of second training samples 350 of observation data SD31 for oil SS30 in a fryer. The oil SS30 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of the present embodiment, when inferring the state of oil in a fryer from the image data SD30 and the observation data SD31, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0136] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD30 and a second sample 355 of observation data SD31 for the oil TT30 in the fryer. The oil TT30 is an example of a target individual during inference. The fryer during inference may be the same as or different from the fryer during learning. The inference device 2 may derive the result of the inference task (inferring the state of the oil TT30) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the oil TT30 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the oil TT30 may be provided to an arbitrary user, such as an administrator who manages the oil TT30. The output device may include a user terminal. When the arrival of the replacement time is inferred, the information indicating the result of inferring the state of the oil TT30 may include instruction information for instructing the replacement of the oil TT30. According to one example of this embodiment, when inferring the state of oil in a fryer from image data SD30 and observation data SD31, the inference results (inference results of the oil state) of each model (55, 56, 57) can be ensembled, thereby improving the accuracy of inference.

[0137] The state of the oil can also be observed using other sensing data, such as sound data containing sounds made during use. Therefore, at least one of the image data SD30 and the observation data SD31 may be replaced with other sensing data. For example, the observation data SD31 may be replaced with sound data recorded while the oil is being used. In this case, the second sensor may be, for example, a microphone.

[0138] (2-7) Case 2-7 FIG. 4G schematically illustrates an example of Case 2-7 to which this embodiment is applied. Case 2-7 is an example of a scenario in which this embodiment is applied to atmospheric condition inference. As shown in FIG. 4G, in one example, the first data 30 may be image data SD32 depicting the atmosphere, and the second data 35 may be atmospheric meteorological data SD33. The inference task may include inferring atmospheric conditions.

[0139] The image data SD32 is an example of the first sensing data SD1. As long as the image data SD32 reflects the atmospheric condition and can be used to infer the atmospheric condition, the type and acquisition method of the image data SD32 are not particularly limited and may be selected appropriately depending on the embodiment. One or more first sensors S32 may be used to acquire the image data SD32. The first sensor S32 may be, for example, an image sensor. The image data SD32 may be, for example, satellite image data, infrared image data, etc.

[0140] The weather data SD33 is an example of the second sensing data SD2. As long as it can be used to infer atmospheric conditions, the type and acquisition method of the weather data SD33 are not particularly limited and may be selected appropriately depending on the embodiment. One or more second sensors S33 may be used to acquire the weather data SD33. The second sensor S33 may be, for example, a weather sensor. The weather data SD33 may include, for example, weather information such as temperature, humidity, atmospheric pressure, rainfall, wind speed, wind direction, and illuminance. The type of weather information included in the weather data SD33 may be selected appropriately depending on the embodiment.

[0141] The form of the result of inferring the atmospheric condition may be selected as appropriate depending on the embodiment. In one example, inferring the atmospheric condition may include identifying a level of a predetermined indicator of the atmospheric condition and / or regressing the value of the predetermined indicator. The indicator of the atmospheric condition may be defined according to any criteria, including known criteria, as appropriate. Inferring the atmospheric condition may also include identifying or regressing a pollution level of the atmospheric condition, thereby enabling monitoring of the level of air pollution.

[0142] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD32 and a plurality of second training samples 350 of meteorological data SD33 for the atmospheric corresponding range SS32. The atmospheric corresponding range SS32 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring atmospheric conditions from the image data SD32 and the meteorological data SD33, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0143] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD32 and a second sample 355 of meteorological data SD33 for the atmospheric target range TT32. The atmospheric target range TT32 is an example of a target individual during inference. The inference device 2 may derive the result of the inference task (inferring the state of the target range TT32) from the acquired first sample 305 and second sample 355. The inference device 2 may then use information related to the result of the inference task. As the information, information indicating the result of inferring the state of the atmospheric target range TT32 may be output to at least one of an arbitrary output device and a storage area. In one example, the information indicating the result of inferring the state of the atmospheric target range TT32 may be provided to an arbitrary user via a server device that provides meteorological information. In this case, the inference device 2 may provide information indicating the inference result to the server device. Alternatively, the server device may operate as the inference device 2. According to one example of the present embodiment, when inferring the atmospheric state from the image data SD32 and the meteorological data SD33, improvement in inference accuracy can be expected by ensembling the inference results (inference results of the atmospheric state) of each model (55, 56, 57).

[0144] (2-8) Case 2-8 FIG. 4H schematically illustrates an example of Case 2-8 to which this embodiment is applied. Case 2-8 is an example of a scenario in which this embodiment is applied to inferring the condition of a crop. As shown in FIG. 4H, in one example, the first data 30 may be image data SD34 related to the crop, and the second data 35 may be weather data SD35 in the environment of the crop. The inference task may include inferring the condition of the crop.

[0145] The image data SD34 is an example of the first sensing data SD1. As long as the image data SD34 can be used to infer the condition of the crop, the type and acquisition method of the image data SD34 are not particularly limited and may be selected appropriately depending on the embodiment. One or more first sensors S34 may be used to acquire the image data SD34. The first sensor S34 may be, for example, an image sensor. The image data SD34 may include general image data as well as satellite image data. The imaging range is not particularly limited and may be determined appropriately as long as the crop is captured to an extent that the condition of the crop can be inferred. The image data SD34 may include an image of at least a portion of the crop. The image data SD34 may be acquired so as to capture only the target crop, or may be acquired so as to capture multiple crops including the target crop.

[0146] The weather data SD35 is an example of second sensing data SD2. As long as it can be used to infer the state of the crop, the type and acquisition method of the weather data SD35 are not particularly limited and may be selected appropriately depending on the embodiment. The weather data SD35 may be similar to the weather data SD33 in the above-mentioned Example 2-7. One or more second sensors S35 may be used to acquire the weather data SD35. The second sensors S35 may be similar to the second sensors S33 in the above-mentioned Example 2-7.

[0147] The format of the result of inferring the crop condition may be selected appropriately depending on the embodiment. In one example, inferring the crop condition may include at least one of identifying the level of an arbitrary crop-related index and regressing the value of the index. The crop-related index may be defined appropriately according to any criteria, including known criteria. The crop-related index may include, for example, an environmental index indicating the degree to which the environment is suitable for growth, a growth index indicating the level of crop growth, etc. The environmental index may include, for example, an index for determining the timing of watering, an index for adjusting the growth environment (amount of water, amount of light, temperature, humidity, etc.), etc. This makes it possible to monitor the growth status of the crop. Note that the type of crop is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the crop may include agricultural crops such as common crops, forage crops, green manure crops, horticultural crops, and industrial crops.

[0148] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD34 and a plurality of second training samples 350 of weather data SD35 for the crop SS34. The individual crop SS34 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to one example of this embodiment, the state of the crop is inferred from the image data SD34 and the weather data SD35. In this case, we can expect improved inference accuracy in the generated trained models (51, 52, 53).

[0149] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD34 and a second sample 355 of weather data SD35 for the crop TT34. The target individual of the crop TT34 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (inferring the state of the crop TT34) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the crop TT34 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the crop TT34 may be provided to an arbitrary user, such as an administrator managing the crop TT34. The output device may include a user terminal. Furthermore, when connected to a cultivation device configured to adjust the environment of the crop TT34, the inference device 2 may issue a command to the cultivation device to adjust the environment of the crop TT34 according to the inference result, as an example of a process of outputting information regarding the result of the inference task. As long as the cultivation device can control the growth environment of the crop TT34, the type of cultivation device is not particularly limited and may be appropriately selected depending on the embodiment. The cultivation device may include, for example, a watering device, a lighting device, a curtain device, a window device, an air conditioning device, etc. The watering device may be, for example, a sprinkler. The lighting device may be, for example, an LED (light emitting diode) light, a fluorescent lamp, etc. The curtain device may be configured to open and close curtains attached to windows of a building. The window device may be configured to open and close windows of a building. The air conditioning device may be, for example, an air conditioner. The watering device may be used to control the amount of water applied to the crop TT34. The lighting device and the curtain device may be used to control the time and amount of light irradiated onto the crop TT34. The air conditioning device may be used to control the temperature and humidity around the crop TT34. For example, if the inference device 2 infers that the state of the crop TT34 is lacking in water, the inference device 2 may issue a command to the watering device to perform watering. This allows at least a part of the cultivation of the crop TT34 to be automated.According to one example of this embodiment, when inferring the condition of crops from image data SD34 and meteorological data SD35, the inference results (inference results of the crop condition) of each model (55, 56, 57) are ensembled, which is expected to improve the accuracy of inference.

[0150] Note that the state of the crops can also be observed using other sensing data. Therefore, at least one of the image data SD34 and the weather data SD35 may be replaced with other sensing data. In one example, either the image data SD34 or the weather data SD35 may be replaced with soil data. Accordingly, either the first sensor S34 or the second sensor S35 may be replaced with a soil sensor.

[0151] (2-9) Case 2-9 FIG. 4I schematically illustrates an example of Case 2-9 to which this embodiment is applied. Case 2-9 is an example of a situation in which this embodiment is applied to inferring a subject's state. As shown in FIG. 4I, in one example, the first data 30 may be image data SD36 depicting the subject, and the second data 35 may be biometric data SD37 derived from the subject's living body. The inference task may include inferring the subject's state.

[0152] The image data SD36 is an example of the first sensing data SD1. As long as it can be used to infer the state of the subject, the type and acquisition method of the image data SD36 are not particularly limited and may be selected appropriately depending on the embodiment. The image data SD36 may include medical image data in addition to general image data. The medical image data may be similar to the medical image data SD10 in the above-mentioned 1-1 case. One or more first sensors S36 may be used to acquire the image data SD36. The first sensors S36 may be, for example, an image sensor, an ultrasound diagnostic device, an X-ray device, a CT device, an MRI device, a PET device, a scintigraphy device, etc. may be selected.

[0153] The biological data SD37 is an example of the second sensing data SD2. The type and acquisition method of the biological data SD37 are not particularly limited as long as they can be used to infer the subject's condition, and may be appropriately selected depending on the embodiment. The biological data SD37 may include, for example, blood pressure, pulse, heart rate, electrocardiogram, electromyogram, body temperature, electrodermal response, brain activity (electroencephalogram, magnetoencephalogram), activity level, blood glucose level, electrooculography, eye movement, and smell. One or more second sensors S37 may be used to acquire the biological data SD37. The second sensors S37 may be selected from, for example, a blood pressure monitor, a pulse rate monitor, a heart rate monitor, an electrocardiogram, an electromyogram, a thermometer, an electrodermal response meter, an electroencephalogram, a magnetoencephalogram, an activity meter, a blood glucose level meter, an electrooculography sensor, an eye movement measuring device, and an odor sensor. At least a portion of the second sensors S37 may be a wearable device.

[0154] The format of the result of inferring the subject's condition may be selected appropriately depending on the embodiment. In one example, inferring the subject's condition may include at least one of identifying the level of any indicator related to the subject and regressing the value of the indicator. The indicator related to the condition may be defined appropriately according to any criteria, including known criteria. The subject's condition may include the subject's health condition. As in Example 1-1 above, inferring the health condition may include determining whether or not a disease has occurred.

[0155] For example, the image data SD36 may be acquired to capture the inside of the subject's mouth. The biometric data SD37 may include odor observation data. Inferring the subject's condition may include inferring the degree of the subject's periodontal disease. For example, the image data SD36 may include medical image data. The biometric data SD37 may include electrocardiogram data. Inferring the subject's condition may include inferring a sign of myocardial infarction in the subject. For example, the image data SD36 may include a temperature distribution or thermography image. The biometric data SD37 may include heart rate data. Inferring the subject's condition may include inferring the subject's core body temperature. For example, the image data SD36 may be acquired to capture pupil diameter, gaze point, eye movements such as blinking, etc. The biometric data SD37 may include brain activity data. Inferring the subject's condition may include inferring the subject's attention (wakefulness, etc.).

[0156] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of image data SD36 and a plurality of second training samples 350 of biometric data SD37 for the subject SS36. The subject SS36 is an example of a target person and an example of an individual during learning. The number of subjects SS36 may be determined appropriately depending on the embodiment. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to an example of the present embodiment, when inferring the state of the subject from the image data SD36 and the biometric data SD37, it is possible to expect improved inference accuracy in the generated trained models (51, 52, 53).

[0157] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of image data SD36 and a second sample 355 of biometric data SD37 for the subject TT36. The subject TT36 is an example of a target individual at the time of inference. The inference device 2 may derive the result of the inference task (inferring the state of the subject TT36) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state of the subject TT36 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state of the subject TT36 may be provided to an arbitrary user, such as the subject TT36 himself. The output device may include a user's terminal. According to one example of this embodiment, the corresponding data is acquired from the image data SD36 and the biometric data SD37. When inferring the subject's state, it is expected that the accuracy of inference will be improved by ensembling the inference results (inference results of the subject's state) of each model (55, 56, 57).

[0158] (2-10) Case 2-10 FIG. 4J schematically illustrates an example of Case 2-10 to which this embodiment is applied. Case 2-10 is an example of a scenario in which this embodiment is applied to inferring the state of a house. As shown in FIG. 4J, in one example, the first data 30 and the second data 35 may be first observation data SD38 and second observation data SD39 about the interior of the house. The inference task may include inferring the state of the interior of the house.

[0159] The first observation data SD38 is an example of first sensing data SD1, and the second observation data SD39 is an example of second sensing data SD2. As long as the first observation data SD38 and the second observation data SD39 can be used to infer the state of the interior of the house, the types and acquisition methods of the first observation data SD38 and the second observation data SD39 are not particularly limited and may be appropriately selected depending on the embodiment. One or more first sensors S38 may be used to acquire the first observation data SD38, and one or more second sensors S39 may be used to acquire the second observation data SD39. The first sensors S38 and the second sensors S39 may each be selected from, for example, various sensors constituting a home security sensor, an odor sensor, etc. As an example, the first sensor S38 may be a home security sensor, and the second sensor S39 may be an odor sensor. The first sensors S38 and the second sensors S39 may be appropriately positioned to observe the interior of the house.

[0160] The format of the result of inferring the state inside the home may be selected appropriately depending on the embodiment. In one example, inferring the state inside the home may include identifying whether an abnormality has occurred inside the home and / or regressing the probability that a specific abnormality has occurred. Inferring the state inside the home may include identifying the type of abnormality occurring inside the home. Inferring the state inside the home may also include regressing the probability of the abnormality occurring for each type. Each model (51, 52, 53, 55, 56, 57) may be configured specifically to determine whether only a specific abnormality has occurred, or may be configured generally to determine whether any abnormality has occurred. The type of abnormality occurring inside the home may be defined appropriately from the perspective of, for example, security (crime prevention, emergency situations, etc.), monitoring (abnormalities of residents, etc.), etc.

[0161] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of the first observation data SD38 and a plurality of second training samples 350 of the second observation data SD39 for the house SS38. The individual of the house SS38 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when inferring the state of the interior of a house from two types of observation data (the first observation data SD38 and the second observation data SD39), it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0162] In the inference stage, the inference device 2 may acquire a first sample 305 of the first observation data SD38 and a second sample 355 of the second observation data SD39 for the house TT38. The target individual of the house TT38 is an example of a target individual at the time of inference. The inference device 2 may derive the result of the inference task (inferring the state inside the house TT38) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of inferring the state inside the house TT38 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of inferring the state inside the house TT38 may be provided to an arbitrary user such as a resident of the house TT38, a relative of the resident, or a manager. The output device may include a user's terminal. According to one example of this embodiment, two types of observation data ( When inferring the state of the interior of a house from the first observation data SD38 and the second observation data SD39, it is expected that the accuracy of inference will be improved by ensembling the inference results (inference results of the state of the interior of the house) of each model (55, 56, 57).

[0163] (others) Which of the two or more types of sensing data is used as the first data 30 and the second data 35 may be determined appropriately depending on the embodiment, etc. In each of the above cases, the first data 30 and the second data 35 may be interchanged. For example, in the above case 2-1, the image data SD20 may be used as the second data 35, and the numerical data SD21 may be used as the first data 30.

[0164] (3) Third Case The third example shown in FIG. 2C can be applied to various situations in which two or more types of data other than sensing data are used for inference. As an example, the system according to this embodiment may be applied to at least one of the following two situations. However, the application of this embodiment is not limited to the following two examples and may be determined appropriately depending on the embodiment.

[0165] (3-1) Case 3-1 FIG. 5A schematically illustrates an example of Case 3-1 to which this embodiment is applied. Case 3-1 is an example of a scenario in which this embodiment is applied to stock price prediction. As shown in FIG. 5A, in one example, the first data 30 may be market data AD50 related to a target of prediction, and the second data 35 may be text data AD51 related to the target. The inference task may include predicting the stock price of the target.

[0166] Market data AD50 is an example of data AD1. As long as it can be used for stock price prediction, the structure and origin of market data AD50 are not particularly limited and may be selected appropriately depending on the embodiment. Market data AD50 may include, for example, current stock prices, trading volumes, market indicators, yields, credit ratings, commodity prices, commodity production volumes, exchange rates, currency trading volumes, interest rates, etc.

[0167] The text data AD51 is an example of the data AD2. As long as it can be used for stock price prediction, the structure and origin of the text data AD51 are not particularly limited and may be appropriately selected depending on the embodiment. The text data AD51 may include, for example, news, SNS data, etc. The SNS data may include, for example, user comments, signatures, emotion identification results, etc.

[0168] The format of the stock price prediction result may be selected appropriately depending on the embodiment. In one example, predicting stock prices may include at least one of identifying whether stock prices will rise or not and regressing future stock prices. The target for stock price prediction is not particularly limited and may be selected appropriately depending on the embodiment. The target may include, for example, a company, an index (such as the Nikkei average), etc.

[0169] In the learning stage, the model generation device 1 may acquire a plurality of first training samples 300 of market data AD50 and a plurality of second training samples 350 of text data AD51 for a target SS50. The target SS50 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when predicting the stock price of a target from the market data AD50 and the text data AD51, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0170] Also, in the inference stage, the inference device 2 may acquire a first sample 305 of market data AD50 and a second sample 355 of text data AD51 for the target TT50. The target TT50 is an example of a target individual during inference. The inference device 2 may derive a result of the inference task (predicting the stock price of the target TT50) from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of predicting the stock price of the target TT50 to at least one of an arbitrary output device and a memory area as information regarding the result of the inference task. In one example, the information indicating the result of predicting the stock price of the target TT50 may be provided to an arbitrary user via a server device. In this case, the inference device 2 may provide the information indicating the prediction result to the server device. Alternatively, the server device may operate as the inference device 2. According to one example of this embodiment, when predicting a target stock price from market data AD50 and text data AD51, the inference results (stock price prediction results) of each model (55, 56, 57) are ensembled, which is expected to improve the inference accuracy.

[0171] (3-2) Case 3-2 FIG. 5B schematically illustrates an example of Case 3-2 to which this embodiment is applied. Case 3-2 is an example of a situation in which this embodiment is applied to content type classification. As shown in FIG. 5B, in one example, the first data 30 may be content data AD52 constituting the content, and the second data 35 may be associated data AD53 related to the content. The inference task may include classifying the type of content.

[0172] The content data AD52 is an example of data AD1. The content data AD52 may be the content itself. The content data AD52 may include, for example, image data, sound data, text data (document data, etc.), etc. The associated data AD53 is an example of data AD2. As long as it is related to the content, the type of the associated data AD53 is not particularly limited and may be selected appropriately depending on the embodiment. The associated data AD53 may include, for example, metadata, user writings on the content (text data), etc. The metadata may include, for example, attribute information such as the title, the name of the creator, and the date of creation.

[0173] The format of the result of classifying the content type may be selected appropriately depending on the embodiment. In one example, classifying the content type may include at least one of identifying a category to which the content belongs and regressing a probability that the content belongs to a target category. Classifying the content type may include classifying whether the content includes specific information such as fake information. Classifying whether the content includes specific information may include either identifying whether the content includes the specific information or regressing a probability that the content includes the specific information.

[0174] In the learning stage, the model generation device 1 may acquire, for the content SS52, a plurality of first training samples 300 of the content data AD52 and a plurality of second training samples 350 of the associated data AD53. The individual of the content SS52 is an example of an individual during learning. The model generation device 1 may use the acquired plurality of first training samples 300 and the plurality of second training samples 350 for machine learning of the first model 51, the second model 52, and the fusion model 53. According to this example of the present embodiment, when classifying the type of content from the content data AD52 and the associated data AD53, it is expected that the inference accuracy of the generated trained models (51, 52, 53) will be improved.

[0175] Furthermore, in the inference stage, the inference device 2 may acquire the first sample 305 of the content data AD52 and the second sample 355 of the associated data AD53 for the content TT52. The target individual of the content TT52 is an example of a target individual at the time of inference. The result of the inference task (classifying the type of content TT52) may be derived from the acquired first sample 305 and second sample 355. The inference device 2 may output information indicating the result of classifying the type of content TT52 to at least one of an arbitrary output device and a storage area as information regarding the result of the inference task. In one example, if the content TT52 is classified as including specific information, the inference device 2 may apply a process to delete the specific information from the content TT52 (content data AD52). If the content TT52 is specific information, the content TT52 (content data AD52) itself may be deleted. According to one example of the present embodiment, when classifying the type of content from the content data AD52 and associated data AD53, the inference results (type classification results) of each model (55, 56, 57) can be ensembled to expect improved inference accuracy.

[0176] (others) Which of the two or more types of data is used as the first data 30 and the second data 35 may be determined appropriately depending on the embodiment, etc. In each of the above examples, the first data 30 and the second data 35 may be interchanged. For example, in the above Example 3-1, the market data AD50 may be used as the second data 35, and the text data AD51 may be used as the first data 30.

[0177] [Model] Each model (51, 52, 53, 55, 56, 57) is configured as a machine learning model. The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used in the calculation of the desired inference. The machine learning model may be configured, for example, as a neural network, a regression model, a decision tree model, a support vector machine, or other functional formula (calculation model). The machine learning method may be appropriately selected depending on the machine learning model employed (for example, backpropagation).

[0178] In one example, each model (51, 52, 53, 55, 56, 57) may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or an autoregressive model. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a long short-term memory (LSTM). The neural network may include any type of model such as a diffusion model, a transformer model, or a generative model. The connection weights between each node included in the neural network and the threshold value of each node are examples of calculation parameters.

[0179] The fusion model (53, 56) receives as input the calculation outputs of the first model (51, 55) and the second model (52, 56). The structure of the fusion model (53, 57) is not particularly limited and may be determined appropriately depending on the embodiment, as long as the result of the inference task can be derived from the calculation outputs of the first model (51, 55) and the second model (52, 56). In one example, the fusion model (53, 57) may have a structure that integrates the calculation outputs of the first model (51, 55) and the second model (52, 56) and derives an inference result from the integrated calculation output. The structure of the fusion model (53, 57) may adopt a known structure proposed in Non-Patent Document 1, etc. The fusion model (53, 57) may be configured separately from the first model (51, 55) and the second model (52, 56), or may be configured integrally with at least a part of the first model (51, 55) and the second model (52, 56). The input calculation outputs of the first model (51, 55) and the second model (52, 56) may be at least one of intermediate outputs and final outputs of the first model (51, 55) and the second model (52, 56).

[0180] Machine learning involves adjusting (optimizing) values ​​of computational parameters using training samples. Typically, trained models (51, 52, 53) may be generated by performing supervised learning using multiple training data sets, each of which is composed of a combination of input samples (training samples) and output samples (teacher signals, labels). However, the method of generating the trained models (51, 52, 53) is not limited to this example and may be modified as appropriate depending on the embodiment.

[0181] In one example of this embodiment, the first model 51 and the second model 52 each individually learn the ability to derive an inference result through the first training and the second training. Furthermore, the fusion model 53 learns the ability to derive an inference result through the third training, using the calculation outputs of the first model 51 and the second model 52 in the first training and the second training as training samples. The learning of the fusion model 53 through this third training is also reflected in the first model 51 and the second model 52. By executing this third training together with the first training and the second training, each of the first model 51 and the second model 52 can learn the ability to derive an inference result while reflecting knowledge of the other's data.

[0182] For example, when each model (51, 52, 53) is configured by a neural network, the training method may employ an error backpropagation method. Accordingly, in the first training, the first training sample 300 is provided to the first model 51, and an error calculated therefrom is backpropagated, thereby adjusting the value of a calculation parameter of the first model 51. In the second training, the second training sample 350 is provided to the second model 52, and an error calculated therefrom is backpropagated, thereby adjusting the value of a calculation parameter of the second model 52. In the third training, the first training sample 300 is provided to the first model 51, and the second training sample 350 is provided to the second model 52, and forward calculation processing of each model (51, 52, 53) is performed, thereby deriving an output (inference result) of the fusion model 53. The error in the output of the fusion model 53 may be backpropagated to the first model 51 and the second model 52 via the fusion model 53. By combining the error from this third training with the errors from the first training and the second training, knowledge of each other's data can be reflected in the learning of the first model 51 and the second model 52. In one example of this embodiment, as a result of this training, each model (51, 52) can more accurately extract features suitable for inferring each data (30, 35), and this can be expected to improve the inference accuracy of each model (51, 52).

[0183] Note that, as long as an inference result can be derived, the input / output format of each model (51, 52, 53, 55, 56, 57) may be determined appropriately depending on the embodiment. In one example, the input data (first data 30, second data 35) may be provided to the first model (51, 55) and the second model (52, 56) as is, or may be provided after being preprocessed. Samples of data other than the first data 30 and the second data 35 may also be provided to the first model (51, 55) and the second model (52, 56). Also, in one example, the output of each model (51, 52, 53, 55, 56, 57) may be configured to directly or indirectly indicate the inference result. When the output of each model (51, 52, 53, 55, 56, 57) is configured to indirectly indicate the inference result, the inference result may be obtained by performing any information processing (such as interpretation processing) on ​​the output of each model (51, 52, 53, 55, 56, 57).

[0184] [Training sample selection method] The training samples (300, 350) may be collected as appropriate depending on the embodiment. The training samples (300, 350) may be collected manually or at least partially automatically. Furthermore, the training samples (300, 350) may be collected in the model generation device 1 or in a computer other than the model generation device 1.

[0185] In one example, the inference task may be determining whether or not a predetermined event has occurred, as in the above examples. The predetermined event may be, for example, the above-mentioned disease, abnormality, stock price rise / fall, or a transition to another specific state. Furthermore, at least one of the first data 30 and the second data 35 may be obtained in the form of a sample group (e.g., a group of images, a group of sound samples, etc.). When the inference task is determining whether or not a predetermined event has occurred and the data is obtained in the form of a sample group, a feature related to the occurrence of the predetermined event may appear only in a portion of the sample group. For example, as in Example 1-1 above, when diagnosing a disease, it is typical that lesions related to the disease appear only in a portion of images in a group of medical images (e.g., a group of CT images, a group of MRI images, etc.). Accordingly, the feature may not appear in the remaining samples included in the sample group (i.e., the state may be the same as when the predetermined event has not occurred). In this case, samples that exhibit a feature related to the occurrence of the predetermined event are preferably used as positive samples for machine learning. On the other hand, if the remaining samples that do not represent features are used as positive samples for machine learning, samples equivalent to negative samples may be trained as positive samples, which may result in a deterioration in the inference accuracy of the first model 51, the second model 52, and the fusion model 53.

[0186] This problem is likely to occur in cases where the amount of samples obtained from the same information source is uneven between the first data 30 and the second data 35, such as in the first example. In other words, in the first example, the first data 30 may be obtained as a group of samples of sensing data SD, while the second data 35 may be obtained as a single sample of data AD other than the sensing data SD. For example, in the above-mentioned example 1-1, for a single subject at a single point in time, the first data 30 is typically obtained as a group of medical image samples, while the second data 35 is typically obtained as a single response sample to a medical interview. The above problem may occur if the group of samples of the first data 30 and the second data 35 obtained from information sources where a predetermined event has occurred are treated collectively as positive samples. Therefore, in one example of this embodiment, when an inference task includes determining whether a predetermined event has occurred, samples may be selected using one of the following two selection methods.

[0187] (I) First Selection Method FIG. 6A schematically illustrates an example of a first sample selection method according to this embodiment. In the example illustrated in FIG. 6A, multiple data sets DS10 may be provided, each of which is composed of a combination of a first sample group 3010 of the first data 30 and a second sample 351 of the second data 35. The first sample group 3010 of each data set DS10 includes multiple first samples 301. The first sample group 3010 and the second samples 351 of each data set DS10 are obtained from a common information source (individuals). That is, the data set DS10 is formed by associating the first sample group 3010 and the second samples 351 obtained from the same information source. The same individual may be considered to be from the same information source, or the same individual may be considered to be from different information sources at different times. The multiple data sets DS10 may be collected by the model generation device 1 or may be provided from a computer other than the model generation device 1. Each sample (301, 351) may be composed of one or more pieces of data (e.g., one or more images).

[0188] (Positive Dataset) Each data set DS10 (at least one of the first sample group 3010 and the second sample group 351) may be associated with a truth value of the inference task via a label. The model may be referred to as a teacher signal or correct answer data. The plurality of data sets DS10 may include one or more positive data sets PS10 associated with a true value indicating the occurrence of a predetermined event. For each of the one or more positive data sets PS10, a likelihood of the occurrence of the predetermined event may be calculated for each of the plurality of first samples 301 included in the first sample group 3010 using a computation model MD51 configured to calculate the likelihood of the occurrence of the predetermined event from the first data 30.

[0189] As long as the likelihood can be calculated, the configuration of the computation model MD51 is not particularly limited and may be appropriately selected depending on the embodiment. The computation model MD51 may be configured using at least one of a machine learning model and a rule-based model. When a machine learning model is used, the computation model MD51 may be trained in advance. The computation model MD51 may be configured similarly to the first model 51. Calculating the likelihood may be regressing a probability. When the inference task includes regressing a probability (likelihood), the first model 51 may be used for the computation model MD51. On the other hand, the rule-based model is configured to derive an inference result (calculation of likelihood in this embodiment) from a given input according to a rule. The rule may be set appropriately. Note that calculating the likelihood of the first sample 301 may be executed by the model generation device 1 or may be executed by a computer other than the model generation device 1. The likelihood of the first sample 301 may be calculated during machine learning of each model (51, 52, 53) or may be calculated in advance.

[0190] In machine learning, among a plurality of first samples 301 included in the first sample group 3010 of each of one or more positive data sets PS10, the first sample 301 with the highest calculated likelihood may be used as the first training sample 300. In addition, in machine learning, the corresponding second sample 351 may be used as the second training sample 350 to be used together with the first training sample 300 (first sample 301). That is, the first sample 301 and the second sample 351 that correspond to each other as the data set DS10 (positive data set PS10) may be used as the first training sample 300 and the second training sample 350 to be simultaneously provided to the first model 51 and the second model 52. Note that using the first sample 301 with the highest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the highest calculated likelihood as the first training samples 300 (n is a natural number greater than or equal to 1). The other first samples 301 may not be used for machine learning (that is, may be excluded from the machine learning target).

[0191] According to one example of this embodiment, by selecting the first sample 301 based on the likelihood calculated using the computational model MD51, the first sample 301 that is highly likely to exhibit the characteristics of the occurrence of a predetermined event can be used as a positive sample for machine learning. This reduces the probability of using a sample equivalent to a negative sample as a positive sample, and as a result, it is expected that the deterioration of the inference accuracy of the trained models (51, 52, 53) can be suppressed.

[0192] (negative dataset) 6A , the plurality of data sets DS10 may further include one or more negative data sets NS10 associated with a true value indicating that a predetermined event has not occurred. As with the positive data set PS10, for each of the one or more negative data sets NS10, the likelihood of each of the plurality of first samples 301 included in the first sample group 3010 may be calculated using the computational model MD51.

[0193] In machine learning, the first sample 301 having the highest calculated likelihood is selected from the plurality of first samples 301 included in the first sample group 3010 of each of one or more negative data sets NS10. In machine learning, a corresponding second sample 351 may be used as the first training sample 300. In addition, in machine learning, a corresponding second sample 351 may be used as the second training sample 350 to be used together with the first training sample 300 (first sample 301). That is, the first sample 301 and the second sample 351 corresponding to each other as the dataset DS10 (negative dataset NS10) may be used as the first training sample 300 and the second training sample 350 to be simultaneously provided to the first model 51 and the second model 52. Note that, similar to the positive dataset PS10, in the negative dataset NS10, using the first sample 301 with the highest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the highest calculated likelihood as the first training samples 300 (n is a natural number equal to or greater than 1).

[0194] According to one example of this embodiment, among the multiple first samples 301 included in the negative dataset NS10, the first sample 301 that is most likely to be positive (the first sample that is likely to lead to an error in inference) can be used as a negative sample for machine learning. This can be expected to improve the inference accuracy of the trained models (51, 52, 53).

[0195] Note that the method of selecting the first samples 301 in the negative data set NS10 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, in machine learning, among the multiple first samples 301 included in each of the first sample groups 3010 of one or more negative data sets NS10, the first sample 301 with the lowest calculated likelihood may be used as the first training sample 300. In this way, by using the first sample 301 with the highest probability of being negative as the negative sample in machine learning, it is possible to train each model (51, 52, 53) to acquire the ability to narrow down to clearly negative cases and discriminate them as negative. Using the first sample 301 with the lowest calculated likelihood as the first training sample 300 may include using n first samples 301 starting from the first sample 301 with the lowest calculated likelihood as the first training samples 300 (n is a natural number greater than or equal to 1). In yet another example, the selection of the first sample 301 in the negative data set NS10 may be omitted, and any first sample 301 included in the first sample group 3010 may be used as the first training sample 300.

[0196] (II) Second Selection Method 6B schematically illustrates an example of a second sample sorting method according to the present embodiment. In the example of FIG. 6B, similar to FIG. 6A, a plurality of data sets DS10 may be provided, each of which is composed of a combination of a first sample group 3010 of the first data 30 and a second sample 351 of the second data 35. The plurality of data sets DS10 may include one or more positive data sets PS10 and one or more negative data sets NS10.

[0197] 6A, a truth value of an inference task may be associated with each sample (301, 302). In one example, a label 302 may be assigned to each first sample 301 included in the first sample group 3010, and a label 352 may be assigned to each second sample 351. Each label 302 may be configured to indicate a truth value of the inference task for the associated first sample 301. The label 352 may be configured to indicate a truth value of the inference task for the associated second sample 351.

[0198] In machine learning, each of the multiple first samples 301 included in the first sample group 3010 of each data set DS10 may be used as a first training sample 300, and the second sample 351 may be used as a second training sample 350. That is, the first sample 301 and the second sample 351 corresponding to each other in the data set DS10 may be used as a first model 51. and may be used as first training samples 300 and second training samples 350 that are simultaneously fed to the second model 52 .

[0199] If the true value of the inference task associated with the first sample 301 matches the true value of the inference task associated with the second sample 351, the first, second, and third training may be performed. On the other hand, if the true value of the inference task associated with the first sample 301 does not match the true value of the inference task associated with the second sample 351, the first and second training may be performed, but the third training may be omitted. That is, if the values ​​of the labels (302, 352) of the first sample 301 and the second sample 351 match, the third training may be performed on the first model 51, the second model 52, and the fusion model 53. On the other hand, if the values ​​of the labels (302, 352) do not match, the third training on the first model 51, the second model 52, and the fusion model 53 may not be performed.

[0200] According to one example of this embodiment, whether or not to perform the third training is controlled depending on whether the associated true values ​​(label values) match. This prevents the first sample 301 and the second sample 351, whose true values ​​for the inference task are different from each other, from being used simultaneously in the third training of the fusion model 53. As part of this effect, it is possible to avoid training samples equivalent to negative samples as positive samples. As a result, it is expected that the deterioration of inference accuracy in the trained models (51, 52, 53) will be suppressed.

[0201] (others) 6A and 6B, the first data 30 is obtained as a sample group, and the above methods are applied to sorting the samples (first samples 301) of the first data 30. However, the data obtained as the sample group does not have to be limited to the first data 30. Instead of the first data 30, or together with the first data 30, second data 35 may be obtained as a sample group. When the second data 35 is obtained as a sample group, the above sorting methods may be applied to the samples of the second data 35.

[0202] [Controlling Machine Learning] Controlling machine learning may include executing machine learning within the model generation device 1 and issuing instructions to an external computer to execute the machine learning. That is, in one example, the model generation device 1 may execute machine learning of the first model 51, the second model 52, and the fusion model 53. In another example, the model generation device 1 may issue instructions to an external computer to execute machine learning of the first model 51, the second model 52, and the fusion model 53. The external computer may execute machine learning of the first model 51, the second model 52, and the fusion model 53 in response to instructions from the model generation device 1. In another example, the model generation device 1 may execute part of the machine learning processing of the first model 51, the second model 52, and the fusion model 53, and the external computer may execute the remaining processing in response to instructions from the model generation device 1. The external computer may be composed of one or more computers. For example, the external computer is a server device. The external computer may be connected to the model generation device 1 via a network or may be directly connected to the model generation device 1. The type of network is not particularly limited and may be selected appropriately depending on the embodiment.

[0203] [Output machine learning results] Outputting the results of the machine learning may include outputting any information related to the results of the machine learning. In one example, outputting the results of the machine learning may include generating learning result data indicating the results of the machine learning and storing the generated learning result data in any storage area. In another example, outputting the results of the machine learning may include outputting any information related to the results of the machine learning. The convergence result may be output, for example, in terms of loss, number of time steps, learning curve, etc.

[0204] [System Configuration] In one example, as shown in Figure 1, a model generation device 1 and an inference device 2 may be connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between each device is not limited to this example and may be appropriately selected depending on the embodiment. In another example, data may be exchanged using a storage medium.

[0205] 1, the model generation device 1 and the inference device 2 are separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. In another example, the model generation device 1 and the inference device 2 may be configured as a single computer. In yet another example, at least one of the model generation device 1 and the inference device 2 may be configured as multiple computers.

[0206] §2 Configuration example [Hardware configuration] (Model generation device) 7 schematically illustrates an example of the hardware configuration of the model generation device 1 according to this embodiment. The model generation device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0207] The control unit 11 includes a CPU (Central Processing Unit) which is a hardware processor, The system includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, etc. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as a generation program 81, a plurality of first training samples 300, a plurality of second training samples 350, first model data 510, second model data 520, and fusion model data 530.

[0208] The generation program 81 is a program for causing the model generation device 1 to execute information processing (see FIG. 11, which will be described later) related to the machine learning of each model (51, 52, 53). The generation program 81 includes a series of instructions for the information processing. The first model data 510 is configured to indicate information related to the trained first model 51 generated by machine learning. The second model data 520 is configured to indicate information related to the trained second model 52 generated by machine learning. The fusion model data 530 is configured to indicate information related to the trained fusion model 53 generated by machine learning. As long as information for executing the arithmetic processing of the trained models (51, 52, 53) can be held, the configurations of the first model data 510, the second model data 520, and the fusion model data 530 are not particularly limited and may be determined appropriately depending on the embodiment. The first model data 510, the second model data 520, and the fusion model data 530 may each be configured to include information indicating values ​​of arithmetic parameters adjusted by machine learning. In some cases, the first model data 510, the second model data 520, and the fusion model data 530 may each be configured to further include information indicating the configuration of each model (51, 52, 53) (for example, the structure of a neural network, etc.). The storage unit of the model data 530 may be selected arbitrarily. In one example, the first model data 510, the second model data 520, and the fusion model data 530 may be held in separate files, or may be held at least partially integrated. In this embodiment, the first model data 510, the second model data 520, and the fusion model data 530 may be generated as a result of executing the generation program 81.

[0209] The communication interface 13 is configured to perform wired or wireless communication via a network. The communication interface 13 may be configured, for example, by a wired LAN (Local Area Network) module, a wireless LAN module, etc. The model generation device 1 may perform data communication with another computer (e.g., the inference device 2, etc.) via the communication interface 13.

[0210] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the model generation device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface. The external interface may be appropriately configured to connect to an external device by wire or wirelessly, for example, via a USB (Universal Serial Bus) port, a dedicated port, etc. The input device 14 and the output device 15 may be integrated into one device, for example, a touch panel display, etc.

[0211] The drive 16 is a device for reading various information such as programs stored in a storage medium 91. At least one of the generation program 81, the first training samples 300, and the second training samples 350 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (such as stored programs) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The model generation device 1 may acquire at least one of the generation program 81, the first training samples 300, and the second training samples 350 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or a DVD, or may be a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface.

[0212] Regarding the specific hardware configuration of the model generating device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a GPU (Gateway Processor), a 3D processor, a 3D image ... The communication interface 13, the input device 14, and the output device 15 may be configured by a PU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), etc. , and at least one of the drive 16 may be omitted. At least one of the first training samples 300 and the second training samples 350 may be held in an external storage device such as a network-attached storage (NAS). At least one of the first model data 510, the second model data 520, and the fusion model data 530 may be stored in at least one of the storage medium 91 and the external storage device. The model generation device 1 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the model generation device 1 may be an information processing device designed specifically for the service to be provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a terminal device, etc.

[0213] (Inference device) 8 shows a schematic diagram of an example of the hardware configuration of the inference device 2 according to this embodiment. The inference device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, a drive 26, and an external interface 27 are electrically connected.

[0214] The control unit 21 to the drive 26 and the storage medium 92 of the inference device 2 may be configured similarly to the control unit 11 to the drive 16 and the storage medium 91 of the model generation device 1, respectively. The control unit 21 (CPU) is an example of a processor resource of the inference device 2. The storage unit 22 (and RAM, ROM) is an example of a memory resource of the inference device 2. In this embodiment, the storage unit 22 stores various information such as an inference program 82, first model data 550, second model data 560, and fusion model data 570.

[0215] The inference program 82 is a program for causing the inference device 2 to execute information processing (see FIG. 12 described below) related to the performance of an inference task. The inference program 82 includes a series of instructions for the information processing. The first model data 550 is configured to indicate information related to the first model 55. The second model data 560 is configured to indicate information related to the second model 56. The fusion model data 570 is configured to indicate information related to the fusion model 57. The first model data 550, the second model data 560, and the fusion model data 570 may be configured in the same manner as the first model data 510, the second model data 520, and the fusion model data 530 described above. When the trained first model 51 is used as the first model 55, the first model data 550 may be the first model data 510. When the trained second model 52 is used as the second model 56, the second model data 560 may be the second model data 520. When the trained fusion model 53 is used as the fusion model 57, the fusion model data 570 may be the fusion model data 530. At least one of the first model data 550, the second model data 560, and the fusion model data 570 may be incorporated into the inference program 82. At least one of the inference program 82, the first model data 550, the second model data 560, and the fusion model data 570 may be stored in a storage medium 92 instead of or together with the storage unit 22. The inference device 2 may acquire at least one of the inference program 82, the first model data 550, the second model data 560, and the fusion model data 570 from the storage medium 92.

[0216] The inference device 2 may communicate data with other computers (e.g., the model generation device 1) via the communication interface 23. An operator can operate the inference device 2 by using the input device 24 and the output device 25.

[0217] The external interface 27 is configured to connect to an external device via a wired or wireless connection. The external interface 27 may be, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like. The type and number of external interfaces 27 may be determined appropriately depending on the embodiment. In one example of this embodiment, when a sensor (sensor S, first sensor S1, second sensor S2, etc.) is used to acquire a sample, the inference device 2 may be connected to the sensor via at least one of the communication interface 23 and the external interface 27.

[0218] It should be noted that with regard to the specific hardware configuration of the inference device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the communication interface 23, input device 24, output device 25, drive 26, and external interface 27 may be omitted. The inference device 2 may be configured with multiple computers. In this case, the hardware configuration of each computer may or may not be the same. The inference device 2 may be configured with an information processing device designed specifically for the service provided, as well as It may be a general-purpose server device, a general-purpose PC, a tablet PC, a terminal device, or the like.

[0219] [Software configuration] (Model generation device) 9 schematically shows an example of the software configuration of the model generation device 1 according to this embodiment. The control unit 11 of the model generation device 1 executes instructions included in the generation program 81 stored in the storage unit 12 using the CPU. As a result, the model generation device 1 operates as a computer including an acquisition unit 111, a learning processing unit 112, and an output processing unit 113 as software modules. That is, in this embodiment, each software module of the model generation device 1 is realized by the control unit 11 (CPU).

[0220] The acquisition unit 111 is configured to acquire a plurality of first training samples 300 of the first data 30 and a plurality of second training samples 350 of the second data 35. The learning processing unit 112 is configured to control machine learning of the first model 51, the second model 52, and the fusion model 53. The machine learning includes first training for the first model 51, second training for the second model 52, and third training for each model (51, 52, 53). The output processing unit 113 is configured to output the results of the machine learning.

[0221] (Inference device) 10 schematically shows an example of the software configuration of the inference device 2 according to this embodiment. The control unit 21 of the inference device 2 executes instructions included in the inference program 82 stored in the storage unit 22 using the CPU. This causes the inference device 2 to operate as a computer equipped with an acquisition unit 211, an inference unit 212, and an output processing unit 213 as software modules. That is, in this embodiment, similar to the model generation device 1, each software module of the inference device 2 is also realized by the control unit 21 (CPU).

[0222] The acquisition unit 211 is configured to acquire a first sample 305 of the first data 30 and a second sample 355 of the second data 35. By holding first model data 550, second model data 560 and fusion model data 570, the inference unit 212 comprises a first model 55, a second model 56 and a fusion model 57. The inference unit 212 is configured to derive a result of an inference task from the acquired first sample 305 and second sample 355 using the first model 55, the second model 56 and the fusion model 57. Deriving a result of the inference task includes deriving a first interim result of the inference task from the first sample 305 using the first model 55, deriving a second interim result of the inference task from the second sample 355 using the second model 56, deriving a third interim result of the inference task from the calculation output of the first model 55 obtained in the process of deriving the first interim result and the calculation output of the second model 56 obtained in the process of deriving the second interim result using the fusion model 57, and deriving a result of the inference task according to the first interim result, the second interim result, and the third interim result. The output processing unit 213 is configured to output information related to the derived result of the inference task.

[0223] (others) In this embodiment, an example is described in which each software module of the model generation device 1 and the inference device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors or chipsets. Each of the above modules may be implemented as a hardware module. With regard to the software configuration of the model generation device 1 and the inference device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0224] §3 Example of operation [Model generation device] 11 is a flowchart showing an example of the processing procedure of the model generation device 1 according to this embodiment. The following processing procedure is an example of a model generation method executed by a computer. However, the following processing procedure of the model generation device 1 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0225] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires a plurality of first training samples 300 of the first data 30 and a plurality of second training samples 350 of the second data 35.

[0226] In one example, the data type of the first training sample 300 and the second training sample 350 may be selected from any of the above-mentioned cases 1 to 3. As a specific example, the data type of the first training sample 300 and the second training sample 350 may be selected from any of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2.

[0227] In one example, the inference task may be determining whether a predetermined event has occurred. Accordingly, the control unit 11 may acquire multiple data sets DS10 shown in FIG. 6A . The multiple data sets DS10 may include one or more positive data sets PS10, and the likelihood of each first sample 301 included in the first sample group 3010 of the positive data set PS10 may be calculated. Furthermore, in one example, the multiple data sets DS10 may include one or more negative data sets NS10, and the likelihood of each first sample 301 included in the first sample group 3010 of the negative data set NS10 may also be calculated.

[0228] In another example, the control unit 11 may acquire multiple data sets DS10 shown in Fig. 6B. Each first sample 301 included in the first sample group 3010 may be assigned a label 302, and each second sample 351 may be assigned a label 352. Each label (302, 352) may be appropriately configured to indicate the truth value of the inference task for each assigned sample (301, 351).

[0229] When machine learning is performed in an external computer, the control unit 11 may acquire the first training samples 300 and the second training samples 350 and provide the acquired first training samples 300 and the second training samples 350 to the external computer. Alternatively, acquiring the first training samples 300 and the second training samples 350 may be configured by issuing an instruction to the external computer to cause the external computer to acquire the first training samples 300 and the second training samples 350. After acquiring the first training samples 300 and the second training samples 350, the control unit 11 proceeds to the next step S102.

[0230] (Step S102) In step S102, the control unit 11 operates as the learning processing unit 112 and controls the machine learning of the first model 51, the second model 52, and the fusion model 53. The acquired multiple first training samples 300 and second training samples 350 are used in the machine learning. The machine learning includes first training for the first model 51 using the first training samples 300, second training for the second model 52 using the second training samples 350, and third training for each model (51, 52, 53) using each training sample (300, 350).

[0231] In one example, the control unit 11 may perform machine learning of each model (51, 52, 53) in the model generation device 1. In another example, the control unit 11 may perform machine learning of each model (51, 52, 53) The external computer may execute the machine learning of each model (51, 52, 53) in response to the instruction from the model generation device 1.

[0232] 6A , for each positive data set PS10, the first sample 301 with the highest calculated likelihood may be used as the first training sample 300. Also, for each negative data set NS10, the first sample 301 with the highest or lowest calculated likelihood may be used as the first training sample 300. For each data set (PS10, NS10), the corresponding second sample 351 may be used as the second training sample 350.

[0233] In another example, when multiple data sets DS10 shown in FIG. 6B are acquired, each first sample 301 included in each data set DS10 may be used as the first training sample 300, and the second training sample 350 may be used as the second training sample 350. For each data set DS10, it may be determined whether the true value (value of label 302) associated with each first sample 301 and the true value (value of label 352) associated with each second sample 351 included in the first sample group 3010 match. If the true values ​​of the first sample 301 and the second sample 351 match, first training, second training, and third training may be performed. On the other hand, if the true values ​​of the first sample 301 and the second sample 351 do not match, the first training and second training may be performed, but the third training may be omitted.

[0234] As a result of executing the machine learning, it is possible to generate trained models (51, 52, 53) that have acquired the ability to perform an inference task within the category of the used data sets (first training sample 300, second training sample 350). By selecting any of the above-mentioned first to third cases as the data type of the first training sample 300 and the second training sample 350, it is possible to generate trained models (51, 52, 53) that have acquired the ability to perform the inference task of any of the cases. By selecting any of the above-mentioned cases 1-1 to 1-5, 2-1 to 2-10, and 3-1 to 3-2 as the data type of the first training sample 300 and the second training sample 350, it is possible to generate trained models (51, 52, 53) that have acquired the ability to perform the inference task of any of the cases. When the machine learning process is completed, the control unit 11 proceeds to the next step S103.

[0235] (Step S103) In step S103, the control unit 11 operates as the output processing unit 113 to output the results of the machine learning. The output destination and the content of the information to be output may be selected appropriately depending on the embodiment.

[0236] In one example, the control unit 11 may generate model data (510, 520, 530) indicating each trained model (51, 52, 53) as the output process and store the generated model data (510, 520, 530) in a predetermined storage area. In this case, the model data (510, 520, 530) is an example of output information related to the results of machine learning. The predetermined storage area may be, for example, RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, a semiconductor memory, or the like. The external storage device may be, for example, a data server such as a NAS. The external storage device may be, for example, an external storage device. When the machine learning calculation process is performed on an external computer, the model data (510, 520, 530) may be generated by the external computer. In another example, the control unit 11 may output a convergence result of the machine learning training as the output process. The output destination may be, for example, the RAM in the control unit 11, the output device 15, an external computer, an external storage device, a storage medium, or a combination of these. In yet another example, the control unit 11 may output the progress of machine learning training as the output process. In this case, the process of step S103 may be executed together with step S102.

[0237] When the output of the machine learning results is completed, the control unit 11 ends the processing procedure of the model generation device 1 according to this operation example.

[0238] When the inference device 2 uses the trained first model 51 as the first model 55, the generated first model data 510 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method. In one example, the first model data 510 may be provided to the inference device 2 via a network, a storage medium, or the like. In another example, the first model data 510 may be pre-installed in the inference device 2. The same applies to the second model 56 and the fusion model 57. When the trained second model 52 is used as the second model 56, the second model data 520 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method. When the trained fusion model 53 is used as the fusion model 57, the fusion model data 530 may be provided to the inference device 2 from the model generation device 1 or an external computer at any time and by any method.

[0239] Furthermore, the control unit 11 may periodically or irregularly repeat the processes of steps S101 to S103 to update or newly generate at least one of the first model data 510, the second model data 520, and the fusion model data 530. During this repetition, at least a portion of the first training samples 300 and the second training samples 350 may be changed, modified, added, deleted, or the like as appropriate. The control unit 11 may then provide at least one of the updated or newly generated first model data 510, the second model data 520, and the fusion model data 530 to the inference device 2 as appropriate. In this way, at least one of the first model data 510 (550), the second model data 520 (560), and the fusion model data 530 (570) held by the inference device 2 may be updated.

[0240] [Inference device] 12 is a flowchart showing an example of a processing procedure of the inference device 2 according to this embodiment. The following processing procedure is an example of an inference method executed by a computer. However, the following processing procedure of the inference device 2 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0241] (Step S201) In step S201, the control unit 21 operates as the acquisition unit 211 and acquires the first sample 305 of the first data 30 and the second sample 355 of the second data 35.

[0242] In one example, the data type of the first sample 305 and the second sample 355 may be selected from any of the above-mentioned first to third cases. As a specific example, the data type of the first sample 305 and the second sample 355 may be selected from any of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2. In one example, the control unit 21 may acquire at least one of the first sample 305 and the second sample 355 directly or indirectly from a sensor. After acquiring the first sample 305 and the second sample 355, the control unit 21 proceeds to the next step S202.

[0243] (Step S202) In step S202, the control unit 21 operates as the inference unit 212 and calculates the first model 55, The second model 56 and the fusion model 57 are used to derive the result of the inference task from the acquired first sample 305 and second sample 355 .

[0244] The control unit 21 provides the first sample 305 to the first model 55 and the second sample 355 to the second model 56, and executes arithmetic processing of the first model 55, the second model 56, and the fusion model 57. The arithmetic processing may be executed appropriately depending on the embodiment, such as the configuration of each model (55, 56, 57). For example, if each model (55, 56, 57) is configured by a neural network, the control unit 21 may execute forward arithmetic processing of each model (55, 56, 57). As a result of executing this arithmetic processing, the control unit 21 can obtain a first interim result from the first model 55, a second interim result from the second model 56, and a third interim result from the fusion model 57.

[0245] The control unit 21 derives a final result of the inference task by ensembling the first interim result, the second interim result, and the third interim result. The ensembling method may be selected appropriately depending on the embodiment. A known method may be adopted as the ensembling method. For example, if the inference task is classification, the control unit 21 may calculate a simple average or a weighted average of the values ​​derived as each interim result. Alternatively, the control unit 21 may calculate a simple sum or a weighted sum of the values ​​derived as each interim result. When a weighted average or weighted sum is adopted, the weight of each interim result may be set arbitrarily. The control unit 21 may then determine the classification result by comparing the calculated value with a threshold. Furthermore, for example, if the inference task is regression, the control unit 21 may calculate a simple average or a weighted average of the values ​​derived as each interim result. The control unit 21 may acquire the calculated value as the final result.

[0246] In one example, the first model 55 may be the trained first model 51. The second model 56 may be the trained second model 52. The fusion model 57 may be the trained fusion model 53.

[0247] In one example, any one of the above-mentioned first to third cases may be selected as the data type of the first sample 305 and the second sample 355. Any one of the above-mentioned cases 1-1 to 1-5, cases 2-1 to 2-10, and cases 3-1 to 3-2 may be selected as the data type of the first sample 305 and the second sample 355. Each model (55, 56, 57) may be configured to have the ability to perform the inference task of any one of the cases. This makes it possible to derive the result of the inference task corresponding to any one of the cases. After deriving the result of the inference task, the control unit 21 proceeds to the next step S203.

[0248] (Step S203) In step S203, the control unit 21 operates as the output processing unit 213 and outputs information related to the result of the inference task.

[0249] The output destination and the content of the information to be output may be selected appropriately depending on the embodiment. In one example, the control unit 21 may simply output the result of the inference task derived in step S202. In another example, the control unit 21 may execute any information processing based on the result of the derived inference task. The control unit 21 may then output the result of the information processing as information related to the result of the inference task. The output of the result of the information processing may include outputting a specific message according to the inference result, controlling the operation of the controlled device according to the inference result, etc. The output destination may be, for example, RAM, the storage unit 22, the output device 25, another computer, the controlled device, etc. Depending on whether any of the above cases is selected as the data type of the first sample 305 and the second sample 355, the output destination and the content of the information to be output are determined in accordance with any of the above cases. It is okay to do so.

[0250] When the output of information is completed, the control unit 21 ends the processing procedure of the inference device 2 according to this operation example. The control unit 21 may execute a series of processes from step S201 to step S203 at any timing such as a user operation or when a condition is satisfied. The control unit 21 may execute the series of processes from step S201 to step S203 in real time, or may execute them as a post-event inference process.

[0251] [Features] In this embodiment, in step S102, training of the fusion model 53 is performed in parallel with training of each model (51, 52) that derives an inference result individually for each data (30, 35). This allows the fusion model 53 to acquire the ability to derive an inference result from the features of each data (30, 35), while allowing each model (51, 52) to acquire the ability to extract features suitable for inferring each data (30, 35). As a result, it is expected that the possibility of losing features suitable for inferring each data (30, 35) can be reduced. Therefore, according to this embodiment, it is expected that the inference accuracy of the generated trained models (51, 52, 53) can be improved.

[0252] Furthermore, in this embodiment, in step S202, the robustness of the inference can be improved by ensembling the inference results of the first model 55, the second model 56, and the fusion model 57. This can be expected to improve the inference accuracy. The inference accuracy can also be expected to be improved by using the trained first model 51, the trained second model 52, and the trained fusion model 53 for the first model 55, the trained second model 56, and the trained fusion model 57, respectively.

[0253] §4 Variations Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs. Furthermore, various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

[0254] <4.1> Even if the machine learning of one of the first model 51 and the second model 52 is omitted, it is possible to perform the machine learning of the other model and the fusion model 53. Even if the machine learning of one model is omitted, it is possible to make the fusion model 53 acquire the ability to derive an inference result and make the other model acquire the ability to extract features suitable for inference. Therefore, in the above embodiment, the configuration related to the machine learning of one of the first model 51 and the second model 52 may be omitted.

[0255] FIG. 13 schematically illustrates an example of a machine learning process according to a modified example. FIG. 13 illustrates an example of a scene in which the configuration related to machine learning of the second model 52 is omitted. In this modified example, the model generation device 1 may control the machine learning of the first model 51 and the fusion model 53, and output the results of the machine learning. The first model 51 may be configured to derive an inference result from the first data 30, as in the above embodiment. A second model 52 may be provided that is configured to derive a result of an inference task from second data 35 of a type different from the first data 30. The fusion model 53 may be configured to derive a result of an inference task from the calculation outputs of the first model 51 and the second model 52. The machine learning may be performed by deriving a result of an inference task from a first training sample 300 of the first data 30. This may include performing a first training on the first model 51 so that the result of the inference task derived by the first model 51 conforms to the true value, and, together with the first training, performing training (fusion training) on ​​the first model 51 and the fusion model 53 so that the result of the inference task derived by the fusion model 53 conforms to the true value from the calculation output of the first model 51 when deriving the result of the inference task and the calculation output of the second model 52 obtained by providing the second model 52 with second training samples 350 of the second data 35. This modification is also expected to improve the inference accuracy of the trained models (51, 53).

[0256] Note that the target for which the machine learning configuration is omitted does not have to be the second model 52. Instead of the second model 52, the machine learning configuration of the first model 51 may be omitted. In this case, the model generation device 1 may control the machine learning of the second model 52 and the fusion model 53 and output the machine learning results. The first model 51 may be provided as appropriate. The machine learning may include performing second training on the second model 52 so that the result of the inference task derived by the second model 52 from the second training sample 350 of the second data 35 conforms to the true value, and, together with the second training, performing training (fusion training) on ​​the second model 52 and the fusion model 53 so that the result of the inference task derived by the fusion model 53 from the calculation output of the first model 51 obtained by providing the first training sample 300 of the first data 30 to the first model 51 and the calculation output of the second model 52 when deriving the result of the inference task conforms to the true value. This modification is also expected to improve the inference accuracy of the trained models (52, 53).

[0257] <4.2> In the above embodiment, an example has been described in which the result of the inference task is derived from two types of data, the first data 30 and the second data 35. However, the number of types of data used for inference is not limited to two, and may be three or more. Deriving the result of the inference task from the first data 30 and the second data 35 may be configured by deriving the result of the inference task from three or more types of data including the first data 30 and the second data 35. Accordingly, in one example, the fusion model (53, 57) may be configured to accept input of three or more types of data. In another example, the fusion model (53, 57) may be provided for each combination of two or more types of data.

[0258] (First Modification) 14A schematically illustrates an example of the configuration of a fusion model (53, 57) according to a modified example. In the example of FIG. 14A, a third model (61, 65) is provided that is configured to derive a result of an inference task from third data of a type different from the first data 30 and the second data 35. The fusion model (53, 57) may be configured to further receive input of the calculation outputs of the first model (51, 55), the second model (52, 56), and the third model (61, 65), and to derive a result of the inference task from the input calculation outputs of the first model (51, 55), the second model (52, 56), and the third model (61, 65).

[0259] In the learning stage, the machine learning may further include training the third model so that a result of an inference task derived by the third model 61 from a third training sample of the third data conforms to a true value. In the third training, the first model 51, the second model 52, the third model 61, and the fusion model 53 may be trained so that a result of an inference task derived by the fusion model 53 from the calculation outputs of the first model 51, the second model 52, and the third model 61 when deriving a result of the inference task conforms to a true value. Alternatively, as in <4.1> above, the machine learning configuration of the third model 61 may be omitted.

[0260] Meanwhile, in the inference stage, the inference device 2 may further acquire a third sample of third data. The inference device 2 may use a third model 65 to infer the inference task from the acquired third sample. An interim result may be further derived. The fusion model 57 may receive the calculation outputs of the first model 55, the second model 56, and the third model 65 obtained in the process of deriving the interim result. The inference device 2 may derive a final result of the inference task from the interim results of the first model 55, the second model 56, the third model 65, and the fusion model 57. In one example, the trained third model 61 may be used for the third model 65.

[0261] According to this modification, improvement in inference accuracy can be expected in situations where the result of an inference task is derived from three or more types of data. Note that the model to be added does not have to be limited to the third model (61, 65). Fourth and subsequent models may also exist. The fusion model (53, 57) may be configured to further accept input of the calculation output of the fourth and subsequent models and derive the result of the inference task from the input calculation output.

[0262] (Second Modification) FIG. 14B schematically illustrates another example of the configuration of a fusion model according to a modified example. In the example of FIG. 14B, similar to FIG. 14A, a third model (61, 65) configured to derive the result of an inference task from third data of a type different from the first data 30 and the second data 35 is provided. A fusion model (531, 571) is provided between the first model (51, 55) and the second model (52, 56). A fusion model (532, 572) is provided between the second model (52, 56) and the third model (61, 65). The fusion model (531, 571) and the fusion model (532, 572) are examples of the fusion model (53, 57). A fusion model may also be provided between the first model (51, 55) and the third model (61, 65).

[0263] In the learning stage, the fusion model 531 may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications. The fusion model 532 may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications, except that the first model 51 is replaced with the third model 61. When a fusion model is also provided between the first model 51 and the third model 61, the fusion model may be trained in the same manner as the fusion model 53 in any of the above embodiments and modifications, except that the second model 52 is replaced with the third model 61.

[0264] Meanwhile, in the inference stage, the inference device 2 may further acquire a third sample of the third data. The inference device 2 may further derive a provisional result of the inference task from the acquired third sample using a third model 65. The inference device 2 may provide the calculation outputs of the first model 55 and the second model 56 obtained in the process of deriving the provisional result to the fusion model 571 and execute the calculation processing of the fusion model 571 to acquire the provisional result of the inference task from the fusion model 571. Similarly, the inference device 2 may provide the calculation outputs of the second model 56 and the third model 65 obtained in the process of deriving the provisional result to the fusion model 572 and execute the calculation processing of the fusion model 572 to acquire the provisional result of the inference task from the fusion model 572. The inference device 2 may derive a final result of the inference task from the provisional results of the first model 55, the second model 56, the third model 65 and each fusion model (571, 572). When a fusion model is also provided between the first model 55 and the third model 65, the inference device 2 may obtain an interim result of the inference task from the fusion model by providing the calculation outputs of the first model 55 and the third model 65 obtained in the process of deriving the interim result to the fusion model and performing calculation processing on the fusion model. The inference device 2 may further use this interim result to derive the final result of the inference task. In one example, the trained third model 61 may be used for the third model 65. The trained fusion model 531 may be used for the fusion model 571. The trained fusion model 532 may be used for the fusion model 572.

[0265] According to this modification, it is possible to expect an improvement in inference accuracy when deriving the result of an inference task from three or more types of data. The model to be added is the third model (61, 65). ). A fourth model and subsequent models may also exist. Fusion models (53, 57) may also be provided between the fourth model and subsequent models. In the example of FIG. 14B, the fusion models (531, 532, 571, 572) are arranged for each combination of two types of data. However, the relationship between the fusion models (53, 57) and the data types is not limited to this example. As in the example of FIG. 14A, the fusion models (53, 57) may be arranged for each of three or more types of data.

[0266] §5 Working Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

[0267] [Preparation] As a preliminary step, we prepared two datasets. The first dataset is the one from Reference 1 (Yuyin Zhou, et al. "RadFusion: Benchmarking Performance and Fairness for "Multimodal Pulmonary Embolism Detection from CT and EHR", [online], [Retrieved March 22, 2024], Internet<URL:https: / / arxiv.org / abs / 2111.11665> ) used We used the Stanford University Pulmonary Embolism Dataset (PE Dataset), which is widely used in clinical trials. This PE dataset consisted of CT images and clinical data extracted from the electronic medical records of the same patients. The clinical data was organized in a table format and included questionnaire responses. Specifically, the clinical data included 1,505 attributes such as age, gender, and pulse rate. Reference 2 (Avanti Shrikumar, et al. "Learning Important Features Through Propagating Activation Differences", [online], [Retrieved March 22, 2024], [Link to the original text]). Internet<URL:https: / / arxiv.org / abs / 1704.02685> Using the method proposed in

[2013] , we selected 250 attributes useful for identifying pulmonary embolism, and used the tabular data consisting of the selected 250 attributes for machine learning. Each CT image was resized from 256 × 256 to 224 × 224. Then, the size of each CT image was converted to 192 × 192 by cropping the center. The intensity of each CT image was normalized to the range [-1, 1]. The PE dataset contained a total of 1,837 combinations of CT images and clinical data (tabular data). Of these, 1,454 combinations were used for training, 190 combinations for validation, and 193 combinations for testing. Each combination consisted of 100 CT images. Five samples (192 × 192 × 24) were obtained by dividing the 100 CT images into 24 consecutive CT images. In the fifth sample (group), four CT images were expanded to 24 CT images.

[0268] The second dataset is based on reference 3 (Duane L Beekly, et al. "The National Alzheimer's Coordinating Center (NACC) Database: an Alzheimer disease database", [online], [searched March 22, 2024], Internet<URL:https: / / pubmed.ncbi.nlm.nih.gov / 15592144 / > ) used the NACC dataset. In the CC dataset, cognitive impairment levels were classified into three categories: 0 (normal), 1 (mild), and 2 (severe). The NACC dataset consisted of brain MRI images and questionnaire response data (table format). MRI images included T1, T2, and FLAIR images. Among these, T1 images were selected and bone removal, resizing, and intensity normalization were performed using imaging software (FreeSurfer). After this preprocessing, 1252 MRI images (3D volumes) measuring 182 × 218 × 182 voxels were obtained. The volume intensities were normalized to the range [-8, 8]. Meanwhile, preprocessing by zero padding, feature selection, and normalization was applied to the response data. After preprocessing, the response data (table data) consisted of 99 attributes, including cognitive questionnaire information, gender, age, etc. These resulted in 1252 combinations of MRI images and questionnaire response data (table data). Of these, 752 combinations were used for training, 249 combinations for validation, and 251 for testing. The NACC dataset includes classification of cognitive impairment levels. It was adopted for machine learning of multi-class classification tasks.

[0269] [Model configuration] (First Comparative Example) FIG. 15A shows the model configuration of the first comparative example. The model of the first comparative example was configured to classify the presence or absence of each disease from each image. The model structure of the first comparative example was based on the model described in Reference 4 (Shih-Cheng Huang, et al. "PENet - a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging", [online], [searched March 22, 2024], Internet).<URL:https: / / www.nature.com / articles / s41746-020-0266-y> ) proposed PENet architecture.

[0270] (Second Comparative Example) FIG. 15B shows the model configuration of the second comparative example. The model of the second comparative example was configured to classify the presence or absence of each disease from tabular data. The model structure of the second comparative example is based on the model described in Reference 5 (Yury Gorishniy, et al. "Revisiting Deep Learning Models for Tabular Data", [online], [searched March 22, 2024], Internet)<URL:https: / / arxiv.org / abs / 2106.11959> ) proposed the Tabular Transformer.

[0271] (Third Comparative Example) FIG. 15C shows the model configuration of the third comparative example. The model of the third comparative example was configured to classify the presence or absence of each disease from images and table data. The model structure of the third comparative example adopted the MMTM architecture proposed in Non-Patent Document 1. The part that extracts image features adopted the PENet architecture similar to that of the first comparative example. The part that extracts table data features The same Tabular Transformer as in the second comparative example was used.

[0272] (Fourth Comparative Example) FIG. 15D shows the model configuration of the fourth comparative example. Similar to the third comparative example, the model of the fourth comparative example was configured to classify the presence or absence of each disease from image and tabular data. The model structure of the fourth comparative example was based on the model described in Reference 6 (Paul Hager, et al. "Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data", [online], [searched March 22, 2024], Internet<URL:https: / / arxiv.org / abs / 2303.14080> ) proposed in The CLIP architecture, which is used in the first comparative example, was adopted for the part that extracts image features. The ENet architecture, which is the same as in the first comparative example, was adopted for the part that extracts table data features. The Tabular Transformer, which is the same as in the second comparative example, was adopted for the part that extracts table data features. As shown in Figure 15D, the model of the fourth comparative example was configured to fuse the features of each data set via a communication unit during the learning stage. On the other hand, the model of the fourth comparative example was configured to obtain a final inference result by ensembling the inference results (first output and second output) derived from the parts that extract the features of various data sets without using a communication unit during the inference stage. The ensemble was performed by simple averaging.

[0273] (Fifth Comparative Example) The model of the fifth comparative example was constructed using the models of the first and second comparative examples. Specifically, the model of the fifth comparative example was configured to derive an inference result by ensembling the inference results of the models of the first and second comparative examples. As a result, the model of the fifth comparative example was also configured to classify the presence or absence of each disease from image and table data, similar to the third comparative example, etc. The ensemble was performed by simple averaging.

[0274] (Sixth Comparative Example) FIG. 15E shows the model configuration of the sixth comparative example. The model of the sixth comparative example was prepared to learn the classification task of the NACC dataset. The model of the sixth comparative example was The system was configured to classify cognitive impairment levels from the table data and image features. A convolutional neural network was used to extract image features. A tabular transformer, similar to the second comparative example, was used to extract table data features.

[0275] (First Example) FIG. 16A shows the model configuration of the first embodiment. The model of the first embodiment was configured to include a first model, a second model, and a fusion model. The first model employed the same PENet architecture as the first comparative example. The second model employed the same Tabular Transformer as the second comparative example. The fusion model employed the same MMTM architecture as the third comparative example. In the inference stage, the model of the first embodiment was configured to obtain a final inference result by ensembling the inference results of the first model, the second model, and the fusion model. The ensemble was performed by simple averaging. The main differences between the first embodiment and the third comparative example were that each of the first and second models had a section for outputting the inference result, and that in the learning stage, machine learning was performed on the fusion model as well as on the first and second models individually.

[0276] (Second Example) FIG. 16B shows the model configuration of the second embodiment. The model of the second embodiment is a concatenated model including a Transformer block, which replaces the fusion model in the model configuration of the first embodiment. In all other respects, the model of the second embodiment was constructed in the same manner as the model of the first embodiment.

[0277] (Third Example) FIG. 16C shows the model configuration of the third embodiment. The model of the third embodiment is obtained by replacing the fusion model in the model configuration of the first embodiment with the TokenFusion-based fusion model proposed in Reference 7 (Yikai Wang, et al. "Multimodal Token Fusion for Vision Transformers", [online], [searched on March 22, 2024], Internet <URL:https: / / arxiv.org / abs / 2204.08721>). In other respects, the model of the third embodiment is configured in the same manner as the first embodiment. Except for this, the model of the third embodiment is configured in the same manner as the first embodiment.

[0278] (Fourth Embodiment) FIG. 16D shows the model configuration of the fourth embodiment. The model of the fourth embodiment is obtained by replacing the fusion model in the model configuration of the first embodiment with the co-attention-based fusion model proposed in Reference 8 (Pei Li, et al. "Multimodal Fusion with Co-attention Mechanism", [online], [searched on March 22, 2024], Internet <URL:https: / / ieeexplore.ieee.org / document / <9190483>). In other respects, the model of the fourth embodiment is configured in the same manner as the first embodiment.

[0279] (Fifth Embodiment) FIG. 16E shows the model configuration of the fifth embodiment. The model of the fifth embodiment was prepared to master the classification task of the NACC dataset. In the model of the fifth embodiment, a first model was constructed by adding a fully connected (linear) layer to the part of the model of the sixth comparative example that extracts image features. A second model was constructed by adding a fully connected (linear) layer to the part of the model of the sixth comparative example that extracts table data features. As a result, the model of the fifth embodiment was configured to include a first model, a second model, and a fusion model. In the inference stage, the model of the fifth embodiment, like the first embodiment, was configured to ensemble the inference results of the first model, the second model, and the fusion model to obtain a final inference result. The ensemble was performed by simple averaging. The first model and the second model each have a part that outputs the inference result, and in the learning stage, machine learning is performed on the fusion model, and individual machines for each of the first model and the second model are used. The implementation of the learning was the main difference between the fifth example and the sixth comparative example.

[0280] (Sixth Example) FIG. 16F shows the model configuration of the sixth embodiment. Similar to the fifth embodiment, the model of the sixth embodiment was prepared to learn the classification task of the NACC dataset. The model of the sixth embodiment was constructed by replacing the fusion model in the model configuration of the fifth embodiment with the same fusion model as the second embodiment. In other respects, the model of the sixth embodiment was constructed in the same way as the fifth embodiment.

[0281] (others) In each figure, "Transformer" refers to the Transformer (second comparative example) proposed in Reference 5. "PENet" refers to the PENet encoder (first comparative example) proposed in Reference 4. "Transformer block" refers to the Transformer block that performs multi-head attention and feedforward. The basic structure of the Transformer is shown below, which is composed of a network. "AP" and "Avgpool" indicate average pooling. "Maxpooling" and "Maxpool" indicate maximum value pooling. "TokenFusion" indicates the TokenFusion block. "Co-attention" indicates the Co-attention block. "Conv" indicates convolution. "Conv3D" indicates 3D convolution. "Reshape" indicates the Reshape function. "Flatten" is, Indicates flattening (making it one-dimensional). "Concat" indicates channel-wise concatenation. "BN" indicates batch normalization. "ReLu" indicates the ReLu function. "CLS Emb" indicates class token embedding.

[0282] [First experiment] Using the PE dataset and the model of Example 1, we compared the first selection method (Fig. 6A) and the second selection method (Fig. 6B). Specifically, we employed both the first and second selection methods under two conditions to obtain trained models.

[0283] First, in the first selection method, we use the pre-trained PENet obtained in Reference 1. The likelihood of each of the five samples was calculated. The sample with the highest likelihood, both positive and negative, was then used for machine learning, and the remaining samples were excluded from machine learning. In the second selection method, a label was assigned to each of the five samples, and training of the fusion model (third training) was performed if the labels of the CT image sample and the table data matched. If the labels of the CT image sample and the table data did not match, training of the fusion model was omitted.

[0284] In the first condition of each selection method ("no prior training"), reference 9 (Joao Carreira, et al. "A Short Note about Kinetics-600", [online], [searched March 22, 2024], Inter -Net<URL:https: / / arxiv.org / abs / 1808.01340> ) only the first model (PENet) was pre-trained on the Kinetics-600 dataset. In the second condition (with pre-training), P Both the first model (PENet) and the second model (Tabular Transformer) were pre-trained using the E dataset.

[0285] After pre-training under each of the above conditions, we used each selection method and performed machine learning using 1,454 combinations to obtain a trained model. The learning rate was 10 -4 The number of epochs was set to 1,000. The optimization algorithm was based on the reference 10 (Ilya Loshchilov, et al. "Decoupled Weight Decay Regularization", [online], [checked on March 22, 2024]). Search, Internet<URL:https: / / arxiv.org / abs / 1711.05101> ) AdamW was used. The loss function L was defined by the following equation 1.

[0286]

number

[0287] [Second experiment] (A) Experiment 2A Using the PE dataset, trained models for the first to fifth comparative examples and the first to fourth examples were generated. CT image samples were used to train the model for the first comparative example. Table data was used to train the model for the second comparative example. CT image samples and table data were used to train the models for the third and fourth comparative examples. The trained model for the fifth comparative example was constructed from the trained models for the first and second comparative examples. The second condition ("with prior learning") of the first selection method in the first experiment was used to select the CT image samples in the first to fourth examples and the third and fourth comparative examples. The other conditions were set the same as in the first experiment. As a result, trained models for the first to fifth comparative examples and the first to fourth examples were obtained. A classification task was performed on 193 test combinations using the obtained trained models, and classification results for each combination were obtained. Evaluation values ​​related to accuracy were calculated from the obtained classification results.

[0288] ROC (Receiver Operating Characteristic) curves were calculated for the first to third comparative examples, the fifth comparative example, and the first embodiment. In addition, the accuracy of each part of the first embodiment was evaluated. Therefore, (i) the first model only, (ii) the second model only, (iii) the fusion model only, (iv) the ensemble of the first and second models, and (v) the ensemble of the first, second, and Each evaluation value related to accuracy was calculated from the classification results of the ensemble of the fusion model and the fusion model (first example). The evaluation values ​​used for calculation were AUPRC, AUROC, ACC, MCC, and Recall.

[0289] Furthermore, in order to visualize the classification process of the first comparative example, the third comparative example, the fourth comparative example, and the first embodiment, a 3D Grad-CAM model was designed and fitted to each model. The Grad-CAM model has , Reference 11 (Ramprasaath R. Selvaraju, et al. "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], [Retrieved March 22, 2020], Internet<URL:https: / / arxiv.org / abs / 1610.02391> ) proposed We adopted the architecture, which has been improved to be applicable to multiple inputs and multiple outputs, and obtained the Grad-CAM heat map.

[0290] (B) Experiment 2B Using the NACC dataset, trained models for the first comparative example, the second comparative example, the fifth comparative example, the sixth comparative example, the fifth example, and the sixth example were generated. Specifically, for the fifth example and the sixth example, pre-training of the first model and the second model was performed using the NACC dataset, similar to the second condition of the first experiment. After pre-training, 752 pairs of A trained model was obtained by machine learning using a combination of MRI images and table data. In the first comparative example, a trained model was obtained by machine learning using MRI images. In the second comparative example, a trained model was obtained by machine learning using table data. Note that in the models of the first and second comparative examples, the number of parameters was changed from that in Experiment 2A so as to fit the NACC dataset. In the fifth and sixth comparative examples, a trained model was obtained by machine learning using MRI images and table data. The learning rate was 10 -3 The number of epochs was set to 100. The algorithm used was Adam. 249 validation combinations were used to adjust the values ​​of each hyperparameter. The resulting trained model was then used to perform a classification task on 251 test combinations, obtaining classification results for each combination. Accuracy evaluation values ​​were calculated from the obtained classification results. The evaluation values ​​used were ACC, MCC, Recall, Jaccard coefficient, and F1 score.

[0291] [Experimental Results] FIG. 17 shows the results of Experiment 1. FIG. 18A shows the results of Experiment 2A. FIG. 18B shows the calculation results of ROC curves for each Comparative Example and each Example in Experiment 2A. FIG. 18C shows the evaluation results for each model of Example 1 in Experiment 2A. FIG. 18D shows heat maps generated by the models of each Comparative Example and Example. FIG. 19 shows the results of Experiment 2B.

[0292] 17, it was found that both the first and second selection methods produced good trained models. In addition, from the viewpoint of improving the accuracy of the trained model, it was found that the first selection method was superior under the condition of pre-learning, and the second selection method was superior under the condition of no pre-learning.

[0293] Furthermore, in the results of Experiment 2A shown in Figures 18A and 18B, each Example outperformed each Comparative Example in all indices except Recall. In Experiment 2B shown in Figure 19, each Example outperformed each Comparative Example in all indices. From these results, it was verified that the training method and inference method of the above embodiment can be expected to improve the accuracy of the trained model. In addition, when the results of the first model shown in Figure 18C and the results of the first Comparative Example shown in Figure 18A are compared, the first model of the first Example outperformed the first Comparative Example in the indices AUPRC, AUROC, and MCC. The performance of the first example exceeded that of the model of the first comparative example. Furthermore, as shown in FIG. 18D, in the generated heat map, the first example captured the location of pulmonary embolism more accurately than each of the comparative examples. These results suggest that by training the first and second models together with the fusion model, the first and second models can reflect the knowledge of the other model's data in their own training, and accordingly, improvement in inference accuracy can be expected.

[0294] This specification includes the following disclosure. [Appendix 1] The computer (1) A step of controlling machine learning of a first model (51), a second model (52), and a fusion model (53); outputting the results of the machine learning; Run the first model (51) is configured to derive an outcome of an inference task from first data (30); the second model (52) is configured to derive a result of the inference task from second data (35) of a type different from the first data (30); The fusion model (53) is configured to derive a result of the inference task from the calculated output of the first model (51) and the calculated output of the second model (52); and The machine learning performing a first training on the first model (51) so that the result of the inference task derived by the first model (51) from a first training sample (300) of the first data (30) matches a true value; performing second training on the second model (52) so that the results of the inference task derived by the second model (52) from second training samples (350) of the second data (35) conform to true values; and performing third training on the first model (51), the second model (52), and the fusion model (53) together with the first training and the second training so that a result of the inference task derived by the fusion model (53) from the calculation output of the first model (51) and the calculation output of the second model (52) when deriving a result of the inference task conforms to a true value; Including, Model generation method. [Appendix 2] the inference task is to determine whether or not a predetermined event has occurred; A plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data (30) and a second sample of the second data (35), the plurality of data sets include one or more positive data sets associated with a true value indicating the occurrence of the predetermined event; For each of the one or more positive data sets, a likelihood of occurrence of the predetermined event is calculated using a computational model configured to calculate the likelihood of occurrence of the predetermined event from the first data (30), and a likelihood of each of a plurality of first samples included in the first sample group is calculated; In the machine learning, a first sample having the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more positive datasets is used as the first training sample (300). 10. The method of generating a model according to claim 1. [Appendix 3] The plurality of data sets further includes one or more negative data sets associated with a true value indicating no occurrence of the predetermined event; For each of the one or more negative data sets, a likelihood of each of a plurality of first samples included in the first sample group is calculated using the computational model; In the machine learning, the first sample having the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more negative datasets is used as the first training sample (300). 1. A method for generating a model as described in Appendix 2. [Appendix 4] the inference task is to determine whether or not a predetermined event has occurred; A plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data (30) and a second sample of the second data (35), In the machine learning, each of a plurality of first samples included in the first sample group of each of the data sets is used as the first training sample (300), and the second sample is used as the second training sample (350); If the truth value of the inference task associated with the first sample does not match the truth value of the inference task associated with the second sample, performing the third training is omitted. 10. The method of generating a model according to claim 1. [Appendix 5] On the computer (2), A first sample (305) of the first data (30) and a first sample (305) different from the first data (30) acquiring a second sample (355) of a second data type (35); deriving a result of an inference task from the acquired first sample (305) and second sample (355) using a first model (55), a second model (56) and a fusion model (57); outputting information about the derived results of the inference task; An inference program (82) for executing the above deriving a result of the inference task, deriving a first interim result of the inference task from the first sample (305) using the first model (55); deriving a second interim result of the inference task from the second sample (355) using the second model (56); Using the fusion model (57), deriving a third interim result of the inference task from the calculation output of the first model (55) obtained in the process of deriving the first interim result and the calculation output of the second model (56) obtained in the process of deriving the second interim result; and deriving a result of the inference task according to the derived first interim result, the second interim result, and the third interim result; Including, Inference program (82). [Appendix 6] The first data (30) is sensing data obtained by a sensor, The second data (35) is other than the sensing data. The inference program (82) described in Appendix 5. [Appendix 7] The first data (30) is medical image data of the subject, The second data (35) is response data to a medical interview for the subject, The inference task includes determining whether or not the subject has a disease. The inference program (82) described in Appendix 5. [Appendix 8] The first data (30) is image data of an object or sound data related to the object, The second data (35) is work history data related to the object, the inference task includes determining whether or not an abnormality has occurred in the object; The inference program (82) described in Appendix 5. [Appendix 9] The first data (30) is image data of a target person or sound data related to the target person, The second data (35) is text data related to the subject, the inference task includes inferring an emotion of the subject; The inference program (82) described in Appendix 5. [Appendix 10] The first data (30) is first sensing data obtained by a first sensor, The second data (35) is second sensing data obtained by a second sensor of a different type from the first sensor. The inference program (82) described in Appendix 5. [Appendix 11] the first data (30) is image data of an object in a chemical plant; the second data (35) is numerical data relating to the environment of an object in the chemical plant; the inference task includes inferring a state of the object; The inference program (82) described in Appendix 5. [Appendix 12] The first data (30) is image data showing an object, the second data (35) is point cloud data obtained by measuring the object, the inference task includes inferring a position of the object; The inference program (82) described in Appendix 5. [Appendix 13] The first data (30) is image data showing the facility, The second data (35) is sound data related to the facility, the inference task includes inferring a state of the equipment; The inference program (82) described in Appendix 5. [Appendix 14] the first data (30) is image data relating to crops, the second data (35) is meteorological data in the environment of the crop; the inference task includes inferring a state of the crop; The inference program (82) described in Appendix 5. [Appendix 15] An inference device including a control unit, The control unit obtaining a first sample (305) of first data (30) and a second sample (355) of second data (35) of a type different from the first data (30); deriving a result of an inference task from the acquired first sample (305) and second sample (355) using a first model (55), a second model (56) and a fusion model (57); outputting information about the derived results of the inference task; configured to run deriving a result of the inference task, deriving a first interim result of the inference task from the first sample (305) using the first model (55); deriving a second interim result of the inference task from the second sample (355) using the second model (56); Using the fusion model (57), deriving a third interim result of the inference task from the calculation output of the first model (55) obtained in the process of deriving the first interim result and the calculation output of the second model (56) obtained in the process of deriving the second interim result; and deriving a result of the inference task according to the derived first interim result, the second interim result, and the third interim result; Including, Reasoning device (2). [Appendix 16] The computer (1) A step of controlling the machine learning of the first model (51) and the fusion model (53); outputting the results of the machine learning; Run the first model (51) is configured to derive an outcome of an inference task from first data (30); a second model (52) configured to derive a result of the inference task from second data (35) of a type different from the first data (30); The fusion model (53) is configured to derive a result of the inference task from the calculated output of the first model (51) and the calculated output of the second model (52); and The machine learning The first model (51) is generated from the first training sample (300) of the first data (30). performing a first training on the first model (51) so that the result of the inference task derived by together with the first training, training the first model (51) and the fusion model (53) so that the result of the inference task derived by the fusion model (53) from the calculation output of the second model (52) obtained by providing the second model (52) with a calculation output of the first model (51) when deriving the result of the inference task and a second training sample (350) of the second data (35) is matched to a true value; Including, Model generation method. [Explanation of symbols]

[0295] 1...Model generation device, 11...control unit, 12...storage unit, 13...communication interface, 14...input device, 15...output device, 16...drive, 81...generation program, 91...storage medium, 111...acquisition unit, 112...learning processing unit, 113...output processing unit, 2... Reasoning device, 21...control unit, 22...storage unit, 23...communication interface, 24...input device, 25...output device, 26...drive, 27...External interface, 82...inference program, 92...storage medium, 30...first data, 300...first training sample, 305...first sample, 35...second data, 350...second training sample, 355...second sample, 51·55...First model, 52·56...Second model, 53·57…Fusion model

Claims

1. The computer Controlling the machine learning of the first model, the second model, and the fusion model; outputting the results of the machine learning; Run the first model is configured to derive an outcome of an inference task from first data; the second model is configured to derive a result of the inference task from second data of a type different from the first data; the fusion model is configured to derive a result of the inference task from a computed output of the first model and a computed output of the second model; and The machine learning performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data matches a true value; performing second training on the second model so that results of the inference task derived by the second model from second training samples of the second data conform to true values; and performing third training on the first model, the second model, and the fusion model together with the first training and the second training so that a result of the inference task derived by the fusion model from the calculation output of the first model and the calculation output of the second model when deriving a result of the inference task conforms to a true value; Including, Model generation method.

2. the inference task is to determine whether or not a predetermined event has occurred; a plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data; the plurality of data sets include one or more positive data sets associated with a true value indicating the occurrence of the predetermined event; For each of the one or more positive data sets, a likelihood of occurrence of the predetermined event is calculated using a computational model configured to calculate the likelihood of occurrence of the predetermined event from the first data, and a likelihood of each of a plurality of first samples included in the first sample group is calculated; In the machine learning, a first sample having the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more positive datasets is used as the first training sample. The model generation method of claim 1 .

3. The plurality of data sets further includes one or more negative data sets associated with a true value indicating no occurrence of the predetermined event; For each of the one or more negative data sets, a likelihood of each of a plurality of first samples included in the first sample group is calculated using the computational model; In the machine learning, a first sample having the highest calculated likelihood among the plurality of first samples included in the first sample group of each of the one or more negative datasets is used as the first training sample. The model generation method according to claim 2 .

4. the inference task is to determine whether or not a predetermined event has occurred; a plurality of data sets are provided, each of which is composed of a combination of a first sample group of the first data and a second sample of the second data; In the machine learning, each of a plurality of first samples included in the first sample group of each of the data sets is used as the first training sample, and the second sample is used as the second training sample; If the truth value of the inference task associated with the first sample does not match the truth value of the inference task associated with the second sample, performing the third training is omitted. The model generation method of claim 1 .

5. On the computer, obtaining a first sample of first data and a second sample of second data of a different type than the first data; deriving a result of an inference task from the acquired first and second samples using a first model, a second model, and a fusion model; outputting information about the derived results of the inference task; An inference program for executing deriving a result of the inference task, deriving a first interim result of the inference task from the first sample using the first model; deriving a second interim result of the inference task from the second sample using the second model; Using the fusion model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result; and deriving a result of the inference task according to the derived first interim result, the second interim result, and the third interim result; Including, Inference program.

6. the first data is sensing data obtained by a sensor, the second data is other than the sensing data; The inference program according to claim 5 .

7. the first data is medical image data of the subject; The second data is response data to a medical interview conducted for the subject, The inference task includes determining whether or not the subject has a disease. The inference program according to claim 5 .

8. the first data is image data of an object or sound data relating to the object, the second data is work history data related to the object, the inference task includes determining whether or not an abnormality has occurred in the object; The inference program according to claim 5 .

9. The first data is image data of a target person or sound data related to the target person, the second data is text data related to the subject; the inference task includes inferring an emotion of the subject; The inference program according to claim 5 .

10. the first data is first sensing data obtained by a first sensor, the second data is second sensing data obtained by a second sensor of a different type from the first sensor; The inference program according to claim 5 .

11. the first data is image data of an object in a chemical plant; the second data is numerical data relating to an environment of an object in the chemical plant, the inference task includes inferring a state of the object; The inference program according to claim 5 .

12. the first data is image data showing an object, the second data is point cloud data obtained by measuring the object, the inference task includes inferring a position of the object; The inference program according to claim 5 .

13. The first data is image data showing the facility, the second data is sound data related to the facility, the inference task includes inferring a state of the equipment; The inference program according to claim 5 .

14. the first data is image data relating to a crop, the second data is meteorological data in an environment of the crop; the inference task includes inferring a state of the crop; The inference program according to claim 5 .

15. An inference device including a control unit, The control unit obtaining a first sample of first data and a second sample of second data of a different type than the first data; deriving a result of an inference task from the acquired first and second samples using a first model, a second model, and a fusion model; outputting information about the derived results of the inference task; configured to run deriving a result of the inference task, deriving a first interim result of the inference task from the first sample using the first model; deriving a second interim result of the inference task from the second sample using the second model; Using the fusion model, deriving a third interim result of the inference task from a calculation output of the first model obtained in the process of deriving the first interim result and a calculation output of the second model obtained in the process of deriving the second interim result; and deriving a result of the inference task according to the derived first interim result, the second interim result, and the third interim result; Including, Reasoning device.

16. The computer Controlling the machine learning of the first model and the fused model; outputting the results of the machine learning; Run the first model is configured to derive an outcome of an inference task from first data; a second model is provided that is configured to derive a result of the inference task from second data of a type different from the first data; the fusion model is configured to derive a result of the inference task from a computed output of the first model and a computed output of the second model; and The machine learning performing a first training of the first model so that a result of the inference task derived by the first model from a first training sample of the first data matches a true value; and together with the first training, training the first model and the fusion model so that the result of the inference task derived by the fusion model conforms to a true value from the calculation output of the first model when deriving the result of the inference task and the calculation output of the second model obtained by providing the second training sample of the second data to the second model; Including, Model generation method.

Images