Information processing device, biological information estimation device, information processing method, and recording medium

Abstract

An information processing device 1 comprises: an acquisition unit 11 that acquires a partial modal set IMS including at least a first type of modal 1M among a plurality of types of modals and not including a second type of modal 2M among the plurality of types of modals; a modal output unit 13 that inputs a partial modal set IGM to a modal generation model GM, which when receiving input of at least one of the plurality of types of modals, outputs at least one of the plurality of types of modals and which is generated by machine learning using a full modal set MMS including the plurality of types of modals, so as to output at least a second type of modal 2M; and a modal generation unit 12 that generates a generated full modal set GMS including the partial modal set IMS and the second type of modal 2M output by the modal generation model GM.

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of an information processing device, a biometric information estimation device, an information processing method, and a recording medium. [Background technology]

[0002] Patent Document 1 describes an example of a device that detects heart rate and respiratory rate from a face image. Patent Document 2 describes an example of a system that determines a subject's stress coping style based on a face image and two or more of mean blood pressure, heart rate, cardiac output, and total peripheral resistance. Patent Document 3 describes an example of a function that re-learns an algorithm based on newly obtained learning data. Patent Document 4 describes a method of automatically re-learning a voice recognition system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-239661 A [Patent Document 2] Patent Publication No. 2021-037287 [Patent Document 3] International Publication No. 2021 / 075288 [Patent Document 4] Special Publication No. 2003-524805 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing device, a biometric information estimation device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents. [Means for solving the problem]

[0005] One aspect of an information processing device includes an acquisition means for acquiring a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals, an output means for inputting the first modal set into a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set including the plurality of types of modals, thereby outputting at least the second type of modal, and a modal generation means for generating a third modal set including the first modal set and the second type of modal output by the modal generation model.

[0006] One aspect of the biometric information estimation device includes an image acquisition means for acquiring a facial image of a subject, and a biometric information estimation means for performing machine learning using a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals, and a third modal set including the second type of modal output by a modal generation model, thereby causing a modal estimation model to output the biometric information of the subject as the output modal, the modal generation model outputting at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, and the modal generation model including the plurality of types of modals outputs at least one of the plurality of types of modals.

[0007] One aspect of the information processing method includes obtaining a first modal set that includes at least a first type of modal among a plurality of types of modals and does not include a second type of modal among the plurality of types of modals, and inputting the first modal set into a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set that includes the plurality of types of modals, thereby outputting at least the second type of modal, and generating a third modal set that includes the first modal set and the second type of modal output by the modal generation model.

[0008] In one embodiment of the recording medium, a computer program is recorded on a computer to execute an information processing method for acquiring a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals, inputting the first modal set into a modal generation model generated by machine learning using a second modal set including the plurality of types of modals, the first modal set being output to at least the second type of modal, and generating a third modal set including the first modal set and the second type of modal output by the modal generation model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of an information processing device in the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of an information processing device according to the second embodiment. [Diagram 3] FIG. 3 shows an example of the data structure of a data group used in the second embodiment. [Figure 4]FIG. 4 is a flowchart showing the flow of a modal set generating operation performed by the information processing device in the second embodiment. [Diagram 5] FIG. 5 is a block diagram showing the configuration of the modal generating unit 212 included in the information processing device in the third embodiment. [Figure 6] FIG. 6 is a block diagram showing the configuration of an information processing device according to the fourth embodiment. [Figure 7] FIG. 7 is a flowchart showing a flow of a modal estimation model generating operation performed by an information processing device in the fourth embodiment. [Figure 8] FIG. 8 is a block diagram showing a configuration of an online diagnosis support system including a diagnosis support device according to the fifth embodiment. [Figure 9] FIG. 9 is a block diagram showing a configuration of a biological information estimation unit included in a diagnosis support apparatus according to the fifth embodiment. [Figure 10] FIG. 10 is a diagram showing a flow of an online diagnosis support operation performed by an online diagnosis support system including a diagnosis support device in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of an information processing device, a biometric information estimation device, an information processing method, and a recording medium will be described with reference to the drawings. (1) First embodiment

[0011] First, a first embodiment of an information processing device, an information processing method, and a recording medium will be described. In the following, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 to which the first embodiment of the information processing device, the information processing method, and the recording medium is applied. (1-1) Configuration of Information Processing Device 1

[0012] 1 is a block diagram showing the configuration of an information processing device 1 in the first embodiment. As shown in FIG. 1, the information processing device 1 includes an acquisition unit 11 and a modal generation unit 12.

[0013] First, modal may refer to biometric information. Furthermore, modal set may refer to data including one or more types of modal related to an individual. For example, assume that data including three types of modal, modal A, modal B, and modal C, related to an individual is required. In this case, in the first embodiment, data including all three types of modal, modal A, modal B, and modal C, is called a full modal set. On the other hand, data including parts of modal A, modal B, and modal C is called a partial modal set. The partial modal set may be any of data including modal A and modal B, data including modal A and modal C, data including modal B and modal C, data including modal A, data including modal B, and data including modal C. The acquisition unit 11 acquires a partial modal set IMS as a first modal set. The partial modal set IMS includes at least a first type modal 1M among a plurality of types of modals, but does not include a second type modal 2M among the plurality of types of modals.

[0014] The modal output unit 13 inputs the partial modal set IMS to the modal generation model, thereby causing the model to output at least a second type of modal. The modal generator 12 generates a generated total modal set GMS as a third modal set including the partial modal set IMS and the second type of modal output by the modal generative model. The generated total modal set GMS is a total modal set including all types of modals. The modal generative model outputs at least one of a plurality of types of modals when at least one of a plurality of types of modals is input. The modal generative model is a model generated by machine learning using a full modal set MMS as a second modal set including all types of modals. (1-2) Technical Effects of Information Processing Device 1

[0015] The information processing device 1 in the first embodiment outputs at least the second type of modal, that is, the modal not included in the partial modal set IMS, from the modal generation model. This allows the modal generation unit 12 to generate a total modal set including multiple types of modals. In other words, it is possible to generate a total modal set including multiple types of modals, which is relatively difficult to collect, by utilizing a partial modal set that does not include some types of modals, which is relatively easy to collect. Therefore, the information processing device 1 in the first embodiment can relatively easily obtain a total modal set including a large number of multiple types of modals. The total modal set including a large number of multiple types of modals can be used for machine learning of a model that realizes highly accurate estimation, and can contribute to realizing highly accurate estimation of modals. (2) Second embodiment

[0016] Next, a second embodiment of the information processing device, the information processing method, and the recording medium will be described. In the following, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 to which the second embodiment of the information processing device, the information processing method, and the recording medium is applied. (2-1) Configuration of information processing device 2

[0017] First, the configuration of the information processing device 2 in the second embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the information processing device 2 in the second embodiment. In the following description, components that have already been described are given the same reference symbols, and detailed description thereof will be omitted.

[0018] 2, the information processing device 2 includes a calculation device 21 and a storage device 22. Furthermore, the information processing device 2 may include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0019] The arithmetic device 21 includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable and non-transient recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the drawings that is provided in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) a computer program from a device (not shown) disposed outside the information processing device 2 via a communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, a logical function block for executing the operation to be performed by the information processing device 2 is realized in the arithmetic device 21. That is, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processes) that the information processing device 2 should perform.

[0020] Fig. 2 shows an example of logical functional blocks realized in the arithmetic device 21 to execute the modal generation operation. As shown in Fig. 2, an acquisition unit 211 which is a specific example of "acquisition means", a modal generation unit 212 which is a specific example of "modal generation means", a modal output unit 213 which is a specific example of "output means", and a first model generation unit 214 which is a specific example of "first model generation means" are realized in the arithmetic device 21. The first model generation unit 214 generates a modal generation model GM by performing machine learning using the entire modal set MMS as the second modal set.

[0021] The operations of the acquisition unit 211, the modal generation unit 212, the modal output unit 213, and the first model generation unit 214 will be described in detail later with reference to Fig. 4. However, the calculation device 21 does not necessarily have to include the first model generation unit 214.

[0022] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores for a long period of time. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0023] The storage device 22 may store the total modal set MMS, the partial modal set IMS, the generated total modal set GMS, and the modal generative model GM. Details of the total modal set MMS and the partial modal set IMS will be described later with reference to Fig. 3. However, the storage device 22 does not have to store the total modal set MMS, the partial modal set IMS, the generated total modal set GMS, and the modal generative model GM. Here, the total modal set MMS is a modal set that includes all types of modals.

[0024] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown).

[0025] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0026] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (so-called a display) capable of displaying an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (so-called a speaker) capable of outputting sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (so-called a printer) capable of printing desired information on paper.

[0027] There is a high demand for estimating modals with high accuracy using a model generated by machine learning. In order to generate a model that realizes highly accurate estimation by machine learning, a training data group containing a large amount of training data is required. As the training data, it is preferable to use a full modal set including multiple types of required modals (e.g., three types: face image, heart rate, and oxygen saturation). However, it is generally difficult to collect a training data group containing a large amount of full modal sets including multiple types of required modals (e.g., three types: face image, heart rate, and oxygen saturation). In addition, as the number of types of modals to be estimated increases, the number of types of modals to be collected also increases, so the cost of collecting training data increases. On the other hand, it is easier to collect a partial modal set in which some types of modals are missing compared to collecting a full modal set including all types of modals. For example, a complete modal set including three modal types, namely, a facial image, heart rate, and oxygen saturation, for an individual can be counted as one sample, and a large complete modal set may refer to a group of complete modal sets including, for example, several thousand to more than 10,000 samples.

[0028] In the second embodiment, the multiple types of modalities may include, for example, a face image, a heart rate, and an oxygen saturation level. Each of Fig. 3 shows an example of the data structure of a data group used in the second embodiment.

[0029] 3A shows the data structure of a learning data group used by the first model generation unit 214 for machine learning of the modal generative model GM. The learning data group may include, for example, 2,000 total modal sets MMS. In the second embodiment, the total modal set MMS may include a face image, a heart rate, and an oxygen saturation level. Note that, in the second embodiment, each modal included in the total modal set MMS may be data obtained by actual measurement.

[0030] FIG. 3B shows the data structure of the first input data group that the modal output unit 213 inputs to the modal generation model GM. The first input data group may include, for example, 4,000 first partial modal sets IMS#1. The partial modal set IMS includes at least a first type of modal 1M among the multiple types of modals, but does not include a second type of modal 2M among the multiple types of modals. In the second embodiment, the first type of modal 1M in the first partial modal set IMS#1 may be a face image and a heart rate, and the first partial modal set IMS#1 may include a face image and a heart rate. In addition, the second type of modal 2M may be oxygen saturation. Since face images and heart rates are often acquired simultaneously, it is relatively easy to acquire a large number of partial modal sets of combinations of face images and heart rates.

[0031] FIG. 3(C) shows the data structure of the second input data group that the modal output unit 213 inputs to the modal generation model GM. The second input data group may include, for example, 4,000 second partial modal sets IMS#2. In the second embodiment, the first type of modal 1M in the second partial modal set IMS#2 may be a heart rate and an oxygen saturation, and the second partial modal set IMS#2 may include a heart rate and an oxygen saturation. In addition, the second type of modal 2M may be a face image. Compared with a face image, the data of the heart rate and the oxygen saturation has a small data capacity. In addition, while it is difficult to collect face images from the viewpoint of privacy, it is relatively easy to collect the heart rate and the oxygen saturation. Therefore, it is relatively easy to obtain a large amount of partial modal sets of combinations of the heart rate and the oxygen saturation. In the second embodiment, each modal included in the partial modal set IMS may be data obtained by actual measurement.

[0032] The modal generation unit 212 inputs the first input data group and the second input data group into the modal generative model GM, and can generate a data group including 8,000 generated total modal sets GMS. The generated total modal set GMS is a modal set including a plurality of types of modals including the partial modal set IMS and the second type of modal 2M output by the modal generative model GM. Therefore, the data structure of the generated total modal set GMS is the same as the data structure shown in FIG. 3(A). That is, in the case of the above example, a learning data group including a total modal set including 10,000 multiple types of modals can be collected by combining the 2,000 total modal sets MMS and the 8,000 generated total modal sets GMS. (2-2) Modal Set Generation Operation Performed by Information Processing Device 2

[0033] Next, a modal set generating operation performed by the information processing device 2 in the second embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of the modal set generating operation performed by the information processing device 2 in the second embodiment.

[0034] 4, the acquisition unit 211 acquires a full modal set MMS (step S21). In the second embodiment, the full modal set MMS may include a face image, a heart rate, and an oxygen saturation level, and may be, for example, 2,000 items. The first model generation unit 214 generates a modal generative model GM by performing machine learning using the full modal set MMS acquired in step S21 (step S22).

[0035] The modal generative model GM is a model capable of outputting at least one of a plurality of types of modals when at least one of a plurality of types of modals is input. When the total modal set MMS used to generate the modal generative model GM includes a face image, a heart rate, and an oxygen saturation level as described above, the first model generation unit 214 may generate the modal generative model GM capable of outputting at least one of a face image, a heart rate, and an oxygen saturation level when at least one of a face image, a heart rate, and an oxygen saturation level is input by machine learning. Specifically, the first model generation unit 214 may generate the modal generative model GM by adjusting the parameters of the modal generative model GM so that a loss function set based on at least one of the error between the face image output by the modal generative model GM and the face image included in the total modal set MMS, the error between the heart rate output by the modal generative model GM and the heart rate included in the total modal set MMS, and the error between the oxygen saturation output by the modal generative model GM and the oxygen saturation included in the total modal set MMS becomes small (preferably, becomes minimum).

[0036] Next, the acquisition unit 211 acquires the partial modal set IMS (step S23). In the second embodiment, the partial modal set IMS may include, for example, 4,000 first partial modal sets IMS#1 including face images and heart rates, and 4,000 second partial modal sets IMS#2 including heart rates and oxygen saturation. The modal output unit 213 inputs the partial modal set IMS to the modal generative model GM (step S24). The modal output unit 213 causes the modal generative model GM to output at least a second type of modal 2M (step S25). The modal generative model GM may output oxygen saturation in response to the input of the partial modal set IMS#1, and may output information indicating a face image in response to the input of the partial modal set IMS#2. The modal generation unit 212 generates, for example, 8,000 generated total modal sets GMS including the partial modal set IMS and the second type of modal 2M output by the modal generation model GM (step S26).

[0037] When a face image and a heart rate are input as the partial modal set IMS, the modal generation model GM may output at least one of the face image and the heart rate as well as the oxygen saturation as the second type of modal 2M. In this case, at least one of the face image and the heart rate has two types: a modal input to the modal generation model GM and a modal output from the modal generation model GM. In the second embodiment, the modal generation unit 212 generates a generated total modal set GMS including the partial modal set IMS and the second type of modal 2M. However, the modal generation unit 212 may generate a generated total modal set GMS including a modal output from the modal generation model GM and a type of modal included in the partial modal set IMS that was not output by the modal generation model GM. For example, when the first type of modal 1M is a face image and a heart rate, the second type of modal 2M is an oxygen saturation, and the modal generative model GM outputs a face image and an oxygen saturation, the modal generation unit 212 may generate a generated all modal set GMS including the face image and the oxygen saturation output by the modal generative model GM, and the heart rate included in the partial modal set IMS (the heart rate is included in the partial modal set IMS and is a type of modal not output by the modal generative model GM). The modal generation unit 212 may obtain a composite face image by combining the face image included in the partial modal set IMS with the face image output by the modal generative model GM, and generate a generated all modal set GMS including the composite face image, the heart rate included in the partial modal set IMS, and the oxygen saturation output by the modal generative model GM. As a result, the number of generated all modal sets GMS that can be generated using the modal generative model GM increases.

[0038] The modals included in the partial modal set IMS may be reliable data obtained by actual measurement, and it is considered preferable that the modal generator 212 generates the generated total modal set GMS using the modals included in the partial modal set IMS. On the other hand, there is a possibility that the modals output from the modal generative model GM have noise removed, and it is considered preferable that the modal generator 212 generates the generated total modal set GMS using the second type of modal 2M output from the modal generative model GM.

[0039] Alternatively, in the second embodiment, since the aim is to collect a large number of modal sets, three types of generated all modal sets GMS may be generated: a first generated all modal set GMS using the modals included in the partial modal set IMS, a second generated all modal set GMS using the modals output from the modal generative model GM, and a third generated all modal set GMS using a composite modal that is a composite of both. Specifically, for example, consider a case where a facial image IA and a heart rate IB are input as the partial modal set IMS, and the modal generation model GM outputs oxygen saturation OC as a second type of modal 2M, as well as a facial image OA and a heart rate OB. Here, a modal obtained by combining facial images IA and OA is called a facial image CA, and a modal obtained by combining heart rates IB and OB is called a heart rate CB. In this case, the modal generation unit 212 generates a generated all-modal set GMS1 including the facial image IA, the heart rate IB, and the oxygen saturation OC, a generated all-modal set GMS2 including the facial image IA, the heart rate OB, and the oxygen saturation OC, a generated all-modal set GMS3 including the facial image OA, the heart rate IB, and the oxygen saturation OC, a generated all-modal set GMS4 including the facial image OA, the heart rate OB, and the oxygen saturation OC, and a generated all-modal set GMS5 including the facial image OA, the heart rate OB, and the oxygen saturation OC. Nine types of generated all-modal sets GMS can be generated: generated all-modal set GMS5, generated all-modal set GMS6 including a facial image CA, heart rate IB, and oxygen saturation OC, generated all-modal set GMS7 including a facial image CA, heart rate OB, and oxygen saturation OC, generated all-modal set GMS8 including a facial image IA, heart rate CB, and oxygen saturation OC, and generated all-modal set GMS9 including a facial image OA, heart rate CB, and oxygen saturation OC.

[0040] In the second embodiment, an example in which a face image and a heart rate are input to the modal generation model GM and at least an oxygen saturation level is output, and an example in which a heart rate and an oxygen saturation level are input to the modal generation model GM and at least a face image is output have been described, but the input and output of modal data to the modal generation model GM are not limited to these two types. For example, a face image and an oxygen saturation level may be input to the modal generation model GM and at least a heart rate may be output, a face image may be input to the modal generation model GM and at least a heart rate and an oxygen saturation level may be output, a heart rate may be input to the modal generation model GM and at least a face image and an oxygen saturation level may be output, or an oxygen saturation level may be input to the modal generation model GM and at least a face image and an oxygen saturation level may be input to the modal generation model GM and at least a face image and a heart rate may be output. (2-3) Technical Effects of Information Processing Device 2

[0041] The information processing device 2 in the second embodiment can generate a modal generation model that estimates and outputs modals other than the input modal with a certain degree of accuracy by performing machine learning using a relatively small amount of the total modal set MMS. The information processing device 2 in the second embodiment can acquire types of modals not included in the partial modal set IMS using the modal generation model. In other words, even if it is difficult to collect a large amount of the total modal set MMS, if it is possible to collect a large amount of the partial modal set IMS, the information processing device 2 in the second embodiment can generate a large amount of generated total modal sets GMS including multiple types of modals using the acquired modals. For example, suppose that a total modal set including three types of modals, namely, face image, heart rate, and oxygen saturation, is required. Even if it is difficult to collect a large number of total modal sets, it may be possible to collect a large number of partial modal sets, such as a partial modal set including heart rate and oxygen saturation, a partial modal set including face image and heart rate, etc. In such a case, the information processing device 2 in the second embodiment can obtain modals of types not included in the partial modal set by using a modal generation model, and generate a large number of total modal sets including three types of modals, namely, face image, heart rate, and oxygen saturation. This full modal set, which contains a large number of different types of modals, can be used for machine learning of models that achieve highly accurate estimation, thereby contributing to the realization of highly accurate modal estimation.

[0042] In addition, when the number of types of modals to be estimated increases, it is often more difficult to collect learning data. For example, it is often more difficult to collect a total modal set with four types of modals (A, B, C, D) than to collect a total modal set with three types of modals (A, B, C). The information processing device 2 in the second embodiment can generate and acquire the increased types of modal D by using the generated modal generation model GM, so that it is relatively easy to collect a total modal set with four types of modals (A, B, C, D). Therefore, the information processing device 2 in the second embodiment can collect a large number of total modal sets relatively easily even when the number of types of modals increases. And, the large number of total modal sets can be utilized for machine learning of a model that realizes highly accurate estimation, so that it can contribute to realizing highly accurate estimation of various types of modals. (3) Third embodiment

[0043] Next, a third embodiment of the information processing device, the information processing method, and the recording medium will be described. In the following, the third embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device to which the third embodiment of the information processing device, the information processing method, and the recording medium is applied.

[0044] The information processing device in the third embodiment may have the same configuration as the information processing device 2 in the second embodiment described above. The information processing device in the third embodiment is different from the information processing device 2 in the second embodiment in that the modal output unit 213 inputs information to the modal generation model GM. Other features of the information processing device may be the same as other features of the information processing device 2. (3-1) Configuration of the modal generation unit 212

[0045] FIG. 5(A) is a block diagram showing the configuration of the modal generation unit 212. The modal generation unit 212 may include at least an input value generation unit 312 and a modal generation model GM. When at least one of the multiple types of modals is input, the input value generation unit 312 may generate an input value of at least one of the multiple types of modals. The input of at least one of the multiple types of modals to the modal generation model GM may be an input of an input value of at least one of the multiple types of modals. The input value may be, for example, a feature value of the modal. In this case, the input value generation unit 312 may be any mechanism capable of extracting a feature value of a modal, such as a mechanism utilizing a neural network.

[0046] The modal generation model GM may include at least an encoder unit GME and a decoder unit GMD. The encoder unit GME may compress an input value of at least one of the multiple types of modals when at least one of the multiple types of modals is input. The decoder unit GMD may expand the compressed input value to generate at least one of the multiple types of modals.

[0047] The modal output unit 213 in the third embodiment may input, to the modal generative model GM, an environment label EL indicating the acquisition environment when the first type of modal 1M included in the partial modal set IMS was acquired together with the partial modal set IMS, thereby causing the modal generative model GM to output the second type of modal 2M. By inputting the environment label EL to the modal output unit 213, it is possible to prevent the acquisition environment of the first type of modal 1M from adversely affecting the estimation of the second type of modal 2M by the modal generative model GM. The modal generation unit 212 generates a generated total modal set GMS including the partial modal set IMS and the second type of modal 2M output by the modal generative model GM. The modal output unit 213 may input the environment label EL to at least one of the encoder unit GME and the decoder unit GMD. That is, as shown in Fig. 5(B), the modal output unit 213 may input the environment label EL to the decoder unit GMD. Also, as shown in Fig. 5(C), the modal output unit 213 may input the environment label EL to both the encoder unit GME and the decoder unit GMD. Also, as shown in Fig. 5(D), the modal output unit 213 may input the environment label EL to the encoder unit GME.

[0048] The environmental label EL is preferably information that provides information that contributes to the estimation accuracy. Examples of the environmental label EL include information that is known at the time of modal acquisition, such as information fixed to the device that acquires the modal, information on the subject, etc. For example, information such as the characteristics of the camera that captured the face image, lighting conditions, the gender, age, and skin color of the subject, and the type of movement (face direction, etc.) at the time of modal acquisition can be included. (3-2) Technical Effects of Information Processing Device 3

[0049] The information processing device 3 in the third embodiment explicitly inputs environmental information indicating the acquisition environment when the input modal was acquired together with the partial modal set IMS, so that the modal generation model GM can estimate the second type of modal 2M with higher accuracy than when the environmental information is not input. Therefore, the information processing device 3 in the third embodiment can generate a more realistic generated all modal set GMS including the accurately estimated second type of modal 2M. In addition, the modal output unit 213 inputs environmental information to at least one of the encoder unit GME and the decoder unit GMD, so that the estimation accuracy of the output second type of modal 2M can be controlled. Then, the information processing device 3 in the third embodiment can control the authenticity of the generated all modal set GMS. When the modal output unit 213 inputs environmental information to the encoder unit, there is a possibility that the influence of the environmental information contributing to the estimation accuracy will be reduced due to the compression process by the encoder unit. Therefore, it is expected that the modal output unit 213 inputs environmental information to the decoder unit, so that the influence of the environmental information contributing to the estimation accuracy can be prevented from being reduced. The more realistic generated full modal set GMS can be utilized in machine learning of models that realize highly accurate estimation, and can contribute to the realization of highly accurate modal estimation. (4) Fourth embodiment

[0050] Next, a fourth embodiment of the information processing device, the information processing method, and the recording medium will be described. In the following, the fourth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 4 to which the fourth embodiment of the information processing device, the information processing method, and the recording medium is applied. (4-1) Configuration of information processing device 4

[0051] Hereinafter, the configuration of the information processing device 4 in the fourth embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the information processing device 4 in the fourth embodiment. In the following description, components that have already been described are given the same reference symbols, and detailed description thereof will be omitted.

[0052] 6, the information processing device 4 in the fourth embodiment differs from the information processing device 2 in the second embodiment (or from the information processing device in the third embodiment) in that the arithmetic device 21 includes a second model generation unit 414. Other features of the information processing device 4 may be the same as other features of the information processing device 2 (or the information processing device in the third embodiment).

[0053] The second model generation unit 414 can perform machine learning using the generated total modal set GMS to generate a modal estimation model EM that outputs a fourth type of modal 4M different from the one type of the multiple types of modals as an output modal when a third type of modal 3M, which is one type of the multiple types of modals, is input as an input modal. The storage device 22 may store the generated modal estimation model EM. The modal estimation model EM generated in the fourth embodiment may be, for example, a learning model that outputs a heart rate and an oxygen saturation level when a face image is input. (4-2) Modal Estimation Model EM Generation Operation Performed by Information Processing Device 4

[0054] Next, a modal estimation model EM generating operation performed by the information processing device 4 in the fourth embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of the modal estimation model EM generating operation performed by the information processing device 4 in the fourth embodiment.

[0055] 7, the acquiring unit 211 acquires a training data group (step S41). The training data group is a total modal set including a plurality of types of modals, and may include a total modal set MMS and a generated total modal set GMS. The second model generating unit 414 generates a modal estimation model EM by performing machine learning using the training data group acquired in step S41 (step S42).

[0056] In the fourth embodiment, the first model generation unit 214 and the second model generation unit 414 are described as separate configurations, and the modal generative model GM and the modal estimation model EM are described as separate models, but the first model generation unit 214 and the second model generation unit 414 may be the same configuration, and the modal generative model GM and the modal estimation model EM may be the same model. That is, the control mechanism (the first model generation unit 214 and the second model generation unit 414) may generate a learning model (the modal generative model GM and the modal estimation model EM) by performing machine learning using the entire modal set MMS, and may re-learn the learning model by performing machine learning using the generated entire modal set GMS, thereby completing a learning model capable of highly accurate estimation.

[0057] In addition to the heart rate and oxygen saturation, the modalities that can be estimated may include, for example, respiratory rate, stress level, blood pressure value, cardiac output, total peripheral resistance, pulse, electrocardiogram, body temperature, body hydration, alcohol concentration, lactate value, blood glucose level, muscle activity, gaze movement, brain activity, and consciousness level, etc. Furthermore, the modalities that can be estimated may be, for example, respiratory rate, stress level, blood pressure value, cardiac output, total peripheral resistance, pulse, electrocardiogram, body temperature, body hydration, alcohol concentration, lactate value, blood glucose level, muscle activity, gaze movement, brain activity, and consciousness level, etc., other than the heart rate and oxygen saturation.

[0058] In addition, it is expected that the accuracy of modal estimation will be higher for subjects for whom a modal set used in machine learning has been collected than for subjects for whom a modal set used in machine learning has not been collected (hereinafter referred to as a "new subject"). Therefore, when estimating the modal of a new subject, a modal set for the new subject may be collected in advance and the modal estimation model EM may be retrained. Furthermore, a modal set with all modals may be generated using a modal set for the new subject that is missing some types of modals, and then the modal estimation model EM may be retrained. This makes it possible to obtain a model that can estimate the modal of a new subject with higher accuracy. (4-3) Technical Effects of Information Processing Device 4

[0059] It is expected that models generated by machine learning will be able to estimate modalities with high accuracy, especially when there are changes in the environment, such as changes in the subject's movements or the shooting environment, etc. It is also expected that machine learning using a large amount of training data will be able to generate models that can achieve more accurate estimation than machine learning using a small amount of training data.

[0060] The information processing device 4 in the fourth embodiment generates a modal generation model using a large amount of learning data, and therefore can obtain a modal estimation model EM that allows for more accurate estimation, thereby achieving highly accurate modal estimation. (5) Fifth embodiment

[0061] Next, a fifth embodiment of the biological information estimation device will be described. In the following, the fifth embodiment of the biological information estimation device will be described using an online diagnosis support system 500 including a diagnosis support device 50 to which the fifth embodiment of the biological information estimation device is applied. (5-1) Configuration of online diagnostic support system 500

[0062] First, the configuration of an online diagnosis support system 500 in the fifth embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the online diagnosis support system 500 in the fifth embodiment. As shown in Fig. 8, the online diagnosis support system 500 includes a diagnosis support device 50 and a plurality of terminal devices 60#1, 60#2, ..., 60#N (when there is no need to distinguish between them, they will simply be referred to as "60").

[0063] The diagnosis support device 50 may be, for example, a device used by a doctor for diagnosis. The terminal device 60 may be, for example, a device used by a patient in a remote location. The terminal device 60 may be equipped with an image generating device 61 that generates a face image by capturing an image of a subject. (5-2) Configuration of diagnosis support device 50

[0064] 8, the diagnosis support device 50 includes a calculation device 51 and a storage device 52. The diagnosis support device 50 may further include a communication device 53, an input device 54, and an output device 55. However, the diagnosis support device 50 does not have to include at least one of the communication device 53, the input device 54, and the output device 55. The calculation device 51, the storage device 52, the communication device 53, the input device 54, and the output device 55 may be connected via a data bus 56.

[0065] The arithmetic device 51 includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 51 reads a computer program. For example, the arithmetic device 51 may read a computer program stored in the storage device 52. For example, the arithmetic device 51 may read a computer program stored in a computer-readable and non-transient recording medium using a recording medium reading device (e.g., an input device 54 described later) not shown in the figure that is provided in the diagnosis support device 50. The arithmetic device 51 may acquire (i.e., download or read) a computer program from a device (not shown) disposed outside the diagnosis support device 50 via the communication device 53 (or other communication device). The arithmetic device 51 executes the read computer program. As a result, a logical function block for executing the operation to be performed by the diagnosis support device 50 is realized in the arithmetic device 51. That is, the arithmetic device 51 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) that the diagnosis support device 50 should perform.

[0066] Fig. 8 shows an example of logical functional blocks realized in the arithmetic device 51 for executing the biometric information estimation operation. As shown in Fig. 8, an image acquisition unit 515, which is a specific example of "image acquisition means", and a biometric information estimation unit 516, which is a specific example of "biometric information estimation means", are realized in the arithmetic device 51. (5-3) Estimation of behavior using modal estimation model EM

[0067] 9, the biometric information estimation unit 516 may include at least an input value generation unit 312 and a modal estimation model EM. When a face image is input as an input modal, the input value generation unit 312 may generate an input value of the face image. The input value generation unit 312 inputs the input value of the face image to the modal estimation model EM, and the biometric information estimation unit 516 can cause the modal estimation model EM to output the subject's biometric information as an output modal.

[0068] The operations of the image acquisition unit 515 and the biometric information estimation unit 516 will be described in detail later with reference to FIG.

[0069] The storage device 52 can store desired data. For example, the storage device 52 may temporarily store a computer program executed by the arithmetic device 51. The storage device 52 may temporarily store data that the arithmetic device 51 temporarily uses when the arithmetic device 51 is executing a computer program. The storage device 52 may store data that the diagnosis support device 50 stores for a long period of time. The storage device 52 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 52 may include a non-temporary recording medium.

[0070] The storage device 52 may store the modal estimation model EM. However, the storage device 52 does not necessarily have to store the modal estimation model EM.

[0071] The communication device 53 is capable of communicating with devices external to the diagnosis support device 50 via a communication network (not shown). The diagnosis support device 50 may be connected to a terminal device 60 via the communication device 53.

[0072] The input device 54 is a device that accepts information input to the diagnostic support device 50 from outside the diagnostic support device 50. For example, the input device 54 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the diagnostic support device 50. For example, the input device 54 may include a reading device that can read information recorded as data on a recording medium that can be attached externally to the diagnostic support device 50.

[0073] The output device 55 is a device that outputs information to the outside of the diagnosis support device 50. For example, the output device 55 may output information as an image. That is, the output device 55 may include a display device (so-called a display) capable of displaying an image showing the information to be output. For example, the output device 55 may output information as sound. That is, the output device 55 may include an audio device (so-called a speaker) capable of outputting sound. For example, the output device 55 may output information on paper. That is, the output device 55 may include a printing device (so-called a printer) capable of printing desired information on paper. (5-2) Online diagnosis support operation performed by the online diagnosis support system 500

[0074] Next, an online diagnosis support operation performed by the online diagnosis support system 500 in the fifth embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the online diagnosis support operation performed by the online diagnosis support system 500 in the fifth embodiment.

[0075] As shown in FIG. 10, the image generating device 61 acquires a face image of a patient as a subject (step S51), and the terminal device 60 transmits the face image to the diagnosis support device 50.

[0076] Next, the image acquisition unit 515 acquires a face image of the subject from the terminal device 60 via the communication line (step S53). The biometric information estimation unit 516 inputs the face image to the modal estimation model EM as a third type of modal 3M, thereby causing the modal estimation model EM to estimate the biometric information of the subject as a third type of modal 4M (step S54). As the modal estimation model EM, for example, a model generated by the information processing device 4 in the fourth embodiment may be used. Furthermore, the biometric information may include information on at least one of the heart rate and the oxygen saturation level.

[0077] Next, for example, the output device 55 notifies the user of the diagnosis support device 50 of the estimated biological information (step S55). (5-4) Technical Effects of the Diagnostic Support Device 50

[0078] For example, while patients suffering from infectious diseases should receive an appropriate diagnosis from a doctor, it is undesirable for the patients to come into contact with patients in medical facilities, medical workers, etc. Due to this circumstance, there is a high demand for online diagnosis, which allows patients who should receive an appropriate diagnosis from a doctor to receive a diagnosis at home without having to meet in person with people in a medical facility, including doctors.

[0079] The diagnosis support device 50 in the fifth embodiment can estimate biometric information from a face image, so that a doctor can receive a plurality of pieces of biometric information, such as heart rate and oxygen saturation, simply by acquiring a face image sent by a patient. Therefore, the diagnosis support device 50 in the fifth embodiment can support a doctor to make an appropriate diagnosis even when the patient is in a remote location. In addition, the online diagnosis system 500 including the diagnosis support device 50 to which the fifth embodiment is applied can support a doctor to make an appropriate diagnosis even when the patient is in a remote location, so that the demand for online diagnosis can be met.

[0080] In the above, an example in which the fifth embodiment of the bioinformation estimation device is applied to an online diagnostic system has been described, but the bioinformation estimation device can be applied to mechanisms for medical purposes other than online diagnostic systems. For example, there is a desire for a subject to know his / her own physical condition regardless of whether he / she is suffering from a disease. In order to meet such a desire, for example, a dedicated app for estimating bioinformation may be installed on a terminal device such as a smartphone carried by the subject. This dedicated app may support a series of operations including capturing a face image, transmitting the face image to a cloud as a bioinformation estimation device, receiving bioinformation such as heart rate and oxygen saturation from the cloud, and presenting the bioinformation. This dedicated app may be made available via the Internet. Also, for example, the bioinformation estimation device may be mounted on a mobile terminal such as a smartphone carried by the subject. In this case, the mobile terminal may include at least a camera, a bioinformation estimation device, and a display. For example, the subject may first capture an image of his / her face with a camera, and then the bioinformation estimation device may acquire the face image, estimate at least one of the subject's heart rate and oxygen saturation, and present at least one of the subject's heart rate and oxygen saturation via a display. In the fifth embodiment, the bioinformation estimation device acquires a face image, but the acquired image may be a skin image of any part of the body other than the face image, for example, a fingerprint image. Furthermore, the bioinformation estimation device may acquire bioinformation such as the subject's voice data, heart rate, blood pressure value, cardiac output, total peripheral resistance, pulse, oxygen saturation, respiratory rate, electrocardiogram, body temperature, body moisture status, alcohol concentration, lactic acid value, blood glucose value, muscle activity, gaze movement, brain activity, consciousness level, and stress level other than the skin image. In this case, the terminal device may be equipped with a detection device such as an optical sensor or a bioelectric potential sensor according to the acquired bioinformation. Furthermore, the bioinformation estimation device may be configured to estimate bioinformation such as heart rate, blood pressure value, cardiac output, total peripheral resistance, pulse, oxygen saturation, respiratory rate, electrocardiogram, body temperature, body moisture status, alcohol concentration, lactic acid value, blood glucose value, muscle activity, gaze movement, brain activity, consciousness level, and stress level. (6) Additional Notes

[0081] The following supplementary notes are further disclosed regarding the above-described embodiment. [Appendix 1] an acquisition means for acquiring a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; an output means for outputting at least the second type of modal by inputting the first modal set into a modal generation model that outputs at least one of the multiple types of modal when at least one of the multiple types of modal is input, the modal generation model being generated by machine learning using a second modal set including the multiple types of modal; a modal generating means for generating a third modal set including the first modal set and the second type of modal output by the modal generation model; An information processing device comprising: [Appendix 2] The apparatus further includes a first model generation means for generating the modal generative model by performing machine learning using the second modal set. 2. An information processing device according to claim 1. [Appendix 3] The output means inputs, to the modal generation model, environmental information indicating an acquisition environment when the first type of modal is acquired, thereby causing the modal generation model to output the second type of modal. 3. The information processing device according to claim 1 or 2. [Appendix 4] the modal generation model includes an encoder unit that compresses a feature of at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, and a decoder unit that expands the compressed feature to generate at least one of the plurality of types of modals, The output unit inputs the environmental information to at least one of the encoder unit and the decoder unit. 4. The information processing device according to claim 3. [Appendix 5] a second model generation means for generating a modal estimation model that outputs a fourth type of modal, which is different from the third type of modal, as an output modal when a third type of modal of the multiple types of modal is input as an input modal by performing machine learning using the third modal set. 5. An information processing device according to any one of claims 1 to 4. [Appendix 6] The multiple types of modalities are multiple types of biometric information including at least one of information on a face image, a heart rate, and an oxygen saturation level. 6. An information processing device according to any one of claims 1 to 5. [Appendix 7] An image acquisition means for acquiring a facial image of a subject; a biological information estimation means for performing machine learning using a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals, and a third modal set including the second type of modal output by a modal generation model, to cause a modal estimation model to output, as the output modal, a fourth type of modal among the plurality of types of modals that is different from the third type of modal among the plurality of types of modals when the third type of modal among the plurality of types of modals is input as the input modal; and Equipped with the modal generation model outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, and is generated by machine learning using a second modal set including the plurality of types of modals; Biometric information estimation device. [Appendix 8] The image acquisition means acquires the face image from an image generation device that generates the face image by capturing an image of the subject to be generated via a communication line. 8. A biometric information estimation device as described in claim 7. [Appendix 9] The biological information includes information regarding at least one of a heart rate and an oxygen saturation level. 9. A biological information estimation device according to claim 7 or 8. [Appendix 10] obtaining a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set including the plurality of types of modals, and outputting at least the second type of modal by inputting the first modal set into the modal generation model; A third modal set is generated that includes the first modal set and the second type of modal output by the modal generation model. Information processing methods. [Appendix 11] On the computer, obtaining a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set including the plurality of types of modals, and outputting at least the second type of modal by inputting the first modal set into the modal generation model; A third modal set is generated that includes the first modal set and the second type of modal output by the modal generation model. A recording medium on which a computer program for executing an information processing method is recorded.

[0082] At least a part of the constituent elements of each of the above-mentioned embodiments can be appropriately combined with at least a part of the other constituent elements of each of the above-mentioned embodiments. Some of the constituent elements of each of the above-mentioned embodiments may not be used. In addition, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in this disclosure above are incorporated by reference as part of the description of this disclosure.

[0083] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. The information processing device, the biometric information estimation device, the information processing method, and the recording medium that involve such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0084] 1, 2, 4 Information processing device 11, 211 Acquisition Department 12, 212 Modal Generation Unit 13, 213 Modal output section IMS Partial Modal Set GMS Generated Full Modal Set 1M First Kind of Modal 2M Second type of modal 21, 51 Arithmetic device 214 First model generation unit 22, 52 storage device MMS Full Modal Set GM Modal Generative Model GME Encoder GMD Decoder EL Environmental Label 312 Input value generation unit 414 Second Model Generation Unit EM Modal Estimation Model 3M 3rd Type Modal 4M The fourth type of modal 500 Online Diagnostic System 50 Diagnostic support device 515 Image Acquisition Unit 516 Biometric Information Estimation Department 53 Communication Equipment 61 Image generating device

Claims

1. an acquisition means for acquiring a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; an output means for outputting at least the second type of modal by inputting the first modal set into a modal generation model that outputs at least one of the multiple types of modals when at least one of the multiple types of modals is input, the modal generation model being generated by machine learning using a second modal set including the multiple types of modals; a modal generating means for generating a third modal set including the first modal set and the second type of modal output by the modal generation model; Equipped with The output means causes the modal generation model to output the second type of modal by inputting environmental information indicating an acquisition environment when the first type of modal is acquired, the environmental information being known at the time of acquiring the first type of modal, to the modal generation model. Information processing device.

2. The apparatus further includes a first model generation means for generating the modal generative model by performing machine learning using the second modal set. The information processing device according to claim 1 .

3. the modal generation model includes an encoder unit that compresses a feature of at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, and a decoder unit that expands the compressed feature to generate at least one of the plurality of types of modals, The output unit inputs the environmental information to at least one of the encoder unit and the decoder unit. The information processing device according to claim 1 .

4. a second model generation means for generating a modal estimation model that outputs a fourth type of modal, which is different from the third type, of the multiple types of modals as an output modal when a third type of modal of the multiple types of modals is input as an input modal by performing machine learning using the third modal set. The information processing device according to claim 1 .

5. The multiple types of modalities are multiple types of biometric information including at least one of information on a face image, a heart rate, and an oxygen saturation level. The information processing device according to claim 1 .

6. An image acquisition means for acquiring a facial image of a subject; a biological information estimation means for causing a modal estimation model to output, as an output modal, a fourth type of modal different from the third type of modal among the multiple types of modals when a third type of modal among the multiple types of modals is input as an input modal by performing machine learning using a first modal set including at least a first type of modal among the multiple types of modals and not including a second type of modal among the multiple types of modals, and a third modal set including the second type of modal output by a modal generation model; Equipped with the modal generation model outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, and is generated by machine learning using a second modal set including the plurality of types of modals; The modal generation model outputs the second type of modal when environmental information indicating an acquisition environment when the first type of modal is acquired, which is known when the first type of modal is acquired, is input. Biometric information estimation device.

7. The image acquisition means acquires the face image from an image generation device that generates the face image by capturing an image of the subject to be generated via a communication line. The biological information estimation device according to claim 6.

8. 1. A computer-implemented information processing method, comprising: obtaining a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set including the plurality of types of modals, by inputting the first modal set into the modal generation model, causing at least the second type of modal to be output; generating a third modal set including the first modal set and the second type of modals output by the modal generation model; Including, The outputting includes inputting environmental information indicating an acquisition environment when the first type of modal is acquired, which is known at the time of acquiring the first type of modal, to the modal generation model, thereby causing the modal generation model to output the second type of modal. Information processing methods.

9. On the computer, obtaining a first modal set including at least a first type of modal among a plurality of types of modals and not including a second type of modal among the plurality of types of modals; a modal generation model that outputs at least one of the plurality of types of modals when at least one of the plurality of types of modals is input, the modal generation model being generated by machine learning using a second modal set including the plurality of types of modals, by inputting the first modal set into the modal generation model, causing at least the second type of modal to be output; generating a third modal set including the first modal set and the second type of modals output by the modal generation model; Including, The outputting includes inputting environmental information indicating an acquisition environment when the first type of modal is acquired, which is known at the time of acquiring the first type of modal, to the modal generation model, thereby causing the modal generation model to output the second type of modal. A computer program for executing an information processing method.

Images