Learning device, estimation device, learning method, estimation method, and program

Abstract

A learning device according to one aspect of the present invention comprises: a multi-modal network that is a neural network including a first encoder for acquiring a feature of input data, a second encoder for acquiring a feature of the input data, a first decoder for decoding the input data, and a second decoder for decoding the input data; and a network control unit that updates the multi-modal network by using a cross self-consistent loss indicating a difference between the input data and a self-consistent result that is the result of converting the input data by the first encoder, the second encoder, the first decoder, and the second decoder, and a common loss indicating a difference between the result of the first encoder encoding data indicating a first attribute of a main subject that is a person, living thing, object, intangible object, or phenomenon having a plurality of attributes and the result of the second encoder encoding data indicating a second attribute that is an attribute of the main subject and different from the first attribute.

Description

[Technical field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program. [Background technology]

[0002] With the recent development of sensing technology and machine learning technology, efforts are being made to recognize a wide variety of objects in the real world using machines. For example, it is possible to recognize human behavior, the surrounding environment, and objects from video information obtained with a general RGB camera. By using a multispectral camera, it is becoming possible to recognize more detailed information, such as analyzing the material of an object. Furthermore, it is possible to recognize not only video but also the content of people's conversations and emotions from audio information.

[0003] In this way, machine learning technologies for utilizing various sensor information and modal information are being actively developed. For example, in Non-Patent Document 1, an image and a text describing it are given, and by directly learning the relationship between image features and text features, it becomes possible to add a description to any image. Meanwhile, in Non-Patent Document 2, an autoencoder is used to learn a new feature representation common to multiple modalities, making it possible to solve tasks in multiple modalities with a single feature representation. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Vicente Ordonez, Girish Kulkarni, Tamara L Berg, “Im2Text: Describing Images Using 1 Million Captioned Photographs” [online], [Retrieved November 26, 2021], Internet〈URL:https: / / papers.nips.cc / paper / 2011 / file / 5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf> [Non-Patent Document 2] Jiquan Ngiam et.al.," Multimodal Deep Learning" [online], [Retrieved November 26, 2021], Internet〈URL:https: / / icml.cc / 2011 / papers / 399_icmlpaper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] However, these prior art techniques are based on the premise that all modal information is attached to each piece of training data. For example, if the modal is image and language, an image and its description must represent the same instance, and image-only or text-only data cannot be training data. In other words, learning cannot be performed unless image data and language data linked to each other exist.

[0006] Incidentally, the accuracy of analysis is higher as the accuracy of acquiring information common to linked data is higher. Therefore, in order to achieve highly accurate recognition, it is necessary to use a large amount of data linked with multiple modal information to increase the accuracy of acquiring information common to the linked data. However, it is very difficult to prepare a large amount of data linked with multiple modal information. Therefore, in conventional techniques such as the above-mentioned prior art documents, it may take a lot of effort to estimate information common to multiple types of data.

[0007] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the effort required to estimate information common to multiple types of data. [Means for solving the problem]

[0008] One aspect of the present invention is a learning device comprising: a multimodal network, which is a neural network including a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a self-consistent result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main subject, which is a person, living thing, thing, intangible object, or event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute of the main subject that is different from the first attribute.

[0009] One aspect of the present invention relates to a multimodal network that is a neural network including a data acquisition unit that acquires first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute different from the first attribute among the attributes of the estimation target, a first encoder that acquires a feature amount of input data, a second encoder that acquires a feature amount of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a self-compensating function that is a result of the input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder. and an estimation unit that performs processing performed by the trained multimodal network obtained by a learning device comprising: a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a contradiction result and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main object which is a person, living thing, thing, intangible object, or event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute which is an attribute of the main object and different from the first attribute, on the first type of target data and the second type of target data.

[0010] One aspect of the present invention is a learning method including: a network execution step of executing a multimodal network, which is a neural network including a first encoder that acquires features of input data, a second encoder that acquires features of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; and a network control unit step of updating the multimodal network using a cross self-consistency loss that indicates the difference between a self-consistent result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data; and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main subject, which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of the second encoder encoding data indicating a second attribute of the main subject, which is different from the first attribute.

[0011] One aspect of the present invention includes a data acquisition step of acquiring first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute different from the first attribute among the attributes of the estimation target, a first encoder for acquiring a feature amount of input data, a second encoder for acquiring a feature amount of the input data, a first decoder for decoding the input data, and a second decoder for decoding the input data, and a self-compensating function that is a result of the input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder. and an estimation step of executing, on the first type target data and the second type target data, a process executed by the trained multimodal network obtained by a learning device including: a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a contradiction result and the data, and a common loss that indicates the difference between a result of encoding, by the first encoder, data indicating a first attribute of a main target which is a person, a living thing, an object, an intangible object, or an event having multiple attributes, and a result of encoding, by the second encoder, data indicating a second attribute which is an attribute of the main target and different from the first attribute.

[0012] One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device.

[0013] One aspect of the present invention is a program for causing a computer to function as the above-described estimation device. Effect of the Invention

[0014] According to the present invention, it is possible to reduce the effort required to estimate information common to multiple types of data. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a first explanatory diagram illustrating an overview of a learning device according to an embodiment. [Diagram 2]FIG. 2 is a second explanatory diagram illustrating an overview of the learning device according to the embodiment. [Diagram 3] FIG. 3 is a third explanatory diagram illustrating an overview of the learning device according to the embodiment. [Figure 4] FIG. 4 is a fourth explanatory diagram illustrating an overview of the learning device according to the embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning apparatus according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a control unit included in the learning device of the embodiment. [Figure 7] 1 is a flowchart illustrating an example of a flow of a process executed by the learning device of the embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit included in the estimation device of the embodiment. [Figure 10] 4 is a flowchart showing an example of a flow of a process executed by the estimation device of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (Embodiment) An overview of a learning device 1 according to an embodiment will be described with reference to Figures 1 to 4. The learning device 1 updates a mathematical model (hereinafter referred to as a "multimodal estimation model") that estimates information common to two types of data through machine learning learning.

[0017] Since the multimodal estimation model estimates information common to two types of data, when the two types of data indicate different attributes of the same target, the target of estimation is the target whose attributes the two types of data indicate. One of the two types of data indicating the attributes of the same target is called the first type of data, and the other of the two types of data is called the second type of data. In other words, the first type of data and the second type of data are information on different attributes related to the same target.

[0018] Hereinafter, the object whose attributes are indicated by the first type data and the second type data is referred to as the main object. The main object may be any person, living thing, thing, intangible object, or event having multiple attributes. In other words, the main object may be a person having multiple attributes, a living thing having multiple attributes, an object having multiple attributes, an intangible object having multiple attributes, or an event having multiple attributes.

[0019] As described above, the multimodal estimation model estimates an object when two types of data indicate different attributes of the same object, so the main object is an example of an object to be estimated by the multimodal estimation model. The first type of data indicates a first attribute of the main object, and the second type of data indicates a second attribute of the main object. The second attribute (hereinafter referred to as the "second attribute") is an attribute that is different from the first attribute (hereinafter referred to as the "first attribute") among the attributes possessed by the main object.

[0020] For example, when the first kind of data indicates the shape of an object, the second kind of data indicates the name of the object. That is, while the first attribute is the shape, the second attribute is, for example, the name. In this way, the difference between the types of the first kind of data and the second kind of data is specifically the difference in the attributes they indicate.

[0021] A mathematical model is a set including one or more processes whose execution conditions and order (hereinafter referred to as "execution rules") are predetermined. Learning refers to updating the mathematical model using machine learning methods. Updating the mathematical model refers to appropriately adjusting the parameter values ​​in the mathematical model. Executing the mathematical model refers to executing each process included in the mathematical model in accordance with the execution rules.

[0022] The mathematical model is updated by learning until a predetermined end condition for learning (hereinafter referred to as a "learning end condition") is satisfied. The learning end condition is, for example, a condition that learning has been performed a predetermined number of times.

[0023] The learning device 1 includes two encoders and two decoders, and updates the processing contents of each encoder and each decoder through learning until a learning end condition is satisfied. The neural network including the two encoders and two decoders included in the learning device 1 is a neural network that represents a multimodal estimation model. Therefore, updating the processing contents of each encoder and each decoder is an update of the multimodal estimation model.

[0024] A neural network is a circuit such as an electronic circuit, an electric circuit, an optical circuit, or an integrated circuit that represents a mathematical model. Updating a neural network means updating the parameter values ​​of the mathematical model that represents the neural network.

[0025] Fig. 1 is a first explanatory diagram illustrating an overview of a learning device 1 of an embodiment. Fig. 2 is a second explanatory diagram illustrating an overview of a learning device 1 of an embodiment. Fig. 3 is a third explanatory diagram illustrating an overview of a learning device 1 of an embodiment. Fig. 4 is a fourth explanatory diagram illustrating an overview of a learning device 1 of an embodiment.

[0026] The learning device 1 includes a first encoder 101, a second encoder 102, a first decoder 103, and a second decoder 104. A neural network including the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 (hereinafter referred to as a "multimodal network") is an example of a neural network that represents a multimodal estimation model. Therefore, updating the multimodal network means updating the multimodal estimation model.

[0027] The first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each a neural network that is updated by learning. In learning of the multimodal network, the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each updated. In learning, the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104 are each updated so as to reduce the difference between the output of the first encoder 101 and the output of the second encoder 102.

[0028] The first encoder 101 acquires a feature of the input data. Hereinafter, the feature acquired by the first encoder 101 is referred to as a first feature. For example, first type data (001) is input to the first encoder. Therefore, the first encoder 101 acquires, for example, a first feature of the first type data.

[0029] A third type of data may be input to the first encoder 101. The third type of data indicates either a first attribute or a second attribute of a person, living thing, thing, intangible object, or event (hereinafter referred to as a "secondary object") different from the main object. The secondary object may or may not have multiple attributes.

[0030] When the third kind of data is input to the first encoder 101, the first encoder 101 acquires a first feature amount of the third kind of data. In the example of Fig. 3, the first kind of data is shown as an example of data input to the first encoder 101, but in Fig. 3, not only the first kind of data but also the third kind of data may be input to the first encoder 101.

[0031] The second encoder 102 acquires a feature of the input data. Hereinafter, the feature acquired by the second encoder 102 is referred to as a second feature. For example, second type data (002) is input to the second encoder. Therefore, the second encoder 102 acquires, for example, a second feature of the second type data.

[0032] Third type data may be input to the second encoder 102. By inputting the third type data to the second encoder 102, the second encoder 102 acquires a second feature amount of the third type data. In the example of Fig. 4, the second type data is shown as an example of data input to the second encoder 102, but in Fig. 4, not only the second type data but also the third type data may be input to the second encoder 102.

[0033] The first decoder 103 decodes the input data. Hereinafter, the result of decoding by the first decoder 103 is referred to as the first decoding result. The first decoder 103 decodes a first feature, for example, as shown in FIG. 1. Hereinafter, the result of decoding the first feature by the first decoder 103 is referred to as the first decoding result of the first feature. The first decoder 103 decodes a second feature, for example, as shown in FIG. 2 or FIG. 3. Hereinafter, the result of decoding the second feature by the first decoder 103 is referred to as the first decoding result of the second feature.

[0034] The second decoder 104 decodes the input data. Hereinafter, the result of decoding by the second decoder 104 is referred to as the second decoding result. The second decoder 104 decodes the second feature, for example, as shown in FIG. 1. Hereinafter, the result of decoding the second feature by the second decoder 104 is referred to as the second decoding result of the second feature. The second decoder 104 decodes the first feature, for example, as shown in FIG. 2 or FIG. 4. Hereinafter, the result of decoding the first feature by the second decoder 104 is referred to as the second decoding result of the first feature.

[0035] The second encoder 102 obtains the second decoded result of the first feature amount, for example, as shown in Fig. 3. The first encoder 101 obtains the first decoded result of the second feature amount, for example, as shown in Fig. 4.

[0036] When a pair of first type data and second type data is input as learning data, the learning device 1 performs learning using a linked loss function as the loss function. When only one of the first type data, the second type data, and the third type data is input as learning data, the learning device 1 performs learning using a non-linked loss function as the loss function. The learning data is data used for learning by the learning device 1. The learning is performed so that the difference indicated by the loss function becomes small.

[0037] <About the linked loss function> The linking loss function includes a reconstruction loss, a common loss, and a cross reconstruction loss. The reconstruction loss includes a first sub-reconstruction loss and a second sub-reconstruction loss. The first sub-reconstruction loss indicates a difference between the first type data and a first decoding result of the first type first feature. The first type first feature is a first feature of the first type data. The second sub-reconstruction loss indicates a difference between the second type data and a second decoding result of the second type second feature.

[0038] The reconstruction loss is expressed, for example, by the following equation (1).

[0039]

number

[0040] d1 indicates first type data. d2 indicates second type data. E1 indicates encoding processing by the first encoder 101. E2 indicates encoding processing by the second encoder 102. D1 indicates decoding processing by the first decoder 103. D2 indicates decoding processing by the second decoder 104. The first term on the right side of equation (1) is an example of the first sub-reconstruction loss. The second term on the right side of equation (2) is an example of the second sub-reconstruction loss.

[0041] Equation (1) shows the difference between the output of an autoencoder composed of the first encoder 101 and the first decoder 103, and the output of an autoencoder composed of the second encoder 102 and the second decoder 104, and the data input to each autoencoder. Therefore, if learning is performed to reduce the reconstruction loss, each autoencoder is updated so that the difference between the input data and the output is reduced.

[0042] Hereinafter, an autoencoder formed by the first encoder 101 and the first decoder 103 will be referred to as a first autoencoder. Hereinafter, an autoencoder formed by the second encoder 102 and the second decoder 104 will be referred to as a second autoencoder.

[0043] The common loss indicates the difference between the first feature amount and the second feature amount, and is expressed by, for example, the following formula (2).

[0044]

number

[0045] Equation (2) shows the difference between the intermediate representation of the first autoencoder (i.e., the first feature) and the intermediate representation of the second autoencoder (i.e., the second feature). Therefore, if learning is performed to reduce the common loss, the first autoencoder and the second autoencoder are updated so that the difference between the first feature and the second feature is reduced.

[0046] As a result, the accuracy of the multimodal estimation model in estimating information common to the first type data and the second type data, which indicate different attributes, is improved by learning to reduce the common loss. As described above, both the first type data and the second type data are information indicating attributes of the main object. Therefore, the higher the accuracy of the multimodal estimation model in estimating information common to the first type data and the second type data, the more accurately the multimodal estimation model can estimate the main object.

[0047] The cross reconstruction loss includes a first sub-cross reconstruction loss and a second sub-cross reconstruction loss. The first sub-cross reconstruction loss indicates a difference between the first type data and a first decoding result of the second type second feature. The second sub-cross reconstruction loss indicates a difference between the second type data and a second decoding result of the first type first feature.

[0048] The cross reconstruction loss is expressed, for example, by the following equation (3).

[0049]

number

[0050] The first term on the right side of equation (3) is an example of the first sub-crossing reconstruction loss. The second term on the right side of equation (3) is an example of the second sub-crossing reconstruction loss.

[0051] Since both the first type data and the second type data are data related to the main subject, the first type first feature and the second type second feature should be substantially identical as the learning progresses. If the first type first feature and the second type second feature are substantially identical, even if the decoding of the first type first feature is executed by the second decoder 104 instead of the first decoder 103, the second decoding result of the first type first feature should be substantially identical to the first data.

[0052] Furthermore, if the first type first feature and the second type second feature are substantially identical, even if the second type second feature is decoded by the first decoder 103 instead of the second decoder 104, the first decoding result of the second type second feature should be substantially identical to the second data. Therefore, a large cross reconstruction loss means that the estimation accuracy of the multimodal estimation model is poor. Therefore, by performing learning so as to reduce the cross reconstruction loss, the estimation accuracy of the multimodal estimation model is improved.

[0053] In this way, the linking loss function includes the i-th sub-reconstruction loss, the common loss, and the i-th sub-cross reconstruction loss, where i is 1 or 2. More specifically, the linking loss function includes the first reconstruction loss (003), the second reconstruction loss (004), the common loss (005), the first cross reconstruction loss (006), and the second cross reconstruction loss (007).

[0054] The i-th sub-reconstruction loss indicates the difference between the i-th type data and the i-th decoded result of the i-th type i-th feature. The i-th type i-th feature is the i-th feature of the i-th type data. The i-th sub-reconstruction loss indicates the difference between the i-th type data and the i-th decoded result of the j-th type j-th feature. Note that j is 1 or 2, and j and i indicate different values. That is, when i is 1, j is 2, and when i is 2, j is 1.

[0055] <About unlinked loss functions> The unlinked loss function includes a crossover self-consistent loss, which includes a first sub-crossover self-consistent loss and a second sub-crossover self-consistent loss.

[0056] The first sub-cross self-consistency loss indicates the difference between the data input to the first encoder 101 (hereinafter referred to as "first input data") and the first cross self-consistent data of the first input data. The first cross self-consistent data is the result of the first decoder 103 decoding the second feature of the second cross decoding result. The second cross decoding result is the result of the second decoder 104 decoding the first feature of the first input data.

[0057] The second cross self-consistent loss indicates the difference between the data input to the second encoder 102 (hereinafter referred to as "second input data") and the second cross self-consistent data of the second input data. The second cross self-consistent data is the result of the second decoder 104 decoding the first feature of the first cross decoding result. The first cross decoding result is the result of the first decoder 103 decoding the second feature of the second input data.

[0058] The first input data may be any data indicating the first attribute, and may be first type data or third type data indicating the first attribute of the sub-object. The second input data may be any data indicating the second attribute, and may be second type data or third type data indicating the second attribute of the sub-object. In other words, the i-th input data may be any data indicating the i-th attribute.

[0059] In this way, the cross self-consistency loss includes the i-th sub-cross self-consistency loss. The i-th sub-cross self-consistency loss indicates the difference between the i-th input data, which is data input to the i-th encoder, and the i-th cross self-consistent data of the i-th input data. The i-th cross self-consistent data is the result of the j-th feature of the j-th cross decoding result being decoded by the i-th decoder. The j-th cross decoding result is the result of the i-th feature of the i-th input data being decoded by the j-th decoder.

[0060] Note that the i-th encoder means the first encoder 101 when i=1, and means the second encoder 102 when i=2. The j-th encoder means the first encoder 101 when j=1, and means the second encoder 102 when j=2. Note that the i-th decoder means the first decoder 103 when i=1, and means the second decoder 104 when i=2. The j-th decoder means the first decoder 103 when j=1, and means the second decoder 104 when j=2.

[0061] The cross self-consistency loss is expressed, for example, by the following equation (4).

[0062]

number

[0063] The first term on the right-hand side of equation (4) is an example of the first sub-crossing self-consistent loss. The second term on the right-hand side of equation (4) is an example of the second sub-crossing self-consistent loss.

[0064] In addition, the value of the jth cross self-consistency loss when the ith input data is input to the ith encoder is 0. That is, when the first input data is input to the multimodal estimation model, the value of the second term on the right side of equation (4) is 0, and when the second input data is input to the multimodal estimation model, the value of the first term on the right side of equation (4) is 0.

[0065] In the examples of Figs. 1 and 2, a set of first type data and second type data is input to the multimodal network. However, there are cases where the user cannot necessarily prepare both the first type data and the second type data. Furthermore, there are cases where the user prepares data that indicates either the first attribute or the second attribute but is not data of the main subject. Even in such cases, the cross self-consistent loss makes it possible to update the multimodal network so as to improve the accuracy of estimation.

[0066] As shown in equation (4), the cross self-consistency loss indicates the difference between the self-consistency result of one piece of data (hereinafter referred to as "single-input data") input to a multimodal network and the short input data. The first input data and the second input data are each an example of single-input data. Below, the effect of using the cross self-consistency loss will be explained by taking the first input data as an example of single-input data.

[0067] <Effects of cross self-consistent losses> The self-consistent result is a result of the single input data being converted by the first encoder 101, the second encoder 102, the first decoder 103, and the second decoder 104. If the estimation accuracy of the multimodal estimation model is improved, the first feature amount and the second feature amount should be substantially identical, and if the first feature amount and the second feature amount are substantially identical, the second feature amount of the second decoded result of the first feature amount should also be substantially identical. As a result, the result of decoding the second feature amount of the second decoded result obtained from the first feature amount of the first input data by the first decoder 103 should be substantially identical to the first input data. Therefore, by performing learning so as to reduce the cross self-consistency loss, the multimodal estimation model (i.e., the multimodal network) is updated so as to improve the estimation accuracy.

[0068] In this way, the unlinked loss function includes the cross-self-consistent loss. More specifically, the unlinked loss function includes the first sub-cross-self-consistent loss (008) and the second sub-cross-self-consistent loss (009). As shown in FIG. 3 or FIG. 4, the unlinked loss function can obtain a value if any one of the first type data, the second type data, or the third type data is obtained. Therefore, by using the unlinked loss function, it is possible to update the multimodal estimation model without using two types of data, the first type data and the second type data.

[0069] As described above, the learning device 1 can update the multimodal network by using a linking loss function when a pair of the first type data and the second type data is input. Also, the learning device 1 can update the multimodal network by using a non-linking loss function even when only one of the first type data, the second type data, and the third type data is input.

[0070] Therefore, even if the learning device 1 performs learning using only one of the first type data, the second type data, and the third type data in some of the multiple learnings, it is possible to update the multimodal network to improve the estimation accuracy. In other parts of the multiple learnings, learning is performed using a pair of the first type data and the second type data, so that the learning device 1 updates the multimodal network to improve the estimation accuracy.

[0071] 5 is a diagram showing an example of a hardware configuration of the learning device 1 in an embodiment. The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. The learning device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15 by executing the program.

[0072] More specifically, in the learning device 1, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0073] The control unit 11 controls the operations of various functional units included in the learning device 1. The control unit 11 performs learning of a multimodal estimation model, for example.

[0074] The input unit 12 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 12 may include an interface that connects these input devices to the study device 1.

[0075] The communication unit 13 includes an interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits third type data indicating the first attribute. The communication unit 13 acquires the third type data indicating the first attribute by communicating with the device that transmits the third type data indicating the first attribute. The external device is, for example, a device that transmits third type data indicating the second attribute. The communication unit 13 acquires the third type data indicating the second attribute by communicating with the device that transmits the third type data indicating the second attribute.

[0076] The external device is, for example, a device that transmits first-kind data. The communication unit 13 acquires the first-kind data by communicating with the device that transmits the first-kind data. The external device is, for example, a device that transmits second-kind data. The communication unit 13 acquires the second-kind data by communicating with the device that transmits the second-kind data.

[0077] The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information related to the learning device 1. The storage unit 14 stores various information generated as a result of processing executed by the control unit 11, for example.

[0078] The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The output unit 15 may include an interface that connects these display devices to the learning device 1.

[0079] 6 is a diagram illustrating an example of a functional configuration of the control unit 11 included in the learning device 1 of the embodiment. The control unit 11 includes a data acquisition unit 111, a data input unit 112, a learning unit 113, a communication control unit 114, a memory control unit 115, and an output control unit .

[0080] The data acquisition unit 111 acquires data input to the communication unit 13. Specifically, candidates for data acquired by the data acquisition unit 111 are first kind of data, second kind of data, and third kind of data.

[0081] The data input unit 112 outputs the data acquired by the data acquisition unit 111 to an output destination according to the attribute indicated by each data. For example, the data input unit 112 outputs the first kind of data acquired by the data acquisition unit 111 to the first encoder 101. For example, the data input unit 112 outputs the second kind of data acquired by the data acquisition unit 111 to the second encoder 102. For example, the data input unit 112 outputs the third kind of data acquired by the data acquisition unit 111, which is third kind of data indicating a first attribute, to the first encoder 101.

[0082] The data input unit 112 outputs, for example, third type data acquired by the data acquisition unit 111, which indicates a second attribute, to the second encoder 102. Each of the first type data, second type data, and third type data has information indicating the attribute indicated by each data. The information indicating the attribute may be any information as long as it indicates the attribute according to a predetermined rule. The information indicating the attribute is, for example, information that expresses a difference in attribute by a difference in data format. The difference in data format is, for example, a difference in data format between image data and text data, etc.

[0083] In this way, the attribute indicated by each data is indicated by the information indicating the attribute. Therefore, the data input unit 112 can determine the attribute of each data based on the difference in the information indicating the attribute. As a result, the data input unit 112 can output the data acquired by the data acquisition unit 111 to an output destination according to the attribute indicated by each data. Note that the information indicating the attribute of each data may be input by the user to the input unit 12 or the communication unit 13.

[0084] The learning unit 113 includes a multimodal network 131 and a network control unit 132. The multimodal network 131 is a multimodal network. Therefore, the multimodal network 131 is a neural network that represents a multimodal estimation model. Therefore, the multimodal network 131 includes a first encoder 101, a second encoder 102, a first decoder 103, and a second decoder 104.

[0085] The network control unit 132 updates the multimodal network 131 based on the results obtained by the multimodal network 131. More specifically, the network control unit 132 updates the multimodal network 131 based on the results obtained by the multimodal network 131 and the data input to the multimodal network 131.

[0086] For example, when the data acquired by the data acquisition unit 111 is a pair of first type data and second type data, the network control unit 132 uses a linking loss function and updates the multimodal network 131 based on the results obtained by the multimodal network 131.

[0087] For example, when the data acquired by the data acquisition unit 111 is not a pair of the first type of data and the second type of data, the network control unit 132 uses a non-linked loss function and updates the multimodal network 131 based on the results obtained by the multimodal network 131.

[0088] The communication control unit 114 controls the operation of the communication unit 13. The memory control unit 115 controls the operation of the memory unit 14. The output control unit 116 controls the operation of the output unit 15.

[0089] 7 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The data acquisition unit 111 acquires data (step S101). Next, the data input unit 112 inputs the data acquired by the data acquisition unit 111 to an input destination according to the attribute indicated by each data (step S102). Specifically, the input destination is the first encoder 101 or the second encoder 102.

[0090] Next, the multimodal network 131 executes the multimodal estimation model on the data input in step S102 (step S103). Next, the network control unit 132 updates the multimodal estimation model based on the result of the execution of the multimodal estimation model (step S104). More specifically, the network control unit 132 updates the multimodal estimation model using either the linking loss function or the non-linking loss function according to the data acquired by the data acquisition unit 111 based on the result of the execution of the multimodal estimation model.

[0091] Next, the network control unit 132 determines whether or not the learning end condition is satisfied (step S105). If the learning end condition is satisfied (step S105: YES), the process ends. On the other hand, if the learning end condition is not satisfied (step S105: NO), the process returns to step S101.

[0092] The trained multimodal estimation model thus obtained is used in a device that estimates an estimation target based on a pair of data on a first attribute of the estimation target and data on a second attribute of the estimation target, such as the estimation device 2 shown in FIG. 8. The trained multimodal estimation model is a multimodal estimation model at the time when a learning end condition is satisfied. Hereinafter, the estimation target of the estimation device 2 is referred to as a target of interest. Hereinafter, data on the first attribute of the target of interest is referred to as first type target data. Hereinafter, data on the second attribute of the target of interest is referred to as second type target data. The target of interest may be an object different from the main object.

[0093] 8 is a diagram illustrating an example of a hardware configuration of the estimation device 2 according to the embodiment. The estimation device 2 acquires a pair of first type target data and second type target data, and estimates a target of interest indicated by the acquired first type target data and second type target data using a trained multimodal estimation model.

[0094] The estimation device 2 includes a control unit 21 having a processor 93 such as a CPU and a memory 94 connected by a bus, and executes a program. The estimation device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25 by executing the program.

[0095] More specifically, in the estimation device 2, the processor 93 reads a program stored in the storage unit 24 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, whereby the estimation device 2 functions as a device including a control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

[0096] The control unit 21 controls the operations of various functional units included in the estimation device 2. The control unit 21 executes, for example, a trained multimodal estimation model.

[0097] The input unit 22 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 22 may include an interface that connects these input devices to the estimation device 2.

[0098] The communication unit 23 includes an interface for connecting the estimation device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits a pair of the first type of target data and the second type of target data. The communication unit 23 acquires the pair of the first type of target data and the second type of target data by communicating with the device that transmits the pair of the first type of target data and the second type of target data. The external device is, for example, the learning device 1. The communication unit 23 acquires a trained multimodal estimation model by communicating with the learning device 1.

[0099] The storage unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 24 stores various information related to the estimation device 2. The storage unit 24 stores various information generated as a result of processing executed by the control unit 21, for example.

[0100] The output unit 25 includes a display device such as a CRT display, a liquid crystal display, an organic EL display, etc. The output unit 25 may include an interface that connects these display devices to the estimation device 2.

[0101] 9 is a diagram illustrating an example of a functional configuration of the control unit 21 included in the estimation device 2 of the embodiment. The control unit 21 includes a data acquisition unit 211, an estimation unit 212, a communication control unit 213, a storage control unit 214, and an output control unit 215.

[0102] The data acquisition unit 211 acquires a set of data of the first type target data and the second type target data input to the communication unit 23. The estimation unit 212 executes a trained multimodal estimation model on the data acquired by the data acquisition unit 211. That is, the estimation unit 212 executes a process of estimating an object of interest using a trained multimodal network on the first type target data and the second type target data acquired by the data acquisition unit 211. The communication control unit 213 controls the operation of the communication unit 23. The memory control unit 214 controls the operation of the memory unit 24. The output control unit 215 controls the operation of the output unit 25.

[0103] 10 is a flowchart showing an example of the flow of processing executed by the estimation device 2 of the embodiment. The data acquisition unit 211 acquires a set of data of the first type of target data and the second type of target data (step S201). Next, the estimation unit 212 executes a trained multimodal estimation model on the data acquired in step S201 (step S202). That is, the estimation unit 212 executes processing executed by the trained multimodal network on the first type of target data and the second type of target data acquired in step S201. The processing executed by the trained multimodal network is, specifically, processing for estimating an object of interest.

[0104] The target object is estimated by executing the trained multimodal estimation model on the data acquired in step S201. After step S202, the output control unit 215 controls the operation of the output unit 25 to cause the output unit 25 to output the result of estimation by the estimation unit 212 (step S203).

[0105] The learning device 1 configured in this way performs learning so as to improve the accuracy of estimation of a mathematical model that estimates information common to multiple types of data. In the learning device 1, the information common to multiple types of data is specifically a main target. The learning device 1 can execute learning using not only a linked loss function but also a non-linked loss function.

[0106] Therefore, the learning device 1 can perform learning not only when linked information such as a pair of first type data and second type data exists, but also when only one of them exists or when only the third type data exists. Therefore, the learning device 1 can obtain a mathematical model that estimates information common to multiple types of data with less effort than a mathematical model obtained by a technology that can perform learning only when linked information exists. In other words, the learning device 1 can reduce the effort required to estimate information common to multiple types of data.

[0107] Furthermore, the estimation device 2 configured in this manner estimates an estimation target using the learned mathematical model obtained by the learning device 1. Therefore, the estimation device 2 can reduce the effort required to estimate information common to multiple types of data.

[0108] (Modification) In addition, in the learning using the unlinked loss function by the learning device 1, it is preferable to perform learning using the third type data rather than the first type data or the second type data, because the occurrence of overlearning can be suppressed by performing learning not only on the main object but also on the secondary object.

[0109] In addition, the unlinked loss function may further include the i-th reconstruction loss in addition to the i-th cross self-consistent loss. As described above, the value of the j-th cross self-consistent loss is 0 when training data is input to the i-th encoder. In addition, the value of the j-th cross self-consistent loss is also 0 when training data is input to the i-th encoder. When the unlinked loss function further includes the i-th reconstruction loss, the estimation accuracy of the trained multimodal estimation model is higher than when the i-th reconstruction loss is not included.

[0110] It is not necessary that the learning device 1 and the estimation device 2 are implemented as different devices. For example, the learning device 1 and the estimation device 2 may be implemented as a single device or system having both functions.

[0111] Furthermore, each functional unit included in the learning device 1 and the estimation device 2 may be implemented using a plurality of information processing devices communicatively connected via a network.

[0112] Each of the learning device 1 and the estimation device 2 may be implemented using a plurality of information processing devices communicatively connected via a network. All or part of each function of the learning device 1 and the estimation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include portable media such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and storage devices such as a hard disk built into a computer system. The program may be transmitted via a telecommunication line.

[0113] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0114] 1...Learning device, 2...Estimation device, 101...First encoder, 102...Second encoder, 103...First decoder, 104...Second decoder, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Memory unit, 15...Output unit, 111...Data acquisition unit, 112...Data input unit, 113...Learning unit, 131...Multimodal network, 132...Network control unit, 114...Communication control unit, 115...Memory control unit, 116...Output control unit, 21...Control unit, 22...Input unit, 23...Communication unit, 24...Memory unit, 25...Output unit, 211...Data acquisition unit, 212...Estimation unit, 213...Communication control unit, 214...Memory control unit, 215...Output control unit, 91, 93...Processor, 92, 94...Memory

Claims

1. a multimodal network that is a neural network including a first encoder that acquires a feature amount of input data, a second encoder that acquires a feature amount of the input data, a first decoder that decodes the input data, and a second decoder that decodes the input data; a network control unit that updates the multimodal network using a cross self-consistency loss that indicates the difference between a self-consistency result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss that indicates the difference between a result of the first encoder encoding data indicating a first attribute of a main object, which is a person, a living thing, an object, an intangible object, or an event having multiple attributes, and a result of the second encoder encoding data indicating a second attribute of the main object that is different from the first attribute; A learning device comprising:

2. the network control unit updates the multimodal network using the self-consistency result of the third type of data, which is data indicating either a first attribute or a second attribute of a person, a living thing, an object, an intangible object, or an event different from the main object; The learning device according to claim 1 .

3. The network control unit updates the multimodal network based on a first secondary reconstruction loss indicating a difference between a result of decoding by the first decoder of a result of encoding by the first encoder of first type data, the first type data being data indicating the first attribute of the main object, and the first type data, and a second secondary reconstruction loss indicating a difference between a result of decoding by the second decoder of a result of encoding by the second encoder of second type data, the second type data being data indicating the second attribute of the main object, and the second type data. The learning device according to claim 1 or 2.

4. a data acquisition unit that acquires first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute, which is different from the first attribute, among attributes of the estimation target; a multimodal network which is a neural network including a first encoder which acquires a feature amount of input data, a second encoder which acquires a feature amount of the input data, a first decoder which decodes the input data, and a second decoder which decodes the input data; and a network control unit which updates the multimodal network using a cross self-consistency loss which indicates a difference between a self-consistency result which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder and the input data, and a common loss which indicates a difference between a result of encoding data by the first encoder which indicates a first attribute of a main object which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of encoding data by the second encoder which indicates a second attribute which is an attribute of the main object and which is different from the first attribute; An estimation device comprising:

5. a network execution step of executing a multimodal network, which is a neural network including a first encoder for acquiring a feature amount of input data, a second encoder for acquiring a feature amount of the input data, a first decoder for decoding the input data, and a second decoder for decoding the input data; a network control unit step of updating the multimodal network using a cross self-consistency loss indicating the difference between a self-consistency result, which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder, and the data, and a common loss indicating the difference between a result of the first encoder encoding data indicating a first attribute of a main object, which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of the second encoder encoding data indicating a second attribute of the main object, which is different from the first attribute; A learning method that has

6. a data acquiring step of acquiring first type target data indicating a first attribute of an estimation target and second type target data indicating a second attribute, which is different from the first attribute, among attributes of the estimation target; an estimation step of executing, on the first type target data and the second type target data, a process executed by the trained multimodal network obtained by a learning device including: a multimodal network which is a neural network including a first encoder which acquires a feature amount of input data, a second encoder which acquires a feature amount of the input data, a first decoder which decodes the input data, and a second decoder which decodes the input data; and a network control unit which updates the multimodal network using a cross self-consistency loss which indicates a difference between a self-consistency result which is a result of input data being converted by the first encoder, the second encoder, the first decoder, and the second decoder and the input data, and a common loss which indicates a difference between a result of encoding data by the first encoder which indicates a first attribute of a main object which is a person, a living thing, an object, an intangible object, or an event having a plurality of attributes, and a result of encoding data by the second encoder which indicates a second attribute which is an attribute of the main object and which is different from the first attribute; The estimation method has the following structure:

7. A program for causing a computer to function as the learning device according to any one of claims 1 to 3.

8. A program for causing a computer to function as the estimation device according to claim 4.

Images