Learning device, estimation device, learning method, estimation method, and program

Abstract

To improve the accuracy of conversion between different modalities.SOLUTION: A control unit configured to perform, in order of first learning, second learning, and third learning, the first learning being learning of a correspondence relationship between a first amount and a second amount by using a first set that is a set of the first amount and the second amount, the second learning being learning of a correspondence relationship between a second amount and a third amount by using a second set that is a set of the second amount and the third amount, the third learning being learning of a correspondence relationship between the third amount and a fourth amount by using a third set that is a set of the third amount and the fourth amount. Wherein modalities of the first amount, the second amount, the third amount, and the fourth amount are different from each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning device, an estimation device, a learning method, an estimation method, and a program. [Background technology]

[0002] There is a demand for a technology that can estimate one of multiple different modalities based on the other. In other words, there is a demand for a technology that can convert between different modalities. Such a technology can be achieved by machine learning. In machine learning, the relationship between a central modality, which is a predetermined one of multiple different modalities, and the other modalities is learned. Specifically, the learning is performed using a pair of the central modality and the other modalities associated with it. Therefore, for the learning, it is necessary to obtain the central modality and the other modalities from the same event. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Girdhar, Rohit, et al. "Imagebind: One embedding space to bind them all." arXiv preprint arXiv: 2305.05665v2 (2023). Summary of the Invention [Problem to be solved by the invention]

[0004] However, some of the multiple modalities may not be available from the event for which the central modality is available, or may be difficult to obtain. In such cases, it is difficult to learn the relationship between the difficult-to-obtain modalities and the central modality. As a result, the accuracy of the conversion may be poor.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for improving the accuracy of conversion between different modalities. [Means for solving the problem]

[0006] One aspect of the present invention is a learning device that includes a control unit that performs first learning, second learning, and third learning in the order of first learning, the first learning using a first set of a first quantity and a second quantity to learn the correspondence between the first quantity and the second quantity, the second learning using a second set of a second quantity and a third quantity to learn the correspondence between the second quantity and the third quantity, and the third learning using a third set of a third quantity and a fourth quantity to learn the correspondence between the third quantity and the fourth quantity, wherein the modalities of the first quantity, the second quantity, the third quantity, and the fourth quantity are different from each other.

[0007] One aspect of the present invention is an estimation device comprising: a control unit that performs first learning, second learning, and third learning in the order of first learning, the first learning being a first set of a first quantity and a second quantity to learn a correspondence relationship between the first quantity and the second quantity; second learning being a second set of a second quantity and a third quantity to learn a correspondence relationship between the second quantity and the third quantity; and third learning being a third set of a third quantity and a fourth quantity to learn a correspondence relationship between the third quantity and the fourth quantity; wherein the modalities of the first quantity, the second quantity, the third quantity, and the fourth quantity are different from each other; and an estimation unit that performs estimation using the learning results obtained by the learning device.

[0008] One aspect of the present invention is a learning method performed by a learning device, the learning method including a control unit that performs first learning, second learning, and third learning in the order of: first learning, which uses a first set of a first quantity and a second quantity to learn the correspondence between the first quantity and the second quantity; second learning, which uses a second set of a second quantity and a third quantity to learn the correspondence between the second quantity and the third quantity; and third learning, which uses a third set of a third quantity and a fourth quantity to learn the correspondence between the third quantity and the fourth quantity; wherein the modalities of the first quantity, the second quantity, the third quantity, and the fourth quantity are different from each other, and the learning method includes a first learning step in which the control unit performs the first learning, a second learning step in which the control unit performs the second learning, and a third learning step in which the control unit performs the third learning.

[0009] One aspect of the present invention is an estimation method executed by an estimation device including a control unit that executes first learning, second learning, and third learning in that order: first learning, which uses a first set of a first quantity and a second quantity to learn a correspondence between the first quantity and the second quantity; second learning, which uses a second set of a second quantity and a third quantity to learn a correspondence between the second quantity and the third quantity; and third learning, which uses a third set of a third quantity and a fourth quantity to learn a correspondence between the third quantity and the fourth quantity; wherein the modalities of the first quantity, the second quantity, the third quantity, and the fourth quantity are different from each other; and an estimation unit that performs estimation using a result of learning obtained by the learning device, the estimation method including an estimation step in which the estimation unit performs the estimation.

[0010] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0011] One aspect of the present invention is a program for causing a computer to function as the above-described estimation device. [Effects of the Invention]

[0012] The present invention makes it possible to improve the accuracy of conversion between different modalities. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is an explanatory diagram illustrating an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to an embodiment. [Figure 3] 10 is a flowchart showing an example of a flow of processing executed by a learning device according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. [Figure 5] 1 is a flowchart showing an example of a flow of processing executed by an estimation device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Embodiment) 1 is an explanatory diagram illustrating an information processing system 100 according to an embodiment. The information processing system 100 includes a learning device 1 and an estimation device 2.

[0015] The learning device 1 includes a control unit 11 having a processor 91, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus, and executes a program.

[0016] The control unit 11 executes, for example, first learning. The first learning is a process of learning the correspondence between the first amount and the second amount using a first set, which is a pair of a first amount and a second amount. Therefore, the first learning is a process of updating the correspondence between the first amount and the second amount using the first set until a predetermined condition for ending the first learning (hereinafter referred to as a "first learning ending condition") is satisfied.

[0017] The first learning termination condition may be any condition related to the termination of the first learning, such as that the correspondence between the first amount and the second amount has been updated a predetermined number of times or more. The first learning termination condition may be that the change in the correspondence between the first amount and the second amount due to the update is smaller than a predetermined change.

[0018] The control unit 11 executes, for example, second learning. The second learning is a process of learning the correspondence between the second quantity and the third quantity using a second set, which is a pair of the second quantity and the third quantity. Therefore, the second learning is a process of updating the correspondence between the second quantity and the third quantity using the second set until a predetermined condition for terminating the second learning (hereinafter referred to as a "second learning termination condition") is satisfied.

[0019] The second learning termination condition may be any condition related to the termination of the second learning, such as that the correspondence between the second quantity and the third quantity has been updated a predetermined number of times or more. The second learning termination condition may be that the change in the correspondence between the second quantity and the third quantity due to the update is smaller than a predetermined change.

[0020] The control unit 11 executes, for example, third learning. The third learning is a process of learning the correspondence between the third quantity and the fourth quantity using a third set, which is a pair of the third quantity and the fourth quantity. Therefore, the third learning is a process of updating the correspondence between the third quantity and the fourth quantity using the third set until a predetermined condition for ending the third learning (hereinafter referred to as a "third learning ending condition") is satisfied.

[0021] The third learning termination condition may be any condition related to the termination of the third learning, such as that the correspondence relationship between the third quantity and the fourth quantity has been updated a predetermined number of times or more. The third learning termination condition may be that the change in the correspondence relationship between the third quantity and the fourth quantity due to the update is smaller than a predetermined change.

[0022] The control unit 11 executes the first learning, the second learning, and the third learning in the order of the first learning, the second learning, and the third learning.

[0023] The first, second, third, and fourth quantities are of different modalities. For example, the second quantity may be image data of a train, the first quantity may be text data describing the train, the third quantity may be sound data indicating the sound of the train when it is running, and the fourth quantity may be data of measurements obtained by an inertial measurement unit attached to the train.

[0024] 1, data D101, which is text data, is an example of a first quantity, 3D skeleton data D102 is an example of a second quantity, data D103 obtained by an IMU (Inertial Measurement Unit) is an example of a third quantity, and data D104 obtained by a GPS (Global Positioning System) is an example of a fourth quantity. Therefore, in the example of Fig. 1, learning the correspondence between data D101 and data D102 is an example of first learning, learning the correspondence between data D102 and data D103 is an example of second learning, and learning the correspondence between data D103 and data D104 is an example of third learning.

[0025] <Effects of implementing the first to third learnings> The effects of performing the first to third learning are explained below. Performing the first to third learning can improve the accuracy of conversion between different modalities when it is easier to obtain the second set than the set of the first and third quantities, and when it is easier to obtain the third set than the set of the fourth quantity and the second quantity and the set of the fourth quantity and the first quantity.

[0026] 1 are examples of the first to fourth quantities when the second set is easier to obtain than the set of the first quantity and the third quantity, and the third set is easier to obtain than the set of the fourth quantity and the second quantity and the set of the fourth quantity and the first quantity. 3D skeleton is modality data that can express the meaning of time-series data. Text data is also modality data that can express the meaning of time-series data.

[0027] In this way, the first quantity and the second quantity may be data of a modality capable of expressing the meaning of the time-series data. Alternatively, one of the first quantity and the second quantity may be time-series data, and the other may be data of a modality capable of expressing the meaning of the time-series data. Alternatively, both the first quantity and the second quantity may be time-series data of different modalities. In this way, at least one of the first quantity and the second quantity may be time-series data.

[0028] Both the 3D skeleton and the text data are examples of data of a modality capable of expressing the meaning of time-series data, and the first or second quantity may be data of another modality capable of expressing the meaning of time-series data. Also, the first or second quantity may be data of a modality capable of expressing the meaning of data other than time-series data.

[0029] In this way, by executing the first to third learning, it becomes possible to process not only non-time-series data but also time-series data based on its modality.

[0030] <More specific examples of the processes in learning 1 to learning 3> A more specific example of the first to third learning processes will be described.

[0031] The first learning may be, for example, learning using a pair of a first quantity and a second quantity, and updating the first encoder and the second encoder until a first learning termination condition is met so as to reduce the difference in output between the first encoder that converts the first quantity and the second encoder that converts the second quantity.

[0032] At this time, as learning progresses, the degree of agreement between the output of the first encoder and the output of the second encoder increases. In other words, as learning progresses, the correspondence between the first quantity and the second quantity becomes clearer. Therefore, learning between the first encoder and the second encoder is learning the correspondence between the first quantity and the second quantity.

[0033] The second learning may be, for example, learning using a pair of a second quantity and a third quantity, and updating the second encoder and the third encoder while freezing at least a portion of the second encoder until a second learning termination condition is met so as to reduce the difference in output between the second encoder and the third encoder that converts the third quantity.

[0034] At this time, as the learning progresses, the degree of agreement between the output of the second encoder and the output of the third encoder increases. In other words, as the learning progresses, the correspondence between the second quantity and the third quantity becomes clearer. Therefore, learning that updates the second encoder and the third encoder while freezing at least a part of the second encoder is learning of the correspondence between the second quantity and the third quantity.

[0035] Furthermore, in this case, the third learning may be, for example, learning using a pair of a third quantity and a fourth quantity, and updating the third encoder and the fourth encoder while freezing at least a portion of the third encoder until a third learning termination condition is met so as to reduce the difference in output between the third encoder and the fourth encoder that converts the fourth quantity.

[0036] At this time, as the learning progresses, the degree of agreement between the output of the third encoder and the output of the fourth encoder increases. In other words, as the learning progresses, the correspondence between the third quantity and the fourth quantity becomes clearer. Therefore, learning that updates the third encoder and the fourth encoder while freezing at least a part of the third encoder is learning of the correspondence between the third quantity and the fourth quantity.

[0037] <About the effects of freezing> We have mentioned that freezing can be performed in the first to third learning stages, and we will now explain the effects of this freezing. "Frozen weight" is a technique that is widely known in the field of machine learning. To explain the effects of freezing, we will introduce a problem known in the field of machine learning.

[0038] The problem is that when an encoder that has been trained on certain data is trained again on different data, the previously learned content may be forgotten. This is a common problem in machine learning. In fact, freezing the data in the first to third training stages has the effect of reducing the risk of this forgetting.

[0039] This is because freezing is a technique for learning in which at least some of the model parameters are not updated, rather than updating all of the model parameters through learning. By freezing in the second learning, the results of the first learning are less likely to be forgotten even when the second learning is performed. By freezing in the third learning, the results of the second learning are less likely to be forgotten even when the third learning is performed. Therefore, freezing has the effect of reducing the risk of forgetting.

[0040] <Estimation device 2> The estimation device 2 performs estimation processing. The estimation processing is processing for making estimations using the results of learning obtained by the learning device 1. Using the results of learning obtained by the learning device 1 means using the results of learning executed by the control unit 11, and for example, when the control unit 11 performs first learning, second learning, and third learning, it means using some or all of the first correspondence relationship, second correspondence relationship, and third correspondence relationship.

[0041] The first correspondence relationship is the correspondence relationship between the first quantity and the second quantity when the first learning termination condition is satisfied. The second correspondence relationship is the correspondence relationship between the second quantity and the third quantity when the second learning termination condition is satisfied. The third correspondence relationship is the correspondence relationship between the third quantity and the fourth quantity when the third learning termination condition is satisfied. Therefore, the estimation device 2, for example, receives an input of the fourth quantity and estimates the first quantity.

[0042] <Example of hardware configuration of learning device 1> 2 is a diagram showing an example of the hardware configuration of the learning device 1 according to the embodiment. The learning device 1 includes a control unit 11 and executes a program, and functions as a device including the control unit 11, an interface unit 12 including a communication interface 121, and a storage unit 13 by executing the program.

[0043] More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0044] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, the learning process as described above. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring the information stored in the memory unit 13 is reading.

[0045] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication.

[0046] The external device is, for example, a device that has transmitted a pair of a first amount and a second amount. In such a case, the interface unit 12 acquires the pair of the first amount and the second amount by communicating with the device that has transmitted the pair of the first amount and the second amount. The external device is, for example, a device that has transmitted a pair of the second amount and a third amount. In such a case, the interface unit 12 acquires the pair of the second amount and the third amount by communicating with the device that has transmitted the pair of the second amount and the third amount.

[0047] The external device is, for example, a device that is the sender of the pair of the third amount and the fourth amount. In this case, the interface unit 12 acquires the pair of the third amount and the fourth amount by communicating with the device that is the sender of the pair of the third amount and the fourth amount.

[0048] The external device may be, for example, the estimation device 2. In this case, the estimation device 2 can perform estimation using the learning results obtained by the learning device 1 through communication via the interface unit 12.

[0049] Interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. Interface unit 12 may be configured as an interface that connects these input devices to learning device 1. In this way, the input devices of interface unit 12 accept input of various information to learning device 1 via wired or wireless connections. Note that information does not necessarily have to be input to the communication interface of interface unit 12, but may also be input to the input devices of interface unit 12.

[0050] Interface unit 12 outputs, for example, various types of information. Interface unit 12 is configured to include a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, and a speaker. Interface unit 12 may be configured as an interface that connects these display devices or speakers to learning device 1. Therefore, interface unit 12 may output, for example, information input to an input device of interface unit 12 as an image or sound.

[0051] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may exist on a cloud, for example.

[0052] 3 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The control unit 11 obtains a first set (step S101). Next, the control unit 11 executes first learning (step S102). Next, the control unit 11 obtains a second set (step S103). Next, the control unit 11 executes second learning (step S104). Next, the control unit 11 obtains a third set (step S105). Next, the control unit 11 executes third learning (step S106).

[0053] <Example of hardware configuration of estimation device 2> 4 is a diagram illustrating an example of a hardware configuration of the estimation device 2 according to an embodiment. The estimation device 2 includes a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected via a bus, and executes a program. By executing the program, the estimation device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0054] More specifically, the processor 93 reads out a program stored in the storage unit 23 and stores the read program in the memory 94. The processor 93 executes the program stored in the memory 94, causing the estimation device 2 to function as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0055] The control unit 21 controls the operation of each functional unit included in the estimation device 2. The control unit 21 executes, for example, an estimation process. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring the information stored in the memory unit 23 is reading.

[0056] The interface unit 22 includes a communication interface for connecting the estimation device 2 to an external device. The interface unit 22 communicates with the external device via wire or wirelessly.

[0057] The external device is, for example, a device that transmits the estimation source information in the estimation process. The estimation source information is information based on which estimation is performed in the estimation process. The estimation source information is, for example, a fourth quantity obtained by the user of the estimation device 2. In the estimation process, for example, if the estimation source information is a fourth quantity, the first quantity is estimated based on the fourth quantity. The interface unit 22 acquires the estimation source information in the estimation process by communicating with the device that transmits the estimation source information in such an estimation process.

[0058] The external device may be, for example, the learning device 1. In this case, the estimation device 2 can use the learning results obtained by the learning device 1 through communication via the interface unit 22.

[0059] The interface unit 22 may be configured to include input devices such as a mouse, a keyboard, a touch panel, etc. The interface unit 22 may be configured as an interface that connects these input devices to the estimation device 2. In this way, the input devices of the interface unit 22 accept input of various information to the estimation device 2 via wired or wireless connections. Note that information does not necessarily have to be input to the communication interface of the interface unit 22, and may also be input to the input devices of the interface unit 22.

[0060] The interface unit 22 outputs, for example, various types of information. The interface unit 22 includes, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the estimation device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 as an image or sound.

[0061] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the estimation device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0062] 5 is a flowchart showing an example of the flow of processing executed by the estimation device 2 in an embodiment. The control unit 21 acquires information on an estimation source (step S201). The control unit 21 executes estimation processing (step S202). By executing the estimation processing, an estimation result is obtained using the first correspondence relationship, the second correspondence relationship, or the third correspondence relationship based on the information on the estimation source obtained in step S201.

[0063] The learning device 1 configured in this manner executes the first to third learnings, and therefore, as described in <Effects of executing the first to third learnings>, the learning device 1 can improve the accuracy of conversion between different modalities.

[0064] Furthermore, the estimation device 2 configured in this manner performs estimation using the learning results obtained by the learning device 1. Therefore, the estimation device 2 can improve the accuracy of conversion between different modalities.

[0065] Furthermore, the information processing system 100 configured in this manner includes the learning device 1. Therefore, the information processing system 100 can improve the accuracy of conversion between different modalities.

[0066] (Variation) Up to this point, the first to fourth quantities have been used as examples for explanation, but this is not limited to the first to fourth quantities, and the correspondence between any two of the fourth to (N+1)th quantities (N is a predetermined integer greater than or equal to 4) may also be learned.

[0067] Specifically, the control unit 11 may perform (N-3) learnings, from fourth learning to Nth learning, after the first learning, second learning, and third learning. The nth learning (n is an integer between 4 and N) is a process of learning the correspondence relationship between the nth quantity and the (n+1)th quantity (hereinafter referred to as the "nth correspondence relationship") using the nth set, which is a pair of the nth quantity and the (n+1)th quantity. Therefore, the nth learning is a process of updating the correspondence relationship between the nth quantity and the (n+1)th quantity using the nth set until a predetermined condition for terminating the nth learning (hereinafter referred to as the "nth learning termination condition") is satisfied.

[0068] The n-th learning termination condition may be any condition related to the termination of the n-th learning, such as that the correspondence between the n-th quantity and the (n+1)-th quantity has been updated a predetermined number of times or more. The n-th learning termination condition may be that the change in the correspondence between the n-th quantity and the (n+1)-th quantity due to the update is smaller than a predetermined change.

[0069] The nth learning is learning using a pair of the nth quantity and the (n+1)th quantity, and is learning that updates the nth encoder and the (n+1)th encoder until the nth learning termination condition is met so as to reduce the difference in output between the nth encoder that converts the nth quantity and the (n+1)th encoder that converts the (n+1)th quantity.

[0070] When the nth learning is performed, the interface unit 12 acquires the pair of the nth quantity and the (n+1)th quantity by communicating with the device that sent the pair of the nth quantity and the (n+1)th quantity.

[0071] Note that, in the first to M-th learnings (M is a predetermined integer equal to or greater than 3), the control unit 11 may execute a unit process, which is a process including at least two of the first to M-th learnings, multiple times. More specifically, the unit process is a process that executes the m-th learning in order from m=p (a predetermined integer equal to or greater than 1 (M-1)) to m=p+q (q is a predetermined integer equal to or greater than 1 (Mp)). Therefore, in the unit process, (q+1) learnings from the p-th learning to the (p+q)-th learning are performed.

[0072] Therefore, for example, when M=3, the control unit 11 may alternately repeat at least two of the first learning to the third learning a predetermined number of times.

[0073] When a unit process is executed multiple times, the number of data sets used in each m-th learning within the unit process can be smaller than when the unit process is executed once. This produces the effect of executing (q+1) learnings simultaneously. This reduces the risk of forgetting earlier learnings due to later learnings.

[0074] When the control unit 11 has performed the first learning through the Nth learning, using the learning results obtained by the learning device 1 means using some or all of the first to Nth correspondence relationships. Therefore, when the control unit 11 has performed the first to Nth learning, the control unit 21 of the estimation device 2 may perform estimation using some or all of the first to Nth correspondence relationships in the estimation process.

[0075] The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0076] The estimation device 2 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0077] All or part of the functions of the information processing system 100, the learning device 1, and the estimation device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0078] The control unit 21 is an example of an estimation unit.

[0079] The first encoder, the second encoder, the third encoder, and the n-th encoder may be configured, for example, by a neural network. The outputs of the first encoder, the second encoder, the third encoder, and the n-th encoder are, for example, vectors.

[0080] The inference process may be, for example, a classification task, which may be, for example, zero-shot classification.

[0081] The information processing system 100, the learning device 1, and the estimation device 2 may be used, for example, to analyze sports videos. Furthermore, the information processing system 100, the learning device 1, and the estimation device 2 may be used, for example, to analyze customer behavior in a retail store.

[0082] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0083] 100...information processing system, 1...learning device, 2...estimation device, 11...control unit, 12...interface unit, 13...storage unit, 21...control unit, 22...interface unit, 23...storage unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. a control unit that executes the following in the order of first learning, second learning, and third learning: first learning for learning a correspondence relationship between a first quantity and a second quantity using a first set that is a set of a first quantity and a second quantity; second learning for learning a correspondence relationship between the second quantity and a third quantity using a second set that is a set of the second quantity and a third quantity; and third learning for learning a correspondence relationship between the third quantity and the fourth quantity using a third set that is a set of the third quantity and a fourth quantity; Equipped with the modalities of the first amount, the second amount, the third amount, and the fourth amount are different from each other; Learning device.

2. the first learning is learning using a pair of the first quantity and the second quantity, and is learning that updates the first encoder and the second encoder so as to reduce a difference between an output of a first encoder that converts the first quantity and an output of a second encoder that converts the second quantity; the second learning is learning using a pair of the second quantity and the third quantity, and is learning that updates the second encoder and the third encoder while freezing at least a part of the second encoder so as to reduce a difference between an output of the second encoder and an output of a third encoder that converts the third quantity; the third learning is learning using a pair of the third quantity and the fourth quantity, and is learning that updates the third encoder and the fourth encoder while freezing at least a part of the third encoder so as to reduce a difference between an output of the third encoder and an output of a fourth encoder that converts the fourth quantity; The learning device according to claim 1 .

3. the control unit further performs (N-3) learnings from fourth learning to Nth learning (N is a predetermined integer equal to or greater than 4) after the third learning, The n-th learning (n is an integer of 4 or more and N or less) is a process of learning the correspondence between the n-th quantity and the (n+1)-th quantity using the n-th set, which is a set of the n-th quantity and the (n+1)-th quantity. The learning device according to claim 1 .

4. The control unit executes a unit process multiple times in the learning from the first learning to the Mth learning (M is a predetermined integer of 3 or more), the unit process being a process including at least two of the first learning to the Mth learning, and sequentially executing the mth learning from m=p (a predetermined integer of 1 or more and (M-1) or less) to m=p+q (q is a predetermined integer of 1 or more and (M-p) or less). The learning device according to claim 3 .

5. the control unit alternately repeats at least two of the first learning to the third learning a predetermined number of times. The learning device according to claim 1 .

6. At least one of the first amount and the second amount is time series data. The learning device according to claim 1 .

7. a control unit that executes the first learning, the second learning, and the third learning in that order: first learning that uses a first set that is a set of a first amount and a second amount to learn a correspondence relationship between a first amount and a second amount; second learning that uses a second set that is a set of the second amount and a third amount to learn a correspondence relationship between the second amount and a third amount; and third learning that uses a third set that is a set of the third amount and a fourth amount to learn a correspondence relationship between the third amount and the fourth amount; wherein the modalities of the first amount, the second amount, the third amount, and the fourth amount are different from each other; and an estimation unit that performs estimation using a result of learning obtained by a learning device. An estimation device comprising:

8. a control unit that executes the first learning, the second learning, and the third learning in that order, of learning a correspondence relationship between a first quantity and a second quantity using a first set that is a set of a first quantity and a second quantity; second learning, the second learning, the third learning, the third learning, the third learning, the third learning, the first quantity, the second quantity, the third quantity, and the fourth quantity, wherein the modalities of the first quantity, the second quantity, the third quantity, and the fourth quantity are different from each other; a first learning step in which the control unit performs the first learning; a second learning step in which the control unit performs the second learning; a third learning step in which the control unit performs the third learning; A learning method that has

9. a control unit that performs first learning, second learning, and third learning in that order, of a first learning that learns a correspondence relationship between a first quantity and a second quantity using a first set that is a set of a first quantity and a second quantity; a second learning that learns a correspondence relationship between the second quantity and a third quantity using a second set that is a set of the second quantity and a third quantity; and a third learning that learns a correspondence relationship between the third quantity and the fourth quantity using a third set that is a set of the third quantity and a fourth quantity, wherein the first quantity, the second quantity, the third quantity, and the fourth quantity have modalities different from one another, and the estimation method is performed by an estimation device that includes an estimation unit that performs estimation using a result of learning obtained by the learning device, an estimation step in which the estimation unit performs the estimation; An estimation method having:

10. A program for causing a computer to function as the learning device according to any one of claims 1 to 6.

11. A program for causing a computer to function as the estimation device according to claim 7.

Images