Learning device, information processing device, learning method, and program

Abstract

To improve cognitive performance of an emotion recognition model.SOLUTION: A learning device 1 of an information processing system includes: a first learning unit 122 which, when first vector data with which a prescribed label is associated is inputted to a first neural network constituted of a plurality of layers, learns parameters of the respective layers of the first neural network so as to output the label associated with the first vector data; a setting unit 124 which sets the parameters of the respective layers constituting the first neural network, which are learned by the first learning unit 122, to respective layers of a second neural network constituted of a plurality of layers; and a second learning unit 123 which, when second vector data with which a label identical to the label associated with the first vector data is associated is inputted to the second neural network, learns parameters of the respective layers of the second neural network so as to output the label with which the second vector data is associated.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a learning device, an information processing device, a learning method, and a program. [Background technology]

[0002] Conventionally, technologies have been proposed for recognizing the mood of people participating in a communication scene. For example, Non-Patent Document 1 discloses a technology for extracting features such as face, pose, gesture, and gaze from an image and realizing a group-level emotion recognition model using a known deep learning method. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Xin Guo, Bin Zhu, Luisa F. Polania, Charles Boncelet, and Kenneth E. Barner. 2018. Group-Level Emotion Recognition using Hybrid Deep Models based on Faces, Scenes, Skeletons and Visual Attentions. ACM International Conference on Multimodal Interaction 2018. Summary of the Invention [Problem to be solved by the invention]

[0004] The above technology attempts to realize mood recognition by extracting visual features contained in an image, constructing multiple models, and fusing the results of recognition by those models. However, it can be difficult to ensure practical recognition performance using only features extracted from one modal type of data such as an image (for example, facial features or facial expression features).

[0005] The present invention has been made in view of these points, and aims to improve the recognition performance of a model. [Means for solving the problem]

[0006] A learning device according to a first aspect of the present invention includes a first learning unit that learns parameters of each layer constituting a first neural network consisting of multiple layers so that when first vector data associated with a predetermined label is input into the first neural network, the first neural network outputs the label associated with the first vector data; a setting unit that sets the parameters of each layer constituting the first neural network learned by the first learning unit to each layer constituting a second neural network consisting of multiple layers; and a second learning unit that learns parameters of each layer constituting the second neural network so that when second vector data associated with the same label as the label associated with the first vector data is input into the second neural network, the second neural network outputs the label associated with the second vector data.

[0007] The learning device may further include a modal data acquisition unit that acquires modal data to which the label is associated, a first conversion unit that converts the modal data into the first vector data that expresses modal features of a type different from the modal type corresponding to the modal data, and a second conversion unit that converts the modal data into the second vector data that expresses modal features corresponding to the modal data.

[0008] The first conversion unit may convert the modal data into a plurality of first vector data each having a different modal type, and the learning device may further have a learning control unit that causes the first learning unit, the setting unit, and the second learning unit to execute in sequence for each of the first vector data converted by the first conversion unit, and the learning control unit may cause the setting unit to execute a process of setting the parameters of each layer constituting the second neural network learned by the second learning unit to each layer constituting the first neural network after executing the process performed by the second learning unit and before executing the process performed by the first learning unit.

[0009] The learning control unit may execute each process in sequence for each of the first vector data converted by the first conversion unit, and then execute each process in sequence again for each of the first vector data converted by the first conversion unit.

[0010] An information processing device according to a second aspect of the present invention includes a memory unit that stores parameters of each layer constituting the second neural network generated by the learning device as a learning model, a homogeneous data acquisition unit that acquires homogeneous data of the same type in modal as the modal data, and a model application unit that outputs a label corresponding to the homogeneous data by applying the learning model stored in the memory unit to the homogeneous data acquired by the homogeneous data acquisition unit.

[0011] A learning method according to a third aspect of the present invention is executed by a computer and includes the steps of: learning parameters of each layer constituting a first neural network consisting of multiple layers so that when first vector data associated with a predetermined label is input into the first neural network, the first neural network outputs the label associated with the first vector data; setting the learned parameters of each layer constituting the first neural network to each layer constituting a second neural network consisting of multiple layers; and learning parameters of each layer constituting the second neural network so that when second vector data associated with the same label as the label associated with the first vector data is input into the second neural network, the second neural network outputs the label associated with the second vector data.

[0012] A fourth aspect of the present invention provides a program that causes a computer to function as a first learning unit that learns parameters of each layer of a first neural network consisting of multiple layers so that when first vector data associated with a predetermined label is input into the first neural network, the first neural network outputs the label associated with the first vector data; a setting unit that sets the parameters of each layer of the first neural network learned by the first learning unit to each layer of a second neural network consisting of multiple layers; and a second learning unit that learns parameters of each layer of the second neural network so that when second vector data associated with the same label as the label associated with the first vector data is input into the second neural network, the second neural network outputs the label associated with the second vector data. [Effects of the Invention]

[0013] The present invention provides an advantage in that it is possible to improve the recognition performance of a model. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of a learning device. [Figure 3] FIG. 1 is a diagram schematically illustrating a functional configuration of an information processing device. [Figure 4] FIG. 2 is a diagram schematically illustrating control of processing by a learning control unit. [Figure 5] 10 is a flowchart illustrating the flow of a learning process executed by the learning device. DETAILED DESCRIPTION OF THE INVENTION

[0015] [Outline of Information Processing System S] 1 is a diagram illustrating an overview of an information processing system S. The information processing system S is a system used to recognize predetermined modal data. The predetermined modal data is, for example, data indicating a state or behavior (e.g., atmosphere) between multiple people, such as image data including multiple people as subjects, audio data including the voices of multiple people, character string data expressing the state or behavior between multiple people in text, and biometric data of each of multiple people detected by a sensor.

[0016] "Recognition of predetermined modal data" is, for example, recognition of the atmosphere between multiple people. Atmosphere is, for example, a concept that indicates whether the atmosphere between subjects included in image data is good or bad. "Recognition of predetermined modal data" is not limited to this, and may also be voice recognition, image recognition, emotion recognition, etc. The information processing system S has a learning device 1 and an information processing device 2.

[0017] The learning device 1 is a computer that generates a learning model used to recognize predetermined modal data. The learning model is a machine learning model that uses a neural network consisting of multiple layers to output a label corresponding to vector data when vector data expressing the characteristics of a predetermined modal is input. The predetermined modal may be, for example, an image, a sound, a character string, or sensing data.

[0018] The information processing device 2 is a computer that executes a recognition process for recognizing predetermined modal data. The information processing device 2 executes the recognition process using a learning model that reflects knowledge of a plurality of different modalities.

[0019] When generating a learning model, the learning device 1 uses, as training data, data in which multiple vector data, each having a different modal type, are associated with a label (a label common to the multiple vector data). The multiple vector data include, for example, vector data expressing the characteristics of image data including multiple people as subjects, vector data expressing the characteristics of audio data of the subjects included in the image data, vector data expressing the characteristics of explanatory text that describes the state or behavior of the subject included in the image data, and vector data expressing the characteristics of biometric data of the subject included in the image data. The label indicates, for example, the atmosphere obtained from the vector data expressing the characteristics of image data including multiple people as subjects.

[0020] "A description that describes the state or behavior of a subject included in image data" is a sentence that describes a subject (including subjects other than people) included in image data, and includes at least a noun and a verb. Specific examples of a description include "two men are arguing," "three women are drinking tea in a cafe," and "a graph is drawn on the whiteboard in the conference room."

[0021] By using this teacher data, the learning device 1 can generate a learning model that reflects knowledge of a plurality of vector data each having a different modal type. Below, an overview of the processing executed by the learning device 1 will be described with reference to FIG.

[0022] In FIG. 1, training data T (first training data T1 and second training data T2) is training data used for learning.

[0023] The first training data T1 is data in which first vector data V1 expressing features of a first modal (e.g., speech) is associated with a label L. The first vector data V1 is, for example, vector data expressing features of a different type of modal from the modal data to be recognized in the recognition process executed by the information processing device 2.

[0024] The second training data T2 is data in which second vector data V2 expressing features of a second modal (e.g., an image) of a type different from the first modal is associated with a label L that is the same as the label L included in the first training data T1. The second vector data V2 is, for example, vector data expressing features of a modal of the same type as the modal data to be recognized in the recognition process executed by the information processing device 2.

[0025] For example, the label L is expressed as a five-level number from 1 to 5, with the higher the number, the better the atmosphere. In the training data T shown in Figure 1, the label L indicating the atmosphere is "5," indicating a good atmosphere.

[0026] The first neural network N1 is a neural network for outputting a label L included in the first teacher data T1 that is the basis of the first vector data V1 when first vector data V1 is input. The second neural network N2 is a neural network for outputting a label L included in the second teacher data T2 that is the basis of the second vector data V2 when second vector data V2 is input. The second neural network N2 is a neural network used in the recognition processing executed by the information processing device 2. The first neural network N1 and the second neural network N2 are neural networks composed of multiple layers including an input layer, an output layer, and a hidden layer, and are neural networks composed of multiple layers in common.

[0027] Although detailed description will be omitted as this is a known technique, the learning device 1 learns the parameters constituting the first neural network N1 by backpropagating the error between the output when first vector data V1 is input to the first neural network N1 and the label L included in the training data T. The same applies to learning of the second neural network N2.

[0028] In a learning model, the recognition performance can be improved when knowledge based on multiple different types of modalities is reflected rather than when knowledge based on one type of modal is reflected. Therefore, after executing the learning process for the first neural network N1, the learning device 1 sets the parameters of each layer constituting the first neural network N1 to each layer constituting the second neural network N2, and then executes the learning process for the second neural network N2.

[0029] As described above, the first neural network N1 is trained so as to output a label L corresponding to the first vector data V1 when the first vector data V1 is input. In other words, the first neural network N1 can be said to be a neural network that learns information or knowledge based on the first modal, and therefore, the parameters of each layer constituting the first neural network N1 are considered to include knowledge based on the first modal.

[0030] Therefore, by setting the parameters of each layer constituting the first neural network N1 to each layer constituting the second neural network N2, knowledge based on the first modal can be reflected in the second neural network N2. Subsequently, the second neural network N2 is trained to output the label L associated with the second vector data V2 when the second vector data V2 is input, thereby further reflecting the knowledge based on the second modal in the parameters of each layer constituting the second neural network N2. This allows the learning model of the second neural network N2 to recognize vector data of one modal type with an unknown label based on knowledge of multiple vector data of different modal types. As a result, the learning device 1 can improve the recognition performance of the learning model.

[0031] [Configuration of learning device 1] FIG. 2 is a diagram showing a schematic functional configuration of the learning device 1. The learning device 1 includes a memory unit 11 and a control unit 12. In FIG. 2, arrows indicate the main data flows, and there may be data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a functional unit configuration, not a hardware (device) unit configuration. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

[0032] The memory unit 11 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the learning device 1, a RAM (Random Access Memory) that serves as the working area of ​​the learning device 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referenced when the application programs are executed.

[0033] The control unit 12 is a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the learning device 1, and executes a program stored in the storage unit 11 to function as a learning control unit 121, a first learning unit 122, a second learning unit 123, a setting unit 124, a modal data acquisition unit 125, a first conversion unit 126, and a second conversion unit 127. The functions of the learning control unit 121, the first learning unit 122, the second learning unit 123, the setting unit 124, the modal data acquisition unit 125, the first conversion unit 126, and the second conversion unit 127 will be described in detail later.

[0034] 2 shows an example in which the learning device 1 is configured as a single device. However, the learning device 1 may also be implemented using multiple processors, memories, and other computing resources, such as a cloud computing system. In this case, each component of the control unit 12 is implemented by at least one of the multiple different processors executing a program.

[0035] [Configuration of information processing device 2] FIG. 3 is a diagram schematically illustrating the functional configuration of an information processing device 2. The information processing device 2 includes a storage unit 21 and a control unit 22. In FIG. 3, arrows indicate main data flows, and there may be data flows not shown in FIG. 3. In FIG. 3, each functional block indicates a configuration in functional units, rather than a configuration in hardware (device) units. Therefore, the functional blocks shown in FIG. 3 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

[0036] The memory unit 21 is a large-capacity storage device such as a ROM that stores the BIOS of the computer that realizes the information processing device 2, a RAM that serves as the working area of ​​the information processing device 2, an HDD or SSD that stores the OS, application programs, and various information referenced when the application programs are executed.

[0037] The control unit 22 is a processor such as a CPU or GPU of the information processing device 2, and functions as a homogeneous data acquisition unit 221 and a model application unit 222 by executing a program stored in the storage unit 21. The functions of the homogeneous data acquisition unit 221 and the model application unit 222 will be described in detail later.

[0038] 3 shows an example in which the information processing device 2 is configured as a single device. However, the information processing device 2 may be realized by multiple processors, memories, and other computing resources, such as a cloud computing system. In this case, each unit constituting the control unit 22 is realized by at least one of multiple different processors executing a program.

[0039] [Learning process] First, a description will be given of the learning process executed by the learning device 1. The learning control unit 121 controls the process executed by the first learning unit 122, the process executed by the second learning unit 123, and the process executed by the setting unit .

[0040] The first learning unit 122 learns the first neural network N1 using the first teacher data T1. Specifically, when first vector data V1 associated with a label L in the first teacher data T1 is input to the first neural network N1, the first learning unit 122 learns parameters of each layer constituting the first neural network N1 so as to output the label L associated with the first vector data V1.

[0041] The second learning unit 123 learns the second neural network N2 using the second teacher data T2. Specifically, when second vector data V2 associated with the same label L as the label L associated with the first vector data V1 in the second teacher data T2 is input to the second neural network N2, the second learning unit 123 learns parameters of each layer constituting the second neural network N2 so as to output the label L associated with the second vector data V2.

[0042] Here, the second learning unit 123 uses the parameters of each layer constituting the first neural network learned by the first learning unit 122 as initial parameters to learn the parameters of each layer constituting the second neural network N2. Specifically, first, the setting unit 124 sets the parameters of each layer constituting the first neural network N1 learned by the first learning unit 122 to each layer constituting the second neural network N2. Then, after the parameters have been set to each layer constituting the second neural network N2, the second learning unit 123 learns the parameters of each layer constituting the second neural network N2 so that when second vector data V2 is input to the second neural network N2, the second neural network N2 outputs a label L associated with the second vector data V2.

[0043] The first vector data V1 and the second vector data V2 are, for example, vector data converted from the same modal data. The modal corresponding to the first vector data V1 and the modal data or the modal corresponding to the second vector data V2 are different types of modal.

[0044] Specifically, first, the modal data acquisition unit 125 acquires modal data associated with the label L. Then, the first conversion unit 126 converts the modal data acquired by the modal data acquisition unit 125 into first vector data V1 that expresses the characteristics of a modal type different from the type of modal corresponding to the modal data.

[0045] For example, if the modal data is an image, the first conversion unit 126 converts it into first vector data V1 that represents the features of a different type of modal (e.g., voice, character string, or sensing) different from an image. The first conversion unit 126 converts the modal data into the first vector data V1 using, for example, a known first converter (e.g., a model such as CLIP (Contrastive Language-Image Pre-Training) or DALL·E).

[0046] The modal corresponding to the second vector data V2 and the modal corresponding to the modal data are of the same type. Specifically, the second conversion unit 127 converts the modal data acquired by the modal data acquisition unit 125 into second vector data V2 that expresses the characteristics of the modal corresponding to the modal data. For example, if the modal of the modal data is an image, the second conversion unit 127 converts it into second vector data V2 that expresses the characteristics of the image. The second conversion unit 127 converts the modal data into second vector data V2 using, for example, a known second converter (for example, a model such as VGG (Visual Geometry Group)). In this way, the learning device 1 can create multiple vector data each having a different modal type.

[0047] In the above, an example has been described in which learning device 1 converts one acquired piece of modal data into a plurality of vector data each having a different modal type, but this is not limiting. For example, learning device 1 may convert a plurality of pieces of modal data each having a different modal type into a plurality of corresponding vector data.

[0048] Specifically, first, the modal data acquisition unit 125 acquires a plurality of vector data each of which is a plurality of modal data associated with a common label L and has a different modal type. Then, the first conversion unit 126 converts the plurality of modal data acquired by the modal data acquisition unit 125 into first vector data V1 expressing modal features corresponding to modal data of a different type from the modal data to be recognized in the recognition processing executed by the information processing device 2. Furthermore, the second conversion unit 127 converts the plurality of modal data acquired by the modal data acquisition unit 125 into second vector data V2 expressing modal features corresponding to modal data of the same modal type as the modal data to be recognized in the recognition processing executed by the information processing device 2.

[0049] For example, when the modal data acquisition unit 125 acquires image data, voice data, and character string data as multiple modal data, and the modal data to be recognized in the recognition process executed by the information processing device 2 is an image, the first conversion unit 126 converts the voice data into first vector data V1 expressing features of the voice, and converts the character string data into first vector data V1 expressing features of the character string. Furthermore, the second conversion unit 127 converts the image data into second vector data V2 expressing features of the image.

[0050] The learning device 1 may convert any of a plurality of modal data, each having a different modal type, into first vector data V1 expressing features of the other type of modal. For example, when the modal data acquisition unit 125 acquires image data and audio data as the plurality of modal data, the first conversion unit 126 first selects modal data from the image data and audio data to be used for conversion into the first vector data V1. Then, the first conversion unit 126 converts the selected modal data (e.g., image data) into first vector data V1 expressing features of a modal type other than an image (e.g., character strings). For example, the first conversion unit 126 selects, from the plurality of modal data, modal data of the same type as the modal type of modal data to be recognized in the recognition process executed by the information processing device 2, and converts the selected modal data into first vector data V1 expressing features of a modal type other than the selected modal data.

[0051] After the first conversion unit 126 and the second conversion unit 127 convert the first vector data V1 and the second vector data V2, respectively, the learning control unit 121 sequentially causes the first learning unit 122 to execute a process based on the first vector data V1 (first teacher data T1), the setting unit 124 to execute a process based on the second vector data V2 (second teacher data T2), and the second learning unit 123 to execute a process based on the second vector data V2 (second teacher data T2). The learning control unit 121 may cause the first learning unit 122 to execute a process based on the first vector data V1, the setting unit 124 to execute a process based on the second vector data V2, and the second learning unit 123 to execute a process based on the second vector data V1, each of which has a different modal type.

[0052] Specifically, first, the first conversion unit 126 converts the modal data into a plurality of first vector data V1 each having a different modal type. For example, the first conversion unit 126 converts the modal data into the first vector data V1 using a different first converter for each modal type.

[0053] Then, for each piece of first vector data V1 converted by first conversion unit 126, learning control unit 121 sequentially causes first learning unit 122 to execute a process based on the first vector data V1, setting unit 124 to execute a process based on second vector data V2, and second learning unit 123 to execute a process based on second vector data V2. After causing second learning unit 123 to execute the process, learning control unit 121 causes setting unit 124 to execute a process of setting the parameters of each layer constituting second neural network N2 learned by second learning unit 123 to each layer constituting first neural network N1 before causing first learning unit 122 to execute the process.

[0054] FIG. 4 is a diagram schematically illustrating the control of processing by the learning control unit 121. FIG. 4 illustrates an example in which the first conversion unit 126 converts first vector data V1a and first vector data V1b. The first training data T1a is data in which the first vector data V1a is associated with a label L. The first training data T1b is data in which the first vector data V1b is associated with a label L (the same label as the label L included in the first training data T1a and the label L included in the second training data T2). For example, the first vector data V1a is vector data representing features of a sound, the first vector data V1b is vector data representing features of a character string, and the second vector data V2 is vector data representing features of an image.

[0055] In this case, first, the learning control unit 121 sequentially executes the following processes based on the first teacher data T1a executed by the first learning unit 122 shown in Fig. 4(a), the process executed by the setting unit 124, and the process based on the second teacher data T2 executed by the second learning unit 123 shown in Fig. 4(b). The learning control unit 121 then executes the process of setting the parameters of each layer constituting the second neural network N2 after the second learning unit 123 has executed the process based on the second teacher data T2 to each layer constituting the first neural network N1b.

[0056] The learning control unit 121 then sequentially executes the processing based on the first teacher data T1b executed by the first learning unit 122 shown in Fig. 4(c), the processing executed by the setting unit 124, and the processing based on the second teacher data T2 executed by the second learning unit 123 shown in Fig. 4(d). In this way, the learning device 1 can reflect knowledge based on multiple first modals in the parameters of each layer constituting the second neural network N2.

[0057] The learning control unit 121 may repeatedly execute each process. Specifically, the learning control unit 121 executes each process in order for each piece of first vector data converted by the first conversion unit 126, and then executes each process in order again for each piece of first vector data converted by the first conversion unit 126.

[0058] In the example shown in Fig. 4, the learning control unit 121 causes the device 1 to execute each of the processes shown in Fig. 4(a) to 4(d), and then causes the device 1 to execute each of the processes shown in Fig. 4(a) to 4(d) again. At this time, the learning control unit 121 causes the setting unit 124 to execute a process of setting the parameters of each layer constituting the second neural network N2 shown in Fig. 4(d) after executing each of the processes shown in Fig. 4(a) to each layer constituting the first neural network N1a shown in Fig. 4(a), and then causes the device 1 to execute each of the processes shown in Fig. 4 again. In this way, the learning device 1 can improve the robustness of the recognition performance of the second neural network N2 by repeatedly executing each process.

[0059] [Recognition processing] Next, we will explain the recognition processing executed by the information processing device 2. In the information processing device 2, the storage unit 21 stores, as a learning model, parameters of each layer constituting the second neural network N2 generated by the above-mentioned learning device 1.

[0060] The same type of data acquisition unit 221 acquires the same type of data. The same type of data is data of the same modal type as the modal data acquired by the modal data acquisition unit 125 of the learning device 1.

[0061] The model application unit 222 applies the learning model stored in the storage unit 21 to the homogeneous data acquired by the homogeneous data acquisition unit 221, thereby outputting a label L corresponding to the homogeneous data. Specifically, the model application unit 222 first converts the homogeneous data acquired by the homogeneous data acquisition unit 221 into vector data using the second converter. Then, the model application unit 222 applies the learning model stored in the storage unit 21 to the converted vector data, thereby outputting a label L corresponding to the homogeneous data. This allows the information processing device 2 to recognize homogeneous data using a learning model that reflects knowledge of multiple different modals.

[0062] In the above, an example has been described in which the model application unit 222 applies the learning model to the homogeneous data to output the label L corresponding to the homogeneous data, but this is not limiting. For example, the model application unit 222 may apply the learning model to other data, the modal type of which is different from the modal data acquired by the modal data acquisition unit 125 of the learning device 1, to output the label L corresponding to the other data.

[0063] For example, first, the model application unit 222 uses the first converter to convert the other type of data into vector data of the same type of modal as the modal data acquired by the modal data acquisition unit 125 of the learning device 1. Then, the model application unit 222 applies the learning model stored in the storage unit 21 to the converted vector data, thereby outputting a label L corresponding to the other type of data.

[0064] Furthermore, for example, the model application unit 222 may use a second converter to convert the different type of data into vector data of the same type in modality, and apply the learning model to the converted vector data, thereby outputting a label L corresponding to the different type of data. Furthermore, for example, when the same type of data acquisition unit 221 acquires unknown data of unknown modality, the model application unit 222 first determines the modal type of the unknown data. The model application unit 222 determines the modal type of the unknown data, for example, using a known technique. Then, the model application unit 222 may convert the unknown data into vector data using a converter (first converter or second converter) corresponding to the determined modal type, and apply the learning model to the converted vector data, thereby outputting a label L corresponding to the unknown data.

[0065] [Learning process flow] Next, the flow of the learning process will be described. Fig. 5 is a flowchart for explaining the flow of the learning process executed by the learning device 1. The process in this flowchart starts, for example, when the learning device 1 is started.

[0066] The modal data acquisition unit 125 acquires modal data associated with a label L (S1). The learning device 1 converts the modal data into a plurality of vector data each having a different modal type (S2). Specifically, the first conversion unit 126 converts the modal data acquired by the modal data acquisition unit 125 into one or more first vector data V1 that expresses features of a modal type different from the modal type corresponding to the modal data. Furthermore, the second conversion unit 127 converts the modal data acquired by the modal data acquisition unit 125 into second vector data V2 that expresses features of the modal type corresponding to the modal data.

[0067] The learning control unit 121 controls the execution of the processes from S3 to S7. The first learning unit 122 learns the parameters of each layer constituting the first neural network N1 so that when first vector data V1 associated with a label L in the first teacher data T1 is input to the first neural network N1, the first neural network N1 outputs the label L associated with the first vector data V1 (S3).

[0068] The setting unit 124 sets the parameters of each layer constituting the first neural network N1 learned by the first learning unit 122 to each layer constituting the second neural network N2 (S4). The second learning unit 123 learns the parameters of each layer constituting the second neural network N2 so that when second vector data V2 associated with the same label L as the label L associated with the first vector data V1 in the second teacher data T2 is input to the second neural network N2, the second neural network N2 outputs the label L associated with the second vector data V2 (S5).

[0069] The learning control unit 121 determines whether learning has been completed for all vector data (S6). If the learning control unit 121 determines that learning has not been completed for all vector data (NO in S6), that is, if there is first vector data V1 that has not been used in learning, the learning control unit 121 causes the setting unit 124 to set the parameters of each layer that constitutes the second neural network N2 learned by the second learning unit 123 to each layer that constitutes the first neural network N1 (S7), and the process returns to S3. On the other hand, if the learning control unit 121 determines that learning has been completed for all vector data (YES in S6), that is, if there is no first vector data V1 that has not been used in learning, the process ends.

[0070] [Effects of Learning Device 1] As described above, the learning device 1 can reflect knowledge of multiple vector data sets with different modal types in a learning model of a neural network that receives vector data as input. This allows the learning model of the second neural network N2 to recognize vector data with an unknown label L based on knowledge of multiple vector data sets with different modal types. As a result, the learning device 1 can improve the recognition performance of the learning model.

[0071] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0072] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0073] 1 Learning device 11 Storage section 12 Control Unit 121 Learning control unit 122 First Learning Section 123 Second Learning Section 124 Settings 125 Modal Data Acquisition Unit 126 First Conversion Unit 127 Second Conversion Unit 2. Information processing equipment 21 Memory section 22 Control Unit 221 Homogeneous Data Acquisition Department 222 Model Application Section S Information Processing System

Claims

1. a first learning unit that learns parameters of each layer constituting a first neural network configured with a plurality of layers so that, when first vector data associated with a predetermined label is input to the first neural network, the first neural network outputs the label associated with the first vector data; a setting unit that sets the parameters of each layer that constitutes the first neural network learned by the first learning unit to each layer that constitutes a second neural network that is composed of a plurality of layers; a second learning unit that learns parameters of each layer constituting the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data; A learning device having the above configuration.

2. a modal data acquisition unit that acquires modal data associated with the label; a first conversion unit that converts the modal data into the first vector data that expresses a feature of a modal type different from the modal type corresponding to the modal data; a second conversion unit that converts the modal data into the second vector data that expresses modal characteristics corresponding to the modal data; further comprising The learning device according to claim 1 .

3. the first conversion unit converts the modal data into a plurality of first vector data each having a different modal type; a learning control unit that sequentially executes the process executed by the first learning unit, the process executed by the setting unit, and the process executed by the second learning unit for each of the first vector data converted by the first conversion unit; the learning control unit, after causing the second learning unit to execute the process and before causing the first learning unit to execute the process, causes the setting unit to execute a process of setting parameters of each layer constituting the second neural network learned by the second learning unit to each layer constituting the first neural network; The learning device according to claim 2 .

4. the learning control unit sequentially executes each process for each of the first vector data converted by the first conversion unit, and then sequentially executes each process again for each of the first vector data converted by the first conversion unit; The learning device according to claim 3 .

5. a storage unit that stores, as a learning model, parameters of each layer that constitutes the second neural network generated by the learning device according to any one of claims 2 to 4; a homogeneous data acquisition unit that acquires homogeneous data of the same type as the modal data; a model application unit that applies a learning model stored in the storage unit to the homogeneous data acquired by the homogeneous data acquisition unit, thereby outputting a label corresponding to the homogeneous data; An information processing device having the above.

6. The computer executes a step of learning parameters of each layer constituting a first neural network so that, when first vector data associated with a predetermined label is input to the first neural network composed of a plurality of layers, the first neural network outputs the label associated with the first vector data; a step of setting the parameters of each layer constituting the trained first neural network to each layer constituting a second neural network composed of a plurality of layers; learning parameters of each layer constituting the second neural network so that when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data; A learning method that has

7. Computer, a first learning unit that learns parameters of each layer constituting a first neural network configured from a plurality of layers when first vector data associated with a predetermined label is input to the first neural network so as to output the label associated with the first vector data; a setting unit that sets the parameters of each layer constituting the first neural network learned by the first learning unit to each layer constituting a second neural network composed of a plurality of layers; and a second learning unit that learns parameters of each layer constituting the second neural network so that, when second vector data associated with the same label as the label associated with the first vector data is input to the second neural network, the second neural network outputs the label associated with the second vector data; A program to function as a

Images