Learning device, learning method, and learning program

Abstract

This learning device receives input of information indicating the type of a unit of analysis in a machine learning model, and data extracted for each unit of analysis of said type. The learning device subsequently adds, to data extracted in a unit of analysis for each type of unit of analysis, embedded information indicating the type of the unit of analysis. The learning device trains the machine learning model by using data obtained by adding, to the data extracted for each unit of analysis, embedded information indicating the type of the unit of analysis. The learning device subsequently outputs a parameter of the trained machine learning model.

Description

Learning device, learning method, and learning program

[0001] The present invention relates to a learning device, a learning method, and a learning program for a machine learning model.

[0002] Conventionally, when modeling conversations using machine learning, a modeling method that is robust to the type of unit of analysis is required.

[0003] Here, conversation modeling refers to, for example, an approach to estimating attributes of the conversation itself and attributes of the conversation participants from data such as audio, video, and text in the conversation. The conversation data may also be, for example, multimodal data including the audio, video, and text of the conversation. Note that conversation modeling also includes an approach to restoring the multimodal data of the conversation itself using, for example, an autoencoder or a transformer.

[0004] A unit of analysis is a category that defines a time interval that separates data and is necessary for instantiating conversational data. Typical units of analysis include utterances and turns, and instances extracted based on these units of analysis are used to train the model.

[0005] A technique for estimating paragraph boundaries in dialogue sequence data containing multiple topics has been proposed (see Patent Document 1).

[0006] Patent No. 7425368

[0007] However, because the above techniques limit the unit of analysis to speech, it is not possible to input data with various types of data segments into the model, and the model is not robust to the types of data segments.

[0008] Therefore, the present invention aims to solve the above-mentioned problems and to construct a model that can input data with various types of data delimiters and is robust to the type of data delimiter (the performance of the model does not deteriorate regardless of the type of data delimiter).

[0009] In order to solve the above-mentioned problems, the present invention is characterized by comprising a data input unit that accepts input of information indicating the type of unit of analysis in a machine learning model and data extracted for each unit of analysis of the type, an addition unit that adds embedded information indicating the type of unit of analysis to the data extracted for the unit of analysis, and a model learning unit that learns the machine learning model using data in which embedded information indicating the type of unit of analysis has been added to the data extracted for each unit of analysis.

[0010] According to the present invention, it is possible to input data with various types of data delimiters, and to construct a model that is robust to the types of data delimiters.

[0011] Fig. 1 is a diagram for explaining the underlying technology of a learning device. Fig. 2 is a diagram for explaining an example of processing executed by the learning device. Fig. 3 is a diagram showing an example of the configuration of a learning device. Fig. 4 is a flowchart showing an example of processing procedures executed by the learning device. Fig. 5 is a diagram showing an example of a computer that executes a learning program.

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0013] [Base Technology] First, the base technology of the learning device of this embodiment will be described using Figure 1. Data d shown in Figure 1 is time-series data showing the state of a conversation between conversation members a, b, ..., z. In Figure 1, rectangles with letters written on them represent the utterances of the members. In the following, an example will be described in which the type of unit (u) of analysis of the machine learning model to be learned by the learning device is utterance (division unit number = k).

[0014] Figure 1 shows the extracted data d from data d. i The extracted data d i The extraction interval is defined by the unit of analysis of the machine learning model to be learned by the learning device. In this example, the unit of analysis is an utterance. Therefore, the learning device calculates the start time of each utterance (k) of the user's conversation (interval start time s i k ) and end time (interval end time ei k ) Based on the extracted data d i Set the extraction interval.

[0015] The unit of analysis (u) can be anything other than an utterance, such as a minute period of time equivalent to one frame of video, a turn of speech, five seconds after an utterance, a randomly sampled three-second interval, etc. All of these must be used appropriately depending on the target of analysis for the machine learning model.

[0016] For example, if the task of a machine learning model is to estimate the intention of a user's speech, the unit of analysis is generally the speech.Also, if the task of a machine learning model is to recognize a user's facial expression, the unit of analysis is generally a very short period of time equivalent to one frame of video.

[0017] Different units of analysis require different focus points in the data and different quantities and qualities of information for task execution. For this reason, in the past, different units of analysis were not simultaneously trained in conversation modeling.

[0018] [Overview] On the other hand, the learning device of this embodiment learns different units of analysis (analysis units) collectively in learning a machine learning model.

[0019] For example, the learning device extracts data from the data for training a machine learning model in a data extraction section of an arbitrary analysis unit, and adds embedding of information indicating the type of analysis unit (AU (Analyze Unit) embedding) to the extracted data. This allows the learning device to embed information indicating the type of analysis unit in the data for training a machine learning model.

[0020] As a result, even if a learning device trains a machine learning model using data extracted from multiple types of analytical units, AU embedding absorbs the differences between the data from different analytical units, making it possible to build a machine learning model that is robust against the types of analytical units. Furthermore, because the learning device can use data extracted from a variety of analytical units to train a machine learning model, it is possible to improve the amount of data used for training and build a machine learning model with higher performance.

[0021] The input data to the machine learning model may be any data. For example, the input data may be any or a combination of audio signals, facial images, or transcripts of speech from each or all participants in the conversation. Deep learning or other techniques are used to train the machine learning model.

[0022] Next, an example of the processing executed by the learning device will be described with reference to Fig. 2. For example, inputs to the learning device include conversation data, the type of analysis unit, and the start and end times of the data for that analysis unit.

[0023] Here, we assume that there are k∈N (N: natural number) types of analysis units u. The learning device calculates the starting time s of the i∈N-th data extraction interval of the k-th analysis unit. i k ∈R (R: real number, ≧0) and end time e i k Using ∈R, extract data d from data d i Extract.

[0024] Then, the learning device extracts the extracted data d i For example, the learning device vectorizes the extracted data d i By vectorizing it, it is embedded into an n∈N-dimensional data vector.

[0025] The learning device also embeds (emb) the number k of the analysis unit u into an n-dimensional vector (AU embedding, analysis unit vector) that indicates the type of analysis unit. Embedding (emb) is, for example, a process of replacing any integer with an n-dimensional vector.

[0026] The learning device inputs the sum of the data vector and the analytical unit vector into a model (machine learning model) and performs model learning. The model can be any model, such as a neural network or SVM (Support Vector Machine).

[0027] Furthermore, when the learning device inputs multimodal data to the model, it executes a series of processes shown by reference numeral 201 in Figure 2 in parallel for each piece of multimodal data. The learning device then fuses, for example, the sum of the data vector and the analytical unit vector for each piece of multimodal data using a multimodal fusion method, and then inputs the result to the model. Examples of multimodal fusion methods include early fusion and vector concatenation.

[0028] By having the learning device perform the above processing, it is possible to construct a model that can perform tasks based on various analytical units for multimodal data.

[0029] In conventional methods, model training targets only one analytical unit, and therefore does not take into account, for example, the path (see reference numeral 202) that creates the analytical unit vector shown in Fig. 2. On the other hand, the training device of this embodiment also includes a path (see reference numeral 202) that creates the analytical unit vector in model training. As a result, when training a model using data extracted via the path indicated by reference numeral 201, the training device can perform weighting using the information (analysis unit vector) obtained via the path indicated by reference numeral 202.

[0030] For example, consider a case where a learning device learns a model using data from two types of analysis units (analysis units A and B). Here, assume that the data from analysis unit A always contains significant data, and the data from analysis unit B always contains faint data.

[0031] In this case, if the learning device were to assign the same weight to the data from the two units of analysis during model training, there is a possibility that the data from unit of analysis B would not have much of an impact on the model. Here, if the weights for the data from unit of analysis A and the data from unit of analysis B can be appropriately assigned during model training, for example, by weighting the data from unit of analysis B heavily and weighting the data from unit of analysis A lightly, then even when training a model using data from two different units of analysis, it is believed that the data from both units of analysis will work well on the model.

[0032] Therefore, in order to create weights that match the type of unit of analysis, the learning device performs a process of embedding information that indicates the type of unit of analysis, as shown by reference numeral 202 in Figure 2. As a result, if the model to be learned is a model based on a neural network, for example, parameters are adjusted to match the type of unit of analysis during the model learning process, i.e., appropriate weights are created. This allows the learning device to input data for various types of unit of analysis and to build a model that is robust to the type of unit of analysis.

[0033] [Configuration Example] Next, a configuration example of the learning device 10 will be described with reference to Fig. 3. The learning device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13.

[0034] The input / output unit 11 is an interface that controls the input and output of various data. For example, the input / output unit 11 receives input of data (data d) for model learning.

[0035] The storage unit 12 stores data, programs, etc. that are referenced when the control unit 13 executes various processes. The storage unit 12 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 12 stores data (data d) for model learning received by the input / output unit 11, parameters of the model obtained by the control unit 13 learning the model, etc.

[0036] The control unit 13 is responsible for overall control of the learning device 10. The functions of the control unit 13 are realized, for example, by a CPU (Central Processing Unit) executing a program stored in the storage unit 12.

[0037] The control unit 13 includes, for example, a learning unit 131 that learns a model. The learning unit 131 includes a data input unit 132, a data extraction unit 133, a vectorization unit 134, an embedding unit 135, an addition unit 136, and a model learning unit 137. An estimation unit 138, indicated by a dashed line, may or may not be provided, and cases in which it is provided will be described later.

[0038] The data input unit 132 receives input of data for model training. The data input unit 132 also receives input of information indicating the type of analysis unit in the model and information (start time and end time) indicating the extraction interval for data of that type of analysis unit.

[0039] The data extraction unit 133 extracts data for each analysis unit from the model training data based on the information indicating the data extraction interval for each type of analysis unit. For example, if the types of analysis units are analysis unit A and analysis unit B, the data extraction unit 133 extracts data for analysis unit A and data for analysis unit B from the model training data.

[0040] The vectorization unit 134 vectorizes the data extracted by the data extraction unit 133. For example, the vectorization unit 134 converts the data extracted by the data extraction unit 133 into an n-dimensional vector (data vector). For example, if the data is audio data, the vectorization unit 134 converts the data into a vector using a pre-training model such as wav2vec or HuBERT, or a toolkit such as OpenSMILE.

[0041] The embedding unit 135 performs an embedding process of the information indicating the type of analysis unit received by the data input unit 132. For example, the embedding unit 135 embeds the information indicating the type of analysis unit into an n-dimensional vector (analysis unit vector) using an embedding layer of a neural network, for example.

[0042] The adder 136 calculates the sum of the data vector and the analysis unit vector for each type of analysis unit. The model training unit 137 inputs the sum of the data vector and the analysis unit vector for each type of analysis unit into the model and trains the model. The model training unit 137 then stores the model parameters obtained by training in the storage unit 12.

[0043] This allows the learning device 10 to build a model that is robust to a variety of analysis units.

[0044] [Example of Processing Procedure] Next, an example of processing procedure executed by the learning device 10 will be described with reference to Fig. 4. First, the data input unit 132 of the learning device 10 accepts input of data for model training, information indicating the type of analysis unit, and information indicating the extraction interval of data for the analysis unit (S1).

[0045] After S1, the learning device 10 executes the following processes shown in S2 to S5 for each type of analysis unit received in S1.

[0046] First, the data extraction unit 133 extracts data for each analysis unit from the model training data based on information indicating the extraction interval of data for each analysis unit received in S1 (S2). Then, the vectorization unit 134 vectorizes the data extracted in S2 (S3). Furthermore, the embedding unit 135 performs an embedding process for information indicating the type of analysis unit received in S1 (S4). Then, the addition unit 136 adds the data vectorized in S3 and the result of the embedding process in S4 (analysis unit vector) (S5).

[0047] When the learning device 10 has performed the processes of S2 to S5 for all types of analysis units received in S1, the model learning unit 137 inputs the result of adding the vectorized data and the analysis unit vector into the model and performs model learning (S6).

[0048] By executing the above process, the learning device 10 can input data of various types of analysis units and can build a model that is robust to the types of analysis units.

[0049] Other Embodiments The learning device 10 may perform an estimation process for input data using the model learned by the above process. In this case, the learning device 10 further includes an estimation unit 138 shown in Fig. 3. The estimation unit 138 includes a data input unit 139 and an estimation processing unit 140.

[0050] The data input unit 139 accepts input data (data to be estimated and information indicating the type of analysis unit) to the trained model.

[0051] The type of analysis unit to be input is at least one of the types of analysis unit used in model training. For example, if the types of analysis unit used in model training are A, B, and C, then any one of A, B, and C or a combination of these can be input.

[0052] The estimation processing unit 140 inputs the data to be estimated and information indicating the type of analysis unit into the trained model, and performs estimation of the data using the analysis unit using the model.The estimation processing unit 140 then outputs the estimation result.

[0053] For example, consider a case where the learning unit 131 learns a model using analytical units A, B, and C, and the data input unit 139 receives input of data to be estimated and information indicating analytical units A and C. In this case, the analysis processing unit 140 inputs the data to be estimated and information indicating analytical units A and C to the model, and estimates data using analytical units A and C using the model. Then, the estimation processing unit 140 outputs the estimation result.

[0054] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0055] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0056] [Program] The learning device 10 can be implemented by installing a program (learning program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the learning device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0057] 5 is a diagram showing an example of a computer that executes a learning program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0058] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0059] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the learning device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the learning device 10 is stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0060] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0061] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0062] REFERENCE SIGNS LIST 10 Learning device 11 Input / output unit 12 Memory unit 13 Control unit 131 Learning unit 132, 139 Data input unit 133 Data extraction unit 134 Vectorization unit 135 Embedding unit 136 Addition unit 137 Model learning unit 138 Estimation unit 140 Analysis processing unit

Claims

1. A learning device comprising: a data input unit that accepts input of information indicating the type of unit of analysis in a machine learning model and data extracted for each unit of analysis of said type; an addition unit that adds embedded information indicating the type of unit of analysis to the data extracted for said unit of analysis; and a model learning unit that trains the machine learning model using data in which embedded information indicating the type of unit of analysis has been added to the data extracted for each unit of analysis.

2. The learning device according to claim 1, characterized in that the data input unit receives input of data extracted from multimodal data for each unit of analysis of the type.

3. The learning device according to claim 1, further comprising a vectorization unit that vectorizes the data extracted in units of analysis, wherein the addition unit adds the embedding information having the same number of dimensions as the vectorized data to the vectorized data.

4. The learning device described in claim 1, further comprising an estimation unit that outputs an estimation result for the data using the unit of analysis of the type by inputting data to be estimated and information indicating the type of unit of analysis of the data into the machine learning model after learning.

5. A learning method executed by a learning device, comprising: a step of accepting input of information indicating the type of unit of analysis in a machine learning model and data extracted for each unit of analysis of the type; a step of adding embedded information indicating the type of unit of analysis to the data extracted for the unit of analysis; and a step of training the machine learning model using data in which embedded information indicating the type of unit of analysis has been added to the data extracted for each unit of analysis.

6. A learning program for causing a computer to execute the following steps: accepting input of information indicating the type of unit of analysis in a machine learning model and data extracted for each unit of analysis of said type; adding embedded information indicating the type of unit of analysis to the data extracted for said unit of analysis; and training the machine learning model using data in which embedded information indicating the type of unit of analysis has been added to the data extracted for each unit of analysis.

Images