Classification device and classification method

Abstract

This classification device comprises: a first estimation unit that calculates a first estimation result that classifies a classification target into a prescribed class on the basis of a first model, using, as input, information identifying an attribute of the classification target; a first training unit that trains the first model using the first estimation result and a first correct label corresponding to the classification target; a second estimation unit that calculates a second estimation result that classifies the classification target into the prescribed class on the basis of a second model, using, as input, audio data and / or video data that are the classification target; and a second training unit that trains the second model using the first estimation result calculated on the basis of the trained first model, the second estimation result, and a second correct label corresponding to the classification target. The classification target is classified into the prescribed class on the basis of the trained second model using, as input, the audio data and / or video data that are the classification target.

Description

Classification device and classification method

[0001] The present invention relates to a classification device and a classification method.

[0002] There are techniques for classifying speakers using machine learning. For example, a technique is known in which a bias-only model is trained using only a portion of the input, and the prediction results of the bias-only model are used to control the training method of a robust model, thereby preventing overtraining of the model and improving classification accuracy with minority data, which tends to deteriorate (see, for example, Non-Patent Document 1).

[0003] In the technology disclosed in Non-Patent Document 1, when a question and an image are used as input and a robust model is trained to perform a task of multiple-choice answering to the question (classification task), the following (1) and (2) are implemented to prevent overtraining of the model: (1) The question is used as input to another model (Bias-only model) to train typical outputs of answers to the question; and (2) the output results of the Bias-only model are used when training the robust model.

[0004] Christopher Clark, Mark Yasar, and Luke Zettlemoyer, "Don't take the easy way out: Ensemble based methods for avoiding known dataset biases," in EMNLP-IJCNLP, 2019, pp. 4069-4082.

[0005] Consider a task of classifying classification targets (images in the prior art) into one of multiple predefined classes. In this case, the technology disclosed in Non-Patent Document 1 has a problem in that it cannot take into account bias caused by the annotators. For example, the technology disclosed in Non-Patent Document 1 cannot explicitly distinguish between majority data and minority data caused by multiple annotators. As a result, for example, multiple annotators may assign different correct labels to attributes of multiple classification targets, resulting in a bias in the number of positive examples for each attribute and each class of the classification target in the dataset, resulting in a problem of reduced classification accuracy for minority attributes or classes of classification targets with a small number of positive examples.

[0006] In order to solve the above-mentioned problems, a classification device according to an embodiment of the present invention includes a first estimation unit that receives, as input, information identifying an attribute of a classification target, and calculates a first estimation result in which the classification target is classified into a predetermined class based on a first model; a first learning unit that trains the first model using the first estimation result and a first correct label corresponding to the classification target; a second estimation unit that receives, as input, audio data and / or video data that is the classification target, and calculates, based on a second model, a second estimation result in which the classification target is classified into the predetermined class; and a second learning unit that trains the second model using the first estimation result calculated based on the trained first model, the second estimation result, and a second correct label corresponding to the classification target, and classifies the classification target into the predetermined class based on the trained second model.

[0007] According to an embodiment of the present invention, it is possible to suppress a decrease in classification accuracy in a task of classifying a classification target into one of a plurality of predefined classes.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of a classification system according to the present embodiment; FIG. 2 is a flowchart illustrating an example of processing of the classification system according to the present embodiment; FIG. 3 is a diagram illustrating an example of a predetermined label according to the present embodiment; FIG. 4 is a diagram illustrating a learning process of a second estimation unit according to the present embodiment; FIG. 5 is a diagram illustrating an estimation process according to the present embodiment; FIG. 6 is a diagram illustrating an example of an experimental result according to the present embodiment; FIG. 7 is a diagram illustrating an example of an embodiment of the present embodiment; and FIG.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] <Overview> (Background) Assume a task in which some classification target is classified into one of multiple predefined classes. For example, assume a task in which at least one of a speaker's audio and video is used as input and the task is to classify (estimate) each utterance into a class related to the speaker's conversational skills (e.g., whether or not the speaker is doing a good icebreaker, whether or not the speaker is nodding sufficiently, etc.). Here, a class refers to the type of classification in the classification task.

[0011] When building a machine learning model for such a task, training data is often first constructed by multiple annotators, in which correct labels corresponding to correct classes are assigned to the speech and video of a speaker, and the machine learning model is then trained using that training data. In this case, classes or speakers with many positive examples and, conversely, classes or speakers with few positive examples often exist in the training set, and while classification accuracy tends to be high for the majority classes or speakers, accuracy tends to deteriorate for the minority classes or speakers.

[0012] In order to avoid overfitting to the training set, the method disclosed in Non-Patent Document 1 trains a bias-only model using only a portion of the input, and uses the prediction results of the bias-only model to control the training method of the robust model. Here, the bias-only model is a model for capturing superficial features in the data, and learns the relationship between a portion of the input and the output.

[0013] (Problem) In datasets where correct labels are assigned by annotators as described above, the correct labels are assigned based on the annotator's subjective opinion, which significantly contributes to bias in the correct labels. For example, speakers with specific attributes (gender, appearance, speaking style, etc.) tend to be judged as having excessively high or low conversational skills by annotators with unique values ​​or confirmation bias. Furthermore, the values ​​that influence such tendency to assign correct labels vary depending on the annotator. While each annotator's values ​​or habits may contribute to biases for each class or speaker in the training data, conventional methods are unable to effectively capture each annotator's values ​​or habits, and therefore are unable to fully avoid overfitting to biases in the training data. This also poses the problem of reduced classification accuracy for minority speakers or classes with a small number of positive examples.

[0014] Therefore, in this embodiment, the above problem is solved by designing the bias-only model to learn the distribution of correct labels assigned by each annotator to a speaker.

[0015] <System Configuration> Fig. 1 is a diagram showing an example of the configuration of a classification system according to this embodiment. The classification system 1 is a system that classifies classification targets using machine learning. Here, the following description will be given assuming that the classification system 1 receives either the speaker's voice or video as input and classifies (estimates) the speaker's dialogue skill on an utterance-by-utterance basis.

[0016] In the example of Fig. 1, the classification system 1 is configured with a classification device 100. The classification device 100 is a device that classifies classification targets using machine learning. However, this is just an example, and the classification system 1 may be configured with multiple information processing devices. Here, the following description will be given assuming that the classification system 1 is configured with one classification device 100.

[0017] (Functional Configuration of Classification Device) The classification device 100 is, for example, an information processing device having a computer configuration, or a system including multiple computers. The classification device 100 realizes, for example, each functional configuration as shown in Fig. 1 by executing a predetermined program on one or more computers included in the classification device 100. In the example of Fig. 1, the classification device 100 has each functional configuration such as a first estimation unit 101, a first learning unit 102, a second estimation unit 103, a second learning unit 104, and a control unit 105.

[0018] The first estimation unit 101 receives input of information identifying a speaker (e.g., a speaker ID) and information identifying an annotator (e.g., an annotator ID), and executes a first estimation process to calculate a first estimation result (probability p1 for each predetermined class) classifying the speaker into a predetermined class based on a first model. However, information identifying the annotator may not be provided. Here, the first model is a neural network that receives input of a speaker ID identifying a speaker and an annotator ID identifying an annotator, performs processing based on first model parameters 111, and outputs a probability p1 for each predetermined class (first estimation result). However, as a minimum configuration, the input may be only the speaker ID. Note that the first model is, for example, a model for capturing superficial features in data, and is a bias-only model that learns the relationship between a portion of the input and the output. Note that the first estimation unit 101 may include the first model, or the first estimation unit 101 may be the first model. In this embodiment, the neural network is simply referred to as a "model," and the parameters of the neural network are referred to as "model parameters."

[0019] The predetermined classes are, for example, eight types of classes that represent the communication skills of the speaker. The eight types of classes include, for example, "consideration of the relationship," "good icebreaker," "selection of a good topic," "selection of a topic that suits the other person," "use of appropriate honorific language," "sufficient nodding," "empathy for the other person," and "sufficient efforts to persuade." However, the predetermined classes are not limited to the above eight types, and may be two or more other types of classes.

[0020] The speaker ID is an example of information for identifying an attribute of a classification target. The speaker ID may be any information that represents identification information of an attribute (such as a speaker) of the final classification target (audio and / or video), and may be other names such as a gender ID, an age ID, or a vehicle model ID depending on the classification target.

[0021] Although the present embodiment has been described using communication skill estimation as an example, the present embodiment can also be applied to other classification tasks that use voice and / or image input. For example, when applied to a voice emotion recognition task or a facial emotion recognition task, classes such as happiness, sadness, and anger may be used as the predetermined classes. In a gender estimation task, classes such as male and female may be used as the predetermined classes. In an age estimation task, classes such as teens, twenties, and so on may be used as the predetermined classes. Note that the classification target is not limited to humans, and may be other objects such as cars.

[0022] The first learning unit 102 executes a first learning process to learn (machine learn) a first model using the first estimation result by the first estimation unit 101 and a correct label (hereinafter referred to as a first correct label) corresponding to the classification target assigned by the annotator. For example, the first learning unit 102 calculates a loss using as input the probability p1 (first estimation result) for each predetermined class estimated by the first estimation unit 101 and the first correct label corresponding to the speaker ID and the annotator ID, and updates (learns) the first model parameters 111 based on the calculated loss (so as to reduce the loss). Note that in this embodiment, learning the first model parameters 111 may be simply referred to as "learning a first model."

[0023] Here, the first model parameters 111 are model parameters of a first model, which is, for example, a neural network. The first model processes input data based on the first model parameters 111 to obtain an output result.

[0024] For example, the first learning unit 102 calculates a binary cross entropy loss, which is a loss function used in the classification task, for each predetermined class. Note that in this embodiment, the binary cross entropy loss is used to perform binary classification for each class, but cross entropy loss may be used in the case of multi-class classification.

[0025] The method for training the first model may be the same as a general training method for training a classification model. As an example, the correct labels are the eight correct labels assigned by annotator A to speaker 1.

[0026] The second estimation unit 103 receives as input speech data and / or video data to be classified, and performs a second estimation process to calculate a second estimation result (probability p2) for classifying the classification target into a predetermined class and a score g based on the second model. Here, the second model is a neural network (Robust model) that receives a speaker's speech and / or video as input, performs processing based on the second model parameters 112, and outputs a second estimation result (probability p2 for each predetermined class) for classifying the speaker into a predetermined class and a score g. Note that the second estimation unit 103 may output only the probability p2 for each predetermined class, without outputting the score g. Note that the second estimation unit 103 may include the second model, or the second estimation unit 103 may be the second model.

[0027] The speech data and / or video data to be classified includes, for example, one or more of speech data of a speaker, video data of the speaker, and video data of a listener conversing with the speaker. The predetermined classes used are the same as those used in the first estimating unit 101. Note that, like the first estimating unit 101, the second estimating unit 103 can also be applied to various other tasks.

[0028] The second learning unit 104 executes a second learning process to learn a second model using the first estimation result of the learned first model, the estimation result of the second model (hereinafter referred to as the second estimation result), and the second correct label. For example, the second learning unit 104 calculates a loss using the first estimation result of the learned first model (e.g., probability p1 for each predetermined class), the second estimation result of the second model (e.g., probability p2 for each predetermined class), and the second correct label as input, and updates (learns) the second model parameters 112 based on the calculated loss (so as to reduce the loss). Note that in this embodiment, learning the second model parameters 112 may be simply referred to as "learning the second model."

[0029] The second correct label is determined, for example, by performing OR and / or majority vote on the first correct labels assigned to each classification target by a plurality of annotators.

[0030] In this embodiment, it is assumed that a correct label is assigned to one classification target by multiple annotators (for example, three people). For example, in the example of classifying into eight types of classes described in FIG. 4, if it is defined as a multi-class classification in which multiple types of correct labels can be assigned to the classification target, it is assumed that the following three types of training data can be generated for one classification target. (1) Annotator A: Classification target X - "Consideration of relationships", "Good icebreaker" (2) Annotator B: Classification target X - "Consideration of relationships", "Selection of good topics", "Selection of topics that suit the other person", "Empathy for the other person" (3) Annotator C: Classification target X - "Consideration of relationships", "Empathy for the other person"

[0031] When training the second model, it is necessary to integrate the above three pieces of data to create a single piece of data. If the same correct label is assigned to the same classification target by multiple annotators, the data can be integrated as is (for example, by "considering relationships"). On the other hand, if different correct labels are assigned to the same classification target, the data must be integrated using the method described below. This is because, as a general rule in machine learning, if data to which different correct labels are assigned to the same classification target is used as training data, proper learning may not be possible.

[0032] When integrating multiple training data from multiple annotators for the same classification target, if the classes corresponding to the assigned correct labels differ between the multiple annotators, the correct labels are integrated by OR or majority vote.

[0033] (OR method) A class that is assigned a correct label by at least one annotator is considered correct. For example, if the data from (1) to (3) above are combined using OR, the result will be as follows: ・Classification target X - "Consideration of relationships," "Good icebreaker," "Selection of good topics," "Selection of topics that suit the other person," "Empathy for the other person"

[0034] (Method by majority vote) The class that more than half of the annotators (two in this case) identified as correct is taken as the correct label for that classification target. For example, if the data from (1) to (3) above are combined by majority vote, the result will be as follows: Classification target X - "Consideration of relationships" and "Empathy for others"

[0035] Even in the case of simple multi-class classification (including binary classification, where one correct label is assigned to one classification target) rather than multi-class classification, adjustments can be made so that one correct label is assigned to one classification target by using a majority vote (adopting the correct label that is assigned most often) or by using predetermined rules (setting priorities or weights for annotators), etc.

[0036] Note that Non-Patent Document 1 is a technology based on the premise that training data has one correct label assigned to one classification target, and therefore does not assume that labels assigned by multiple annotators to the same classification target will be used as is for training the first model. Therefore, Non-Patent Document 1 uses the same first correct label as the second correct label.

[0037] The loss function may be, for example, as follows (see, for example, Reference 1): First, in equation (1), one probability p' is calculated from two probabilities p1 and p2, and the binary cross entropy loss calculated using p' and the correct label is used as the loss function.

[0038] Here, p1 is the probability p1 of the first estimation result by the trained first model, p2 is the probability p2 of one of the second estimation results by the second estimation unit 103, and g is the score g of one of the second estimation results by the second estimation unit 103.

[0039] The score g is an adjustment value that adjusts the extent to which the first estimation result by the first estimation unit 101 is reflected (the degree of reflection) in the learning of the second estimation unit 103. For example, the score g is a one-dimensional value greater than or equal to 0, and the value of g is set to 0 for the voice and / or video of one speaker, and to a value greater than or equal to 0 for the voice and / or video of another speaker. This makes it possible to adjust the extent to which the first estimation result is reflected in the learning of the second estimation unit 103 for each speaker. Note that this score g is automatically determined by learning, for example. Note that setting the value of score g to a value greater than or equal to 0 is just an example, and the value of score g may also be a value less than or equal to 0.

[0040] Note that the score g does not have to be used in calculating the loss function. For example, the second learning unit 104 may obtain p′ using the following equation (2).

[0041] The loss function may be calculated by using an average of the probabilities. For example, the second learning unit 104 may obtain p′ using the following equation (3):

[0042]

[0043] After determining one probability p' from two probabilities p1 and p2, a loss can be calculated using a correct label using an existing general method.

[0044] The control unit 105 executes a control process for controlling the first estimation unit 101, the first learning unit 102, the second estimation unit 103, the second learning unit 104, and the entire classification device 100. Note that the functions of the control unit 105 may be distributed among, for example, the first estimation unit 101, the first learning unit 102, the second estimation unit 103, the second learning unit 104, etc.

[0045] <Processing Flow> Next, the processing flow of the classification method according to this embodiment will be described. Fig. 2 is a flowchart showing an example of processing of the classification system according to this embodiment. This processing shows an example of processing executed by, for example, the classification device 100 and classification system 1 described in Fig. 1.

[0046] In step S201, the first learning unit 102 uses the first estimation result and the first correct label by the first estimation unit 101 to train the first estimation unit 101. Note that here, training the first estimation unit 101 refers to training the first model or the first model parameters 111.

[0047] 3 is a diagram illustrating the learning process of the first estimation unit 101 according to this embodiment. The learning process of the first estimation unit 101 uses training data including multiple sets of four pieces of data: "audio data and / or video data (classification target)," "information indicating the attributes of the classification target (speaker ID)," "information identifying the annotator (annotator ID)," and "correct label assigned by the annotator."

[0048] FIG. 4 is a diagram showing examples of predetermined classes according to this embodiment. Table 400 in FIG. 4 lists Class IDs (1 to 8) and descriptions for each Class ID. Here, the Class IDs (1 to 8) correspond to eight types of predetermined classes. For each Class ID, data in which one binary label is annotated for each utterance by, for example, ten annotators is stored in the dataset as ground truth data.

[0049] For example, the control unit 105 inputs a speaker ID corresponding to the "audio data and / or video data" and an "annotator ID" to the first estimation unit 101. Note that, as a minimum configuration, only the speaker ID may be input to the first estimation unit 101. Here, the following description will be given assuming that the speaker ID and the annotator ID are input to the first estimation unit 101.

[0050] As a result, the first estimation unit 101 processes the input speaker ID and annotator ID based on the first model parameters 111 and outputs a first estimation result (e.g., a probability p1 for each predetermined class). As an example, if there are eight communication skill classes for each speaker ID, the first estimation unit 101 outputs the probability for each of the eight classes as the first estimation result. Note that the probability for each class is a classification probability.

[0051] Next, the control unit 105 inputs the correct label (first correct label) corresponding to the speaker ID and the annotator ID input to the first estimation unit 101 to the first learning unit 102 .

[0052] As a result, the first learning unit 102 uses the first estimation result by the first estimation unit 101 and the first correct label to learn the first estimation unit 101. For example, the first learning unit 102 calculates a loss from the probability p1 for each predetermined class estimated by the first estimation unit 101 and the first correct label corresponding to the speaker ID and the annotator ID, and updates (learns) the first model parameters 111 based on the calculated loss.

[0053] The first learning unit 102 learns the first model parameters 111 by performing the process described with reference to FIG. 3 on each data set and updating the first model parameters 111 .

[0054] 2 , the flowchart will be described again. In step S202, the second learning unit 104 uses the trained first estimating unit 101 to train the second estimating unit 103. Here, training the second estimating unit 103 refers to learning the second model or the second model parameters 112.

[0055] 5 is a diagram illustrating the learning process of the second estimation unit 103 according to this embodiment. The learning process of the second estimation unit 103 uses the same data set and predetermined classes as the learning process of the first estimation unit 101.

[0056] This dataset stores data in which multiple annotators have assigned correct labels to multiple speakers. For example, the dataset stores the number of pieces of data (speech j from speaker i, correct labels from annotator n) equal to the number of speakers × the number of utterances × the number of annotators.

[0057] For example, the control unit 105 inputs speaker ID=1 and annotator ID=1, speaker ID=1 and annotator ID=2, ..., speaker ID=1 and annotator ID=N to the first estimation unit 101 that has learned the first model parameters 111. Here, N is the number of annotators (for example, 10).

[0058] As a result, the first estimation unit 101 processes the input speaker ID and annotator ID based on the first model parameters 111, and calculates N probabilities (probability 1, probability 2, ... probability N) corresponding to each annotator. The first estimation unit 101 also outputs the maximum value of the calculated N probabilities as the first estimation result (probability p1).

[0059] Next, the control unit 105 inputs the voice data and / or video data 501 corresponding to the speaker ID=1 to the second estimation unit 103. Note that the voice data and / or video data 501 corresponding to the speaker with speaker ID=1 includes, for example, voice data of the speaker with speaker ID=1 (hereinafter referred to as speaker voice A), video data of the speaker corresponding to speaker voice A, and video data of the listener corresponding to speaker voice A. Note that the data input to the second estimation unit 103 may be at least one of the above-mentioned three pieces of data.

[0060] As a result, the second estimation unit 103 processes the input audio data and / or video data 501 corresponding to speaker ID=1 based on the second model parameters 112, and outputs the probability p2 for each predetermined class and the score g as the second estimation result. For example, the second estimation unit 103 outputs the probability p2 for each of eight communication skill classes for the speaker with speaker ID=1 as the second estimation result.

[0061] Next, the control unit 105 inputs the second correct answer label corresponding to the audio data and / or video data 501 corresponding to the speaker with speaker ID = 1, which was input to the second estimation unit 103, to the second learning unit 104. Note that the score g is a continuous value, and a correct answer for g is not required.

[0062] As a result, the second learning unit 104 learns the second estimating unit 103 using the first estimation result by the first estimating unit 101 that has learned the first model parameters 111, the second estimation result by the second estimating unit 103 (probability p2 and score g, or probability p2), and the second correct label. For example, the second learning unit 104 calculates a loss using the first estimation result by the first estimating unit 101 that has learned the first model parameters 111, the second estimation result by the second estimating unit 103, and the second correct label, and updates (learns) the second model parameters 112 based on the calculated loss.

[0063] 2, the flowchart will be further described. In step S203, the control unit 105 uses the trained second estimation unit 103 to classify the speakers.

[0064] 6 is a diagram for explaining the estimation process according to this embodiment. The control unit 105 inputs the speech data and / or video data 601 of the speaker with speaker ID=1 to the trained second estimation unit 103, and obtains a second estimation result.

[0065] The voice data and / or video data 601 of the speaker with speaker ID = 1 includes, for example, the speaker voice A of the speaker ID = 1, video data of the speaker corresponding to the speaker voice A, and video data of the listener corresponding to the speaker voice A. Note that the data input to the second estimation unit 103 may be at least one of the above-mentioned three pieces of data, but it is necessary to match it with the data used during learning in step S202.

[0066] The second estimation result also includes, for example, a probability p2 and a score g for each predetermined class. Alternatively, the second estimation result also includes a probability p2 for each predetermined class. Based on this second estimation result, the control unit 105 classifies, for example, the utterance of the speaker with speaker ID = 1 into which of the eight Class IDs described in FIG. 4 it belongs.

[0067] <Example of Experimental Results> Figure 7 shows the experimental results when the first estimation unit 101 was trained using only the speaker ID and annotator ID in a compliment estimation task in which eight types of dialogue skills are classified as high or low from the audio and video of a speaker's utterances, applying this embodiment.

[0068] 7 is a diagram showing an example of experimental results according to this embodiment. As shown in Table 700, it was confirmed through experiments that experimental results 720 according to this embodiment mitigate the deterioration of accuracy for minority speakers and classes compared to experimental results 710 according to the conventional method.

[0069] FIG. 8 is a diagram for explaining an example according to this embodiment. This diagram shows an example when the experimental result 720 in FIG. 7 was obtained. In the example of FIG. 8, a bias-only model 801 was used as the first estimation unit 101, as in the conventional method. In addition, speaker ID=s, annotator ID=1, ..., speaker ID=s, and annotator ID=A were input to the trained bias-only model 801 to obtain A probabilities 803, and the maximum value 804 of the A probabilities 803 was input to the second training unit 104 as the first estimation result.

[0070] 8 , similarly to the conventional method, a Robust Model 802 was used as the second estimation unit 103. The speaker voice of speaker ID=s, the speaker video, and the listener video were input to the Robust Model 802, and a probability 805 and a score g for each predetermined class were calculated.

[0071] Among these, the probability 805 for each predetermined class is input to the second learning unit 104 as the second estimation result, and the second loss is calculated without the score g. The experimental result is MAD LM722 in Fig. 7. Furthermore, the probability 805 for each predetermined class and the score g are input to the second learning unit 104 as the second estimation result, and the second loss is calculated with the score g. The experimental result is MAD PoE721 in Fig. 7. As such, better experimental results are obtained with the score g, but even without the score g, better experimental results are obtained compared to conventional methods.

[0072] <Hardware Configuration> The classification device 100 according to this embodiment has, for example, the hardware configuration of a computer 900 as shown in Fig. 9. Alternatively, the classification device 100 is configured by a plurality of computers 900. Note that the computer is not limited to a physical machine, and may be, for example, a virtual machine on a cloud.

[0073] Fig. 9 is a diagram showing an example of the hardware configuration of a computer. In the example of Fig. 9, a computer 900 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, and an output device 1008, all of which are interconnected by a bus B. The computer 900 may further include another processor such as a GPU (Graphics Processing Unit).

[0074] A program for implementing processing on the computer 900 is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0075] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the classification device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a communication network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, and / or a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0076] The CPU 1004 may be another processor such as a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0077] According to the present embodiment, it is possible to suppress a decrease in classification accuracy in a task of classifying a classification target into one of a plurality of predefined classes. For example, in the classification device 100 (or the classification system 1) that classifies speakers using machine learning, it is possible to suppress a decrease in classification accuracy even for minority speakers or classes that are prone to accuracy degradation.

[0078] <Supplementary Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0079] (Supplementary Item 1) A classification device that classifies an object to be classified by machine learning, comprising: a memory; and at least one processor connected to the memory, wherein the processor: executes a first estimation process that takes as input information that identifies an attribute of the object to be classified, and calculates a first estimation result that classifies the object to be classified into a predetermined class based on a first model; executes a first learning process that trains the first model using the first estimation result and a first correct label corresponding to the object to be classified; executes a second estimation process that takes as input audio data and / or video data that is the object to be classified, and calculates a second estimation result that classifies the object to be classified into the predetermined class based on a second model; executes a second learning process that trains the second model using the first estimation result calculated based on the trained first model, the second estimation result, and a second correct label corresponding to the object to be classified; and executes an input audio data and / or video data that is the object to be classified, and classifies the object to be classified into the predetermined class based on the trained second model. (Supplementary Item 2) The classification device according to Supplementary Item 1, wherein the first correct labels are assigned by a plurality of annotators, the first estimation unit estimates the first estimation result based on information identifying an attribute of the classification target and information identifying the annotators, and the first learning unit learns the first model using the first estimation result by the first estimation unit and the first correct labels assigned by the annotators. (Supplementary Item 3) The classification device according to Supplementary Item 1, wherein the second estimation result further includes an adjustment value that adjusts the degree to which the first estimation result is reflected in learning of the second model.(Supplementary Item 4) A classification method for classifying an object to be classified by machine learning, comprising: a process in which a first estimation unit receives information identifying the object to be classified as an input, and calculates a first estimation result in which the object to be classified is classified into a predetermined class based on a first model; a process in which a first learning unit learns the first model using the first estimation result and a first correct label corresponding to the object to be classified; a process in which a second estimation unit receives audio data and / or video data as the object to be classified, and calculates a second estimation result in which the object to be classified is classified into the predetermined class based on a second model; a process in which a second learning unit learns the second model using the first estimation result calculated based on the learned first model, the second estimation result, and a second correct label corresponding to the object to be classified; and a process in which a control unit receives audio data and / or video data as an input, and classifies the object to be classified into the predetermined class based on the learned second model. (Supplementary Item 5) A non-transitory storage medium storing a program that causes a computer used as a classification device that classifies objects to be classified by machine learning to function as: a first estimation unit that receives information identifying attributes of the object to be classified as an input, and calculates a first estimation result that classifies the object to be classified into a predetermined class based on a first model; a first learning unit that trains the first model using the first estimation result and a first correct label corresponding to the object to be classified; a second estimation unit that receives audio data and / or video data that are the object to be classified as an input, and calculates a second estimation result that classifies the object to be classified into the predetermined class based on a second model; a second learning unit that trains the second model using the first estimation result calculated based on the trained first model, the second estimation result, and a second correct label corresponding to the object to be classified; and a control unit that receives audio data and / or video data that are the object to be classified as an input, and classifies the object to be classified into the predetermined class based on the trained second model.

[0080] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0081] (References) Reference 1: Christopher Clark, Mark Yatskar, and Luke Zettlemoyer, "Don't take the easy way out: Ensemble-based methods for avoiding known dataset biases," in EMNLP-IJCNLP, 2019, pp. 4069-4082.

[0082] 1 Classification system 100 Classification device 101 First estimation unit 102 First learning unit 103 Second estimation unit 104 Second learning unit 105 Control unit 111 First model parameters 112 Second model parameters 400 Table (an example of a predetermined label) 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A classification device comprising: a first estimation unit that receives as input information identifying attributes of an object to be classified, and calculates a first estimation result that classifies the object to be classified into a predetermined class based on a first model; a first learning unit that trains the first model using the first estimation result and a first correct label corresponding to the object to be classified; a second estimation unit that receives as input audio data and / or video data that is the object to be classified, and calculates a second estimation result that classifies the object to be classified into the predetermined class based on a second model; and a second learning unit that trains the second model using the first estimation result calculated based on the trained first model, the second estimation result, and a second correct label corresponding to the object to be classified, wherein the classification device receives as input audio data and / or video data that is the object to be classified, and classifies the object to be classified into the predetermined class based on the trained second model.

2. The classification device described in claim 1, wherein the first correct label is assigned by a plurality of annotators; the first estimation unit estimates the first estimation result based on information identifying the attribute of the classification target and information identifying the annotators; and the first learning unit learns the first model using the first estimation result by the first estimation unit and the first correct label assigned by the annotators.

3. The classification device according to claim 1, wherein the second estimation result further includes an adjustment value that adjusts the degree to which the first estimation result is reflected in the learning of the second model.

4. A classification method for classifying an object to be classified by machine learning, comprising: a process in which a first estimation unit receives information identifying the object to be classified as input and calculates a first estimation result that classifies the object to be classified into a predetermined class based on a first model; a process in which a first learning unit trains the first model using the first estimation result and a first correct label corresponding to the object to be classified; a process in which a second estimation unit receives audio data and / or video data that is the object to be classified as input and calculates a second estimation result that classifies the object to be classified into the predetermined class based on a second model; a process in which a second learning unit trains the second model using the first estimation result calculated based on the trained first model, the second estimation result, and a second correct label corresponding to the object to be classified; and a process in which a control unit receives audio data and / or video data that is the object to be classified as input and classifies the object to be classified into the predetermined class based on the trained second model.

Images