Support device, support method, and program

Abstract

To provide a support device, a support method, and a program capable of efficiently evaluating training data creators.SOLUTION: A training data confirmation support device 10 includes an evaluation unit 17 which presents types of difference patterns indicating confusion or replacement of correct labels and inference labels, for each of a plurality of elements, the correct labels corresponding to the elements being different from the inference labels corresponding to the element. The inference labels are labels inferred, as labels corresponding to the elements, using an inference model trained using training data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an assistance device, an assistance method, and a program. [Background technology]

[0002] In recent years, with the aim of improving the quality of service at contact centers, systems have been proposed that perform real-time speech recognition of call content and automatically present appropriate information to the operator using natural language processing technology.

[0003] For example, Non-Patent Document 1 discloses a technology that presents anticipated questions and answers to those questions (FAQs) to an operator during a dialogue between an operator and a customer. In this technology, the dialogue between the operator and the customer is speech-recognized and converted into semantically coherent spoken text through "end-of-speech determination," which determines whether the speaker has finished speaking. Next, "interaction scene estimation" is performed to estimate which interaction scene in the dialogue the utterance corresponding to the spoken text is from, such as the operator's greeting, confirmation of the customer's business, response to the business, or closing the dialogue. The dialogue is structured through "interaction scene estimation." From the results of "interaction scene estimation," "FAQ search utterance determination" is performed to extract utterances that include the customer's business or utterances in which the operator confirms the customer's business. A search is performed on a pre-prepared FAQ database using a search query based on the utterance extracted by "FAQ search utterance determination," and the search results are presented to the operator.

[0004] The above-mentioned "end of speech determination," "interaction scene estimation," and "FAQ search utterance determination" use models constructed by training, using a deep neural network or the like, training data in which labels that categorize utterances are assigned to speech text. Therefore, "end of speech determination," "interaction scene estimation," and "FAQ search utterance determination" can be considered as a sequence labeling problem in which sequential elements (utterances in a dialogue) are labeled. Non-Patent Document 2 describes a technology for estimating an interaction scene by training, using a deep neural network with long and short-term memory, a large amount of training data in which sequential utterances are assigned labels corresponding to the interaction scenes in which the utterances are included. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] T. Hasegawa, Y. Sekiguchi, S. Yamada, and M. Tamoto, "Automatic Knowledge Support System for Supporting Operators' Responses," NTT Technical Review, vol. 31, no. 7, pp. 16-19, July 2019. [Non-patent document 2] R. Masumura, S. Yamada, T. Tanaka, A. Ando, H. Kamiyama, and Y. Aono, “Online Call Scene Segmentation of Contact Center Dialogues based on Role Aware Hierarchical LSTM-RNNs,” Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Nov. 2018. Summary of the Invention [Problem to be solved by the invention]

[0006] The techniques described in the above-mentioned Non-Patent Documents 1 and 2 require a large amount of training data to achieve a practical level of estimation accuracy. For example, according to Non-Patent Document 1, high estimation accuracy can be achieved by creating training data from conversation logs of a call center containing approximately 1,000 calls and training a model. The training data is created by an operator (training data creator) who assigns a label to each utterance text while referring to the utterance text obtained by speech recognition of the utterance.

[0007] Training data must be created to suit the application (e.g., each industry of contact centers) of the model trained using that training data. As mentioned above, a large amount of training data is required to achieve high estimation accuracy, and therefore the task of creating training data to which labels are assigned is often carried out by multiple workers. However, because each worker has different experience or detailed labeling policies, inconsistencies in labeling can occur, with different labels being assigned to the same utterance. When labeling inconsistencies occur in training data, the estimation accuracy of the model trained using that training data decreases. However, there is no established method for efficiently identifying which training data creator is responsible for the inconsistencies in labeling. Traditionally, this has required analysis based on tacit knowledge by experts or repeated trial and error.

[0008] Therefore, there is a need for a technology that can more efficiently evaluate training data creators.

[0009] The purpose of the present disclosure, made in consideration of the above-mentioned problems, is to provide an assistance device, an assistance method, and a program that can more efficiently evaluate teacher data creators. [Means for solving the problem]

[0010] In order to solve the above problem, the support device disclosed herein is a support device that supports the evaluation of a teacher data creator who creates teacher data consisting of pairs of elements and correct labels corresponding to the elements, and includes: a label inference unit that infers inferred labels, which are labels corresponding to the elements that make up the teacher data, using a model that has been trained using the teacher data and that infers labels corresponding to the elements; and an evaluation unit that generates an evaluation result for the teacher data creator based on a comparison between the correct labels corresponding to the elements that make up the teacher data and the inferred labels of the elements.

[0011] In addition, in order to solve the above problem, the support method disclosed herein is a support method in a support device that supports the evaluation of a teacher data creator who creates teacher data consisting of pairs of elements and correct labels corresponding to the elements, and includes a step of inferring inferred labels, which are labels corresponding to the elements that make up the teacher data, using a model that is trained using the teacher data and infers labels corresponding to the elements, and a step of generating an evaluation result for the teacher data creator based on a comparison between the correct labels corresponding to the elements that make up the teacher data and the inferred labels of the elements.

[0012] In addition, in order to solve the above problem, a program according to the present disclosure causes a computer to function as the above-mentioned assistance device. [Effects of the Invention]

[0013] According to the support device, support method, and program disclosed herein, evaluation of teacher data creators can be performed more efficiently. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a schematic configuration of a computer functioning as a support device according to a first embodiment of the present disclosure. [Figure 2] 1 is a diagram illustrating an example of a functional configuration of an assistance device according to a first embodiment of the present disclosure. [Figure 3]3 is a flowchart showing an example of the operation of the support device shown in FIG. 2. [Figure 4] 3 is a diagram showing an example of a call-specific evaluation result by the call-specific inference result evaluation unit shown in FIG. 2. FIG. [Figure 5] 3 is a diagram showing an example of a call-specific confirmation screen generated by a call-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 6] 10 is a diagram showing another example of the call-specific confirmation screen generated by the call-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 7] 3 is a diagram showing an example of an utterance-specific evaluation result by the utterance-specific inference result evaluation unit shown in FIG. 2. FIG. [Figure 8] 3 is a diagram showing an example of an utterance-specific confirmation screen generated by an utterance-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 9] FIG. 10 is a diagram illustrating an example of a functional configuration of an assistance device according to a second embodiment of the present disclosure. [Figure 10] 10 is a flowchart showing an example of the operation of the support device shown in FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of a functional configuration of an assistance device according to a third embodiment of the present disclosure. [Figure 12] 12 is a flowchart showing an example of the operation of the support device shown in FIG. [Figure 13] FIG. 12 is a diagram showing an example of a teacher data creator evaluation result by the teacher data creator evaluation unit shown in FIG. [Figure 14] 12 is a diagram showing an example of a call-specific evaluation result by the call-specific inference result evaluation unit shown in FIG. 11. FIG. [Figure 15] 12 is a diagram showing an example of a call-specific confirmation screen generated by a call-specific confirmation screen generating unit shown in FIG. 11. FIG. [Figure 16] 12 is a diagram showing an example of an evaluation result for each utterance by the utterance-specific inference result evaluation unit shown in FIG. 11. FIG. [Figure 17] 12 is a diagram showing another example of an utterance-specific evaluation result by the utterance-specific inference result evaluation unit shown in FIG. 11. FIG. [Figure 18] 12 is a diagram showing an example of an utterance-specific confirmation screen generated by an utterance-specific confirmation screen generating unit shown in FIG. 11. FIG. [Figure 19]FIG. 10 is a diagram showing an example of a label structure made up of multiple items. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0016] (First embodiment) 1 is a block diagram showing a hardware configuration of an assistance device 10 according to a first embodiment of the present disclosure, which is a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a personal computer (PC), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0017] 1, the assistance device 10 includes a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a storage 140, an input unit 150, a display unit 160, and a communication interface (I / F) 170. Each component is connected to each other via a bus 190 so as to be able to communicate with each other. The processor 110 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an SoC (System on a Chip), or the like, and may be configured by multiple processors of the same or different types.

[0018] The processor 110 controls each component and performs various arithmetic operations. That is, the processor 110 reads a program from the ROM 120 or the storage 140 and executes the program using the RAM 130 as a work area. The processor 110 controls each component and performs various arithmetic operations in accordance with the program stored in the ROM 120 or the storage 140. In this embodiment, the ROM 120 or the storage 140 stores a program according to the present disclosure.

[0019] The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0020] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0021] The input unit 150 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0022] The display unit 160 is, for example, a liquid crystal display, and displays various information. The display unit 160 may function as the input unit 150 by adopting a touch panel system.

[0023] The communication interface 170 is an interface for communicating with other devices such as external devices (not shown), and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0024] Next, the functional configuration of the support device 10 according to this embodiment will be described.

[0025] FIG. 2 is a diagram illustrating an example of the configuration of an assistance device 10 according to this embodiment. The assistance device 10 according to this embodiment supports the task of checking teacher data, such as checking whether or not the labeling criteria are inconsistent, by an operator who creates teacher data consisting of pairs of elements and labels assigned to those elements (hereinafter referred to as "correct labels"). Note that correct labels are labels assigned when the teacher data is created and are the subject of the checking task. Therefore, the assigned correct labels are not necessarily correct. Supporting the checking of teacher data also facilitates the extraction of labels that require correction, thereby improving the efficiency of the teacher data correction task. Below, an example of assigning labels to spoken text obtained by speech recognition of utterances in a dialogue between multiple speakers (operators and customers) at a contact center, as shown in FIG. 19, will be described. In FIG. 19, the spoken text corresponding to the operator's utterance (hereinafter, the spoken text corresponding to the utterance may be simply referred to as "spoken text") is indicated by a solid-line speech bubble, and the spoken text of the customer is indicated by a dotted-line speech bubble.

[0026] In the example shown in FIG. 19 , training data for "end of speech determination" is created by assigning an end-of-speech label to each utterance text, indicating whether the utterance is the end of the speech. Furthermore, training data for "reception scene estimation" is created by assigning a scene label indicating the reception scene in which the utterance is included to each utterance text. Furthermore, training data for "FAQ search utterance determination" is created by assigning a subject label indicating that the utterance indicates the customer's subject among utterances included in the reception scene for "request understanding" in which the customer's subject is understood, and assigning a subject confirmation label indicating that the utterance confirms the customer's subject to an utterance in which the operator confirms the customer's subject. However, the present disclosure is not limited to the example of training data shown in FIG. 19 , and can be applied to training data consisting of any combination of multiple elements and their labels. Furthermore, the utterance text may be not only a text version of a call utterance, but also a text-based conversation such as a chat. Furthermore, the speaker in the dialogue is not limited to a human being, but may be a robot or a virtual agent.

[0027] 2, the assistance device 10 according to this embodiment includes a model learning unit 11, a label inference unit 12, a call-specific inference result evaluation unit 13, a call-specific confirmation screen generation unit 14, an utterance-specific inference result evaluation unit 15, and an utterance-specific confirmation screen generation unit 16. The call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 constitute an evaluation unit 17. The model learning unit 11, the label inference unit 12, the call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured by one or more processors as described above.

[0028] The model learning unit 11 receives training data consisting of pairs of spoken text (elements) and correct labels assigned to the spoken text. The model learning unit 11 uses the input training data to learn a model that infers a label corresponding to the spoken text. Any learning method can be applied as the model learning method depending on the purpose of the system to which the model is applied. The model learning unit 11 outputs a model created by learning the training data (hereinafter referred to as a "trained model") to the label inference unit 12. Note that the trained model may be prepared in advance. Therefore, the assistance device 10 does not need to include the model learning unit 11.

[0029] The label inference unit 12 receives as input the training data and the trained model created by the model learning unit 11. The training data input to the label inference unit 12 is the same as the training data used to train the trained model. The label inference unit 12 uses the trained model to infer labels of utterance text (elements) that make up the training data (hereinafter, labels inferred by the trained model are referred to as "inferred labels"). The label inference unit 12 outputs the inferred labels of each utterance text that makes up the training data as inference results to the call-specific inference result evaluation unit 13 and the utterance-specific inference result evaluation unit 15.

[0030] The evaluation unit 17 compares and evaluates the correct labels assigned to the elements constituting the teacher data with the inferred labels inferred by the label inference unit 12, and outputs the evaluation result to the external output interface 1. The evaluation unit 17 also generates a teacher data confirmation screen for confirming the teacher data, which includes the elements constituting the teacher data, the correct labels assigned to the elements, and the inferred labels of the elements. The evaluation unit 17 outputs the generated teacher data confirmation screen to the external output interface 1.

[0031] The external output interface 1 is a device used by a worker who creates and modifies teacher data or an administrator who manages the work of the worker. The external output interface 1 presents, for example, a comparison result between the correct label assigned to the teacher data output from the evaluation unit 17 and the inferred label inferred by the trained model. The external output interface 1 may have any configuration as long as it has the function of communicating with the assistance device 10, the function of presenting (displaying) the evaluation results of the evaluation unit 17 and a teacher data confirmation screen, and the function of accepting operation input.

[0032] As described above, the evaluation unit 17 includes the call-specific inference result evaluation unit 13 , the call-specific confirmation screen generation unit 14 , the utterance-specific inference result evaluation unit 15 , and the utterance-specific confirmation screen generation unit 16 .

[0033] The call-specific inference result evaluation unit 13 receives as input the training data and the inference results of the label inference unit 12. Typically, the training data includes a group of utterance texts, each consisting of a plurality of utterance texts in calls by a plurality of speakers, for the number of calls. In other words, the training data includes a plurality of element groups, each consisting of a plurality of sequential elements. The call-specific inference result evaluation unit 13 evaluates the input training data and the inference results of the label inference unit 12 for each call. The call-specific inference result evaluation unit 13 outputs the evaluation results (call-specific evaluation results) to the call-specific confirmation screen generation unit 14 and the external output interface 1. The call-specific evaluation results will be described in detail below.

[0034] The call-specific confirmation screen generation unit 14 generates a teacher data confirmation screen for each call (hereinafter referred to as a "call-specific confirmation screen") based on the call-specific evaluation results output from the call-specific inference result evaluation unit 13, and outputs it to the external output interface 1. The call-specific confirmation screen will be described in detail later.

[0035] The utterance-specific inference result evaluation unit 15 receives as input the teacher data and the inference results of the label inference unit 12. The utterance-specific inference result evaluation unit 15 evaluates the input teacher data and the inference results of the label inference unit 12 for each utterance. The utterance-specific inference result evaluation unit 15 outputs the evaluation results (utterance-specific evaluation results) to the utterance-specific confirmation screen generation unit 16 and the external output interface 1. The utterance-specific evaluation results will be described in detail later.

[0036] The utterance-specific confirmation screen generation unit 16 generates a teacher data confirmation screen for each utterance (hereinafter referred to as "utterance-specific confirmation screen") based on the utterance-specific evaluation results output from the utterance-specific inference result evaluation unit 15, and outputs it to the external output interface 1. The utterance-specific confirmation screen will be described in detail later.

[0037] In this embodiment, a teacher data confirmation screen is generated that includes the spoken text (elements) that constitute the teacher data, the correct labels assigned to the spoken text, and the inferred labels inferred by a trained model trained using the teacher data. Therefore, with the assistance device 10 according to this embodiment, the worker can easily compare the correct labels and the inferred labels of the elements on the teacher data confirmation screen and confirm the teacher data, thereby improving the efficiency of the teacher data confirmation work. Furthermore, improving the efficiency of the teacher data confirmation work makes it easier to extract labels that need to be corrected, thereby improving the efficiency of the label correction work.

[0038] Next, the operation of the support device 10 according to this embodiment will be described.

[0039] FIG. 3 is a flowchart showing an example of the operation of the support device 10, and is a diagram for explaining the support method by the support device 10 according to this embodiment.

[0040] The model learning unit 11 uses training data to learn a model that infers labels that distinguish between spoken texts (step S11).

[0041] The label inference unit 12 infers inference labels corresponding to elements of the training data using the trained model trained by the model training unit 11 (step S12). As described above, the training data used for training the trained model is the same as the training data used for the training data inference process by the label inference unit 12.

[0042] The call-specific inference result evaluation unit 13 evaluates the teacher data and the inference result of the label inference unit 12 for each call, and outputs the evaluation result (call-specific evaluation result) (step S13). Specifically, the call-specific inference result evaluation unit 13 compares, for each call, the difference between the correct label assigned to the utterance text constituting the teacher data and the inferred label inferred by the label inference unit 12. Then, the call-specific inference result evaluation unit 13 sorts the evaluation values for each call in order of the worst evaluation result (e.g., utterances with evaluation values below a threshold), and outputs them as call-specific evaluation results. That is, the call-specific inference result evaluation unit 13 outputs the evaluation results for each element group in order of the worst evaluation result (calls consisting of multiple utterances). The evaluation value of a call can be the precision, recall, F-measure, or matching rate between the correct label and the inferred label of each utterance text constituting the teacher data.

[0043] FIG. 4 is a diagram showing an example of the call-specific evaluation results.

[0044] As shown in Fig. 4, the call-specific inference result evaluation unit 13 outputs, as call-specific evaluation results, call indices, which are identification information for identifying calls, in association with evaluation values such as the match rate for the calls. Here, the call-specific inference result evaluation unit 13 lists the call indices and evaluation values in descending order of evaluation results, and outputs them as text data, for example. The call-specific evaluation results may include the start and end times of the calls.

[0045] Referring back to FIG. 3, the call-specific confirmation screen generating unit 14 generates a call-specific confirmation screen based on the call-specific evaluation result (step S14), and outputs it to the external output interface 1.

[0046] FIG. 5 is a diagram showing an example of the call-specific confirmation screen.

[0047] As shown in FIG. 5, the call-specific confirmation screen generation unit 14 generates, for each call, a call-specific confirmation screen including a start time, which is the time when the utterance constituting the call started, an end time, which is the time when the utterance ended, the utterance text, and the correct label and inferred label of each utterance text. In this way, the call-specific confirmation screen generation unit 14 generates a teacher data confirmation screen including elements constituting the teacher data, the correct label of the element, and the inferred label of the element. Specifically, the call-specific confirmation screen generation unit 14 generates a teacher data confirmation screen that displays the correct label and the inferred label corresponding to the element constituting the teacher data in a comparable manner (for example, as shown in FIG. 5, the correct label and the inferred label corresponding to the element are displayed side by side). Here, the call-specific confirmation screen generation unit 14 presents the call-specific confirmation screen in descending order of the evaluation result. For example, as shown in FIG. 5, the call-specific confirmation screen generation unit 14 may display the call-specific confirmation screen closer to the front for calls with worse evaluation results. That is, the call-specific confirmation screen generating unit 14 may generate a call-specific confirmation screen for each element so that calls can be checked in order of worst-case evaluation results. As described above, the call-specific confirmation screen includes the start and end times of utterances. This allows the worker to check whether utterances overlap. Note that the start and end times do not necessarily have to be included in the call-specific confirmation screen.

[0048] In this way, the call-specific inference result evaluation unit 13 constituting the evaluation unit 17 evaluates, for each element group, the difference between the correct label assigned to the element constituting the element group and the inference label inferred by the trained model. Also, the call-specific confirmation screen generation unit 14 constituting the evaluation unit 17 generates a teacher data confirmation screen (call-specific confirmation screen) for each element group based on the call-specific evaluation results, and presents the call-specific confirmation screen in order of the element groups with the worst evaluation results.

[0049] Furthermore, the call-specific confirmation screen generating unit 14 constituting the evaluation unit 17 may switchably present a call-specific confirmation screen for each call. In the example shown in Fig. 5, the call-specific confirmation screen generating unit 14 may switch the utterance-specific confirmation screen displayed in the foreground in response to a switching operation by the operator, for example. In this way, the call-specific confirmation screen generating unit 14 may switchably present the evaluation results for each element group.

[0050] By presenting a call-specific confirmation screen for each call, the worker can discover and correct poor-quality teacher data on a call-by-call basis. Furthermore, by making the call-specific confirmation screen switchable, the worker can, for example, continuously check the evaluation results for each call, thereby improving the efficiency of the teacher data confirmation work. Furthermore, by generating a call-specific confirmation screen that allows the worker to check calls in order of poor evaluation results, the worker can discover trends in poor-quality teacher data on a call-by-call basis and grasp the key points for correction. As a result, the efficiency of the teacher data correction work can be improved. Note that, instead of presenting the call-specific confirmation screen shown in FIG. 5 in a switchable manner, the call-specific confirmation screen generation unit 14 may output a group of text data files corresponding to the call-specific confirmation screen to the external output interface 1, sorting them into directories or the like based on the evaluation value for each call.

[0051] The call-specific confirmation screen is not limited to the example shown in Fig. 5. Fig. 6 is a diagram showing another example of the call-specific confirmation screen generated by the call-specific confirmation screen generating unit 14.

[0052] As shown in Fig. 6, the call-specific confirmation screen generation unit 14 may arrange the spoken texts of the operator and the customer in a row in chronological order on the call-specific confirmation screen. Furthermore, the call-specific confirmation screen generation unit 14 may arrange, in association with each spoken text, the start time at which the utterance began, the end time at which the utterance ended, and the labels assigned to the utterance (scene label, subject label, subject confirmation label, and end-of-speech label). As shown in Fig. 6, the call-specific confirmation screen generation unit 14 may display the spoken text of the operator and the spoken text of the customer in different colors. Note that in Fig. 6, the difference in color is represented by different hatching.

[0053] As shown in FIG. 6, the call-specific confirmation screen generation unit 14 may arrange multiple elements in a row on the call-specific confirmation screen, and may allocate the labels of the multiple items to one side and the other side of the element corresponding to the label based on the structure of the labels of the multiple items.

[0054] Generally, it is easier to check and correct labels if they are placed in an area closer to the spoken text. Therefore, by placing the spoken text in a line and allocating labels for multiple items to both sides of the spoken text, we can effectively utilize the area close to the spoken text and improve the efficiency of checking and correcting labels.

[0055] In the example shown in FIG. 6, the scene label, the subject label, and the subject confirmation label are arranged to the left of the utterance text, and the end-of-speech label is arranged to the right of the utterance text. When assigning a scene label, a subject label, and a subject confirmation label to an utterance text, not only the utterance text itself but also the content of the utterance texts before and after it is taken into consideration. In other words, the scene label, the subject label, and the subject confirmation label are labels that take into consideration the long-term context and are determined based on the content of multiple utterance texts including the utterance text. On the other hand, when assigning an end-of-speech label to an utterance text, it is sufficient to mainly consider only the utterance text itself. Therefore, the call-specific confirmation screen generation unit 14 may arrange labels that take into consideration the long-term context to the left of the utterance text and labels that do not take into consideration the long-term context to the right of the utterance text.

[0056] In the example shown in FIG. 6 , the call-specific confirmation screen generation unit 14 places the subject label and the subject confirmation label closer to the utterance text than the scene label. Typically, a subject label or a subject confirmation label is assigned to utterance text to which a scene label of “understanding the subject” is assigned. That is, the scene label is a higher-level label, and the subject label / subject confirmation label is a lower-level label. Therefore, the call-specific confirmation screen generation unit 14 may place labels of multiple items having a hierarchical structure closer to the utterance text, the lower the level of the label. This improves work efficiency because the lower the level of the label, the easier it is to check and correct the label by looking at the utterance text. Furthermore, the end-of-speech label is assigned mainly with a focus on the end of the utterance. Therefore, by placing the end-of-speech label to the right of the utterance text, the worker can easily see the end of the utterance text, thereby improving the work efficiency of checking and correcting the end-of-speech label.

[0057] Furthermore, when a label to be corrected is selected by the worker in the correction of training data, the call-specific confirmation screen generation unit 14 may change the display mode of labels (upper-level labels and lower-level labels) related to the label to be corrected based on the hierarchical structure of labels for multiple items. In the example shown in FIG. 6, it is assumed that the scene label "understanding the subject" is selected as the label to be updated. In this case, the call-specific confirmation screen generation unit 14 changes the display mode by, for example, changing the display colors of the subject label and the subject confirmation utterance label, which are labels in the lower level of the scene label. This makes it easier for the worker to understand the labels related to the label to be corrected, thereby improving the efficiency of the label assignment work.

[0058] Furthermore, if a contradiction occurs between related labels when updating a label in a higher hierarchy or a label in a lower hierarchy, the call-specific confirmation screen generating unit 14 may change the display mode of the label in question. This can eliminate contradictions between labels of multiple items having a hierarchical structure and improve the accuracy of label correction.

[0059] Furthermore, the call-specific confirmation screen generating unit 14 may display utterance text that is not included in the training data, such as fillers and short utterance text such as "yes," in a different manner from other utterance text. This allows the worker to easily identify utterance text that does not require labeling, thereby improving work efficiency.

[0060] 3 again, the utterance-specific inference result evaluation unit 15 evaluates the teacher data and the inference result of the label inference unit 12 for each utterance, and outputs the evaluation result (utterance-specific evaluation result) (step S15). The utterance-specific inference result evaluation unit 15 compares the label of the teacher data with the label of the inference result of the label inference unit 12 for each utterance, tallying up difference patterns, which are patterns in which the label of the teacher data and the label of the inference result are different, and outputs the result as the utterance-specific evaluation result.

[0061] FIG. 7 is a diagram showing an example of the evaluation results for each utterance.

[0062] As shown in Figure 7, the utterance-specific inference result evaluation unit 15 outputs, as utterance-specific evaluation results, for example, results showing the number of occurrences of difference patterns in a confusion matrix, and evaluation values for each label (precision, recall, F-score (f1-score), number of occurrences (support)) as text data.

[0063] Referring back to FIG. 3, the utterance-specific confirmation screen generating unit 16 generates an utterance-specific confirmation screen based on the utterance-specific evaluation result (step S16), and outputs it to the external output interface 1.

[0064] FIG. 8 is a diagram showing an example of the confirmation screen for each utterance.

[0065] As shown in FIG. 8, the utterance confirmation screen generation unit 16 generates an utterance confirmation screen that associates utterance text, a line number indicating the order of the utterance text in the call including the utterance, and the correct label and inference label of the utterance text. In this way, the utterance confirmation screen generation unit 16 generates a teacher data confirmation screen that includes elements constituting the teacher data, correct labels assigned to the elements, and inference labels of the elements. Specifically, the utterance confirmation screen generation unit 16 generates a teacher data confirmation screen that displays the correct labels and inference labels corresponding to the elements constituting the teacher data in a comparable manner (for example, as shown in FIG. 8, the correct labels and inference labels corresponding to the elements are displayed side by side). The utterance confirmation screen generation unit 16 generates an utterance confirmation screen for each utterance in which the correct label and inference label are different. In FIG. 8, the utterance of line number 41 enclosed in a dotted rectangle is the utterance to be displayed. As shown in FIG. 8, the utterance-specific confirmation screen generating unit 16 may mark the utterance text to be displayed (the utterance text whose correct label and inference label are different) with a predetermined mark ("**" in FIG. 8). The utterance-specific confirmation screen generating unit 16 may include utterances before and after the utterance to be displayed in the utterance-specific confirmation screen of the utterance to be displayed. That is, the utterance-specific confirmation screen generating unit 16 may generate an utterance-specific confirmation screen that includes an element whose correct label and inference label are different, and elements before and after the element. FIG. 8 shows an example in which the utterance text from line number 38 to line number 44 is included in the utterance-specific confirmation screen in which the utterance text of line number 41 is to be displayed.

[0066] The utterance-specific confirmation screen generating unit 16 presents the utterance-specific confirmation screen for each utterance in order from the utterance text including the difference pattern that appears most frequently among the difference patterns in which the teacher data and the inference label are different. In other words, the utterance-specific confirmation screen generating unit 16 may present the utterance-specific confirmation screen in order from the element including the difference pattern that appears most frequently among the difference patterns in which the teacher data and the inference label are different.

[0067] In this way, the utterance-specific inference result evaluation unit 15 constituting the evaluation unit 17 compares, for each element constituting the teacher data, the correct label assigned to that element with the inference label inferred by the trained model, and outputs the evaluation result. Also, the utterance-specific confirmation screen generation unit 16 constituting the evaluation unit 17 generates and presents, for each element constituting the teacher data, a teacher data confirmation screen (utterance-specific confirmation screen) in descending order of occurrence of the difference patterns in which the correct label and the inference label are different.

[0068] 8, the utterance confirmation screen generating unit 16 may present multiple utterance confirmation screens in a partially overlapping manner, and may switch the utterance confirmation screen displayed in the foreground in response to a switching operation by the operator, for example. That is, the utterance confirmation screen generating unit 16 may generate the utterance confirmation screens so that elements including difference patterns with the highest frequency of appearance can be checked in order. This allows only the training data that needs to be checked to be quickly checked in order of greatest impact.

[0069] By displaying the utterance confirmation screen, the worker can discover and correct teacher data with erroneous labels on an utterance-by-utterance basis. Furthermore, by presenting elements whose inferred labels differ from the correct label and the elements before and after those elements, the worker can correct the label of the utterance text to be displayed while taking into account the content of the utterance text (elements) before and after, thereby improving the efficiency of the label correction work. Furthermore, by displaying multiple utterance confirmation screens with the same difference pattern in a switchable manner, the worker can continuously check the utterance confirmation screens with the same difference pattern and grasp the key points of correction for each difference pattern. As a result, the efficiency of the teacher data correction work can be improved. Note that instead of displaying the utterance confirmation screens shown in FIG. 8 in a switchable manner, the utterance confirmation screen generation unit 16 may output a group of text data files corresponding to the utterance confirmation screens to the external output interface 1, dividing them into directories or the like.

[0070] As described above, the support device 10 according to this embodiment includes a label inference unit 12 and an evaluation unit 17. The label inference unit 12 infers inference labels of elements constituting the training data using a trained model trained using the training data. The evaluation unit 17 generates a training data confirmation screen including the elements constituting the training data, the correct labels assigned to the elements, and the inference labels inferred by the trained model.

[0071] Furthermore, the teacher data correction method according to this embodiment includes a step of inferring labels (step S12) and steps of generating a teacher data confirmation screen (steps S14 and S16). In the step of inferring labels, inferred labels of elements constituting the teacher data are inferred using a trained model trained using the teacher data. In the step of generating the teacher data confirmation screen, a teacher data confirmation screen is generated that includes the elements constituting the teacher data, the correct labels assigned to the elements, and the inferred labels of the elements.

[0072] In this way, according to the support device 10 and support method of this embodiment, the teacher data confirmation screen, which includes the correct label and inference label of the element, makes it easier for the worker to confirm the teacher data, thereby making the teacher data confirmation work more efficient.

[0073] (Second embodiment) Fig. 9 is a diagram showing a configuration example of a support device 10A according to the second embodiment of the present disclosure. In Fig. 9, the same components as those in Fig. 2 are denoted by the same reference numerals, and description thereof will be omitted.

[0074] The support device 10A according to this embodiment differs from the support device 10 according to the first embodiment in that an inference error elimination unit 18 is added.

[0075] The inference error elimination unit 18 receives the evaluation results for each utterance by the utterance-specific inference result evaluation unit 15. The inference error elimination unit 18 performs an inference error elimination process to exclude elements for which the inference labels inferred by the trained model are determined to be incorrect according to predetermined rules. Specifically, the inference error elimination unit 18 excludes utterances whose inference labels are clearly incorrect from the evaluation results for each utterance by the utterance-specific inference result evaluation unit 15. An obviously erroneous utterance is, for example, an utterance in which a single scene is composed of only one utterance, or an utterance in which, despite being the beginning of a call, a label indicating the end of the call or a response to the customer's request is assigned to the utterance text. The criteria for determining an obviously erroneous utterance are manually determined in advance.

[0076] Next, the operation of the support device 10A according to this embodiment will be described. Fig. 10 is a flowchart showing an example of the operation of the support device 10A. In Fig. 10, the same processes as those in Fig. 3 are denoted by the same reference numerals, and the description thereof will be omitted.

[0077] When the utterance-specific inference result evaluation unit 15 outputs the utterance-specific evaluation result (step S15), the inference error exclusion unit 18 excludes utterances for which the inference label inferred by the trained model is clearly incorrect from the utterance-specific evaluation result (step S21).

[0078] In this embodiment, the inference error elimination unit 18 has been described using an example in which it excludes clearly erroneous utterances from the utterance-specific evaluation results, but the present disclosure is not limited to this. Essentially, the inference error elimination unit 18 only needs to exclude clearly erroneous utterances from the evaluation results and the teacher data confirmation screen. Therefore, the inference error elimination unit 18 may be provided, for example, between the label inference unit 12 and the call-specific inference result evaluation unit 13 and the utterance-specific inference result evaluation unit 15.

[0079] As described above, in this embodiment, the assistance device 10A further includes an inference error exclusion unit 18 that excludes elements for which the inference label inferred by the trained model is determined to be incorrect according to a predetermined rule.

[0080] Therefore, obvious errors are eliminated, reducing the amount of training data that the worker must check, and improving the efficiency of the training data correction work.

[0081] (Third embodiment) 11 is a diagram illustrating an example of the functional configuration of a support device 10B according to a third embodiment of the present disclosure. The support device 10B according to this embodiment supports the evaluation of a teacher data creator who creates teacher data by assigning labels to elements that make up the teacher data. In FIG. 11, the same components as those in FIG. 2 are denoted by the same reference numerals, and their description will be omitted.

[0082] 11, the support device 10B according to this embodiment includes a model learning unit 11, a label inference unit 12, a call-specific inference result evaluation unit 13B, a call-specific confirmation screen generation unit 14B, an utterance-specific inference result evaluation unit 15B, an utterance-specific confirmation screen generation unit 16B, and a teacher data creator evaluation unit 21. The call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, the utterance-specific confirmation screen generation unit 16B, and the teacher data creator evaluation unit 21 constitute an evaluation unit 17B. That is, the support device 10B of this embodiment differs from the support device 10 of the first embodiment in that the call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 have been changed to a call-specific inference result evaluation unit 13B, a call-specific confirmation screen generation unit 14B, an utterance-specific inference result evaluation unit 15B, and an utterance-specific confirmation screen generation unit 16B, respectively, and in that a teacher data creator evaluation unit 21 has been added.

[0083] The evaluation unit 17B generates an evaluation result of the teacher data creator based on a comparison between the correct labels of the elements constituting the teacher data and the inferred labels of the elements inferred by the label inference unit 12. As described above, the call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, the utterance-specific confirmation screen generation unit 16B, and the teacher data creator evaluation unit 21 constitute the evaluation unit 17B.

[0084] The call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, and the utterance-specific confirmation screen generation unit 16B receive input of teacher data creator information, which identifies the teacher data creator who created the teacher data used to create the trained model. As described above, creating a model with estimation accuracy that is practically usable requires a large amount of teacher data. Therefore, teacher data is usually created by multiple teacher data workers. The teacher data creator information is information that identifies each of the multiple teacher data creators who created the teacher data.

[0085] The call-specific inference result evaluation unit 13B, like the call-specific inference result evaluation unit 13, evaluates the teacher data and the inference results of the label inference unit 12 for each call, and outputs the evaluation result (call-specific evaluation result) to the call-specific confirmation screen generation unit 14B and the external output interface 1. Here, the call-specific inference result evaluation unit 13B generates a call-specific evaluation result for each teacher data creator based on the teacher data creator information. That is, the call-specific inference result evaluation unit 13B, which is part of the evaluation unit 17B, generates an evaluation result for each element group by comparing the correct label of the element that makes up the element group with the inferred label for each teacher data creator. As will be described in detail later, the call-specific inference result evaluation unit 13B may switch between presenting the call-specific evaluation results generated for each teacher data creator.

[0086] Similar to the call-specific confirmation screen generation unit 14, the call-specific confirmation screen generation unit 14B generates a teacher data confirmation screen for each call (call-specific confirmation screen) based on the call-specific evaluation results output from the call-specific inference result evaluation unit 13B, and outputs it to the external output interface 1. Here, the call-specific confirmation screen generation unit 14B generates a call-specific confirmation screen for each teacher data creator based on the teacher data creator information. That is, the call-specific confirmation screen generation unit 14B, which is part of the evaluation unit 17B, generates a teacher data confirmation screen for each element group, including the elements that make up the element group, the correct answer labels of the elements, and the inference labels of the elements, for each teacher data creator. As will be described in detail later, the call-specific confirmation screen generation unit 14B may switch between teacher data confirmation screens generated for the same teacher data creator.

[0087] Like the utterance-specific inference result evaluation unit 15, the utterance-specific inference result evaluation unit 15B evaluates the teacher data and the inference results of the label inference unit 12 for each utterance, and outputs the evaluation results (utterance-specific evaluation results) to the utterance-specific confirmation screen generation unit 16B and the external output interface 1. That is, the utterance-specific inference result evaluation unit 15B, which constitutes the evaluation unit 17B, generates evaluation results for each teacher data creator based on a comparison between the correct label and the inferred label for each element that constitutes the teacher data.

[0088] Similar to the utterance-specific confirmation screen generation unit 16, the utterance-specific confirmation screen generation unit 16B generates a teacher data confirmation screen (utterance-specific confirmation screen) for each utterance based on the utterance-specific evaluation results output from the utterance-specific inference result evaluation unit 15B, and outputs the generated screen to the external output interface 1. Here, the utterance-specific confirmation screen generation unit 16B generates an utterance-specific confirmation screen for each teacher data creator based on the teacher data creator information. That is, the utterance-specific confirmation screen generation unit 16B constituting the evaluation unit 17B generates a teacher data confirmation screen for each teacher data creator, the teacher data confirmation screen including the elements constituting the teacher data, the correct answer labels of the elements, and the inference labels of the elements. As will be described in detail later, the utterance-specific confirmation screen generation unit 16B may generate an utterance-specific confirmation screen (a screen on which the evaluation results for each element group can be confirmed) that can be switched for each teacher data creator.

[0089] The teacher data creator evaluation unit 21 receives the teacher data, the inference result by the label inference unit 12, and the teacher data creator information as input. The teacher data creator evaluation unit 21 generates an evaluation result of the teacher data creator (hereinafter referred to as the "teacher data creator evaluation result") based on a comparison between the correct labels of the elements that make up the teacher data and the inferred labels of those elements, and outputs the evaluation result to the external output interface 1.

[0090] In this embodiment, the evaluation results of the teacher data creators are generated based on a comparison between the correct labels assigned to the elements that make up the teacher data and the inferred labels of those elements, allowing for more efficient evaluation of the teacher data creators. Furthermore, it is possible to perform a detailed analysis of the tendency of errors made when creating teacher data for each teacher data creator, enabling efficient training of teacher data creators on the teacher data creation policy.

[0091] Next, the operation of the support device 10B according to this embodiment will be described.

[0092] 12 is a flowchart showing an example of the operation of the support device 10B, and is a diagram for explaining the support method by the support device 10B according to this embodiment. In FIG. 12, the same processes as those in FIG. 3 are denoted by the same reference numerals, and the explanation thereof will be omitted.

[0093] When the label inference unit 12 infers the inferred labels of the elements that make up the teacher data (step S12), the teacher data creator evaluation unit 21 generates a teacher data creator evaluation result based on a comparison between the correct label of the element that makes up the teacher data and the inferred label of that element, and outputs it to the external output interface 1 (step S31).

[0094] FIG. 13 is a diagram showing an example of the teacher data creator evaluation result.

[0095] As shown in FIG. 13, the teacher data creator evaluation unit 21 outputs, as a teacher data creator evaluation result, a teacher data creator index, which is identification information for identifying the teacher data creator, and an evaluation value of the teacher data created by that teacher data creator, in association with each other. The evaluation value of the teacher data is, for example, the average value of values such as the precision, recall, F-score, or match rate of the inferred label relative to the correct labels of multiple teacher data created by the teacher data creator. That is, the teacher data creator evaluation unit 21 generates an evaluation result for each element group based on a comparison between the correct labels and the inferred labels corresponding to the elements constituting the element group, so that it can be confirmed for each teacher data creator. Teacher data creators with high evaluation values for the teacher data they created are likely to have assigned appropriate labels. On the other hand, teacher data creators with low evaluation values for the teacher data they created are likely to have not assigned appropriate labels and to require training, such as learning label assignment policies. The teacher data creator evaluation unit 21 outputs the teacher data index and evaluation values, for example, in descending order of evaluation value. This makes it easy to identify training data creators who are likely to create low-quality training data and require training such as learning labeling policies.

[0096] Referring again to FIG. 12, the call-specific inference result evaluation unit 13B evaluates the correct label of the training data and the inference result of the label inference unit 12 for each call, and outputs the call-specific evaluation result (step S32).

[0097] FIG. 14 is a diagram showing an example of the call-specific inference result output by the call-specific inference result evaluation unit 13B.

[0098] 14, the call-specific inference result evaluation unit 13B, like the call-specific inference result evaluation unit 13, outputs call indexes and evaluation values such as the match rate for the calls in association with each other as call evaluation results. Also, like the call-specific inference result evaluation unit 13, the call-specific inference result evaluation unit 13B may list the call indexes and evaluation values in descending order of evaluation results and output them as text data, for example. The call-specific evaluation results may include the start and end times of the calls.

[0099] The call-specific inference result evaluation unit 13B generates call-specific evaluation results for each teacher data creator, as shown in Figure 14. The call-specific inference result evaluation unit 13 may then switch between presenting call-specific evaluation results for each teacher data creator. By generating call-specific evaluation results for each teacher data creator, it is possible to easily grasp the labeling trends for each teacher data creator.

[0100] Referring back to FIG. 12, the call-specific confirmation screen generating unit 14B generates a call-specific confirmation screen based on the call-specific evaluation result (step S33), and outputs it to the external output interface 1.

[0101] FIG. 15 is a diagram showing an example of the call-specific confirmation screen.

[0102] As shown in FIG. 15, the call-specific confirmation screen generation unit 14B, like the call-specific confirmation screen generation unit 14, generates a call-specific confirmation screen for each call, including the start time of the utterance constituting the call, the end time of the utterance, the utterance text, and the correct label and inference label of the utterance text. Here, the call-specific confirmation screen generation unit 14B generates a call-specific confirmation screen for each teacher data creator. In order to indicate for which teacher data creator the call-specific confirmation screen has been generated, the call-specific confirmation screen generation unit 14B includes a teacher data creator index in the call-specific confirmation screen, as shown in FIG. 15. The call-specific confirmation screen generation unit 14B may superimpose and switchably present call-specific confirmation screens generated for the same teacher data creator, as shown in FIG. 15. That is, the call-specific confirmation screen generation unit 14B may generate a teacher data confirmation screen (call-specific confirmation screen) for each teacher data creator, which includes, for each element group, the elements constituting the element group, the correct label corresponding to the element, and the inference label of the element, and which can be switched for each element group. In this case, the call-specific confirmation screen generating unit 14B may display the call-specific confirmation screen closer to the front for calls with worse evaluation results.

[0103] Referring again to FIG. 12, the utterance-specific inference result evaluation unit 15B evaluates the teacher data and the inference result of the label inference unit 12 for each utterance, and outputs the evaluation result (utterance-specific evaluation result) (step S34).

[0104] FIG. 16 is a diagram showing an example of the evaluation results for each utterance.

[0105] As shown in FIG. 16, the utterance-specific inference result evaluation unit 15B, like the utterance-specific inference result evaluation unit 15, outputs, as text data, the utterance-specific evaluation results, such as a confusion matrix showing the frequency of occurrence of difference patterns and evaluation values for each label (precision, recall, F-score (f1-score), and number of occurrences (support)). The utterance-specific inference result evaluation unit 15B outputs the utterance-specific evaluation results for each teacher data creator. To indicate the teacher data creator for whom the utterance-specific evaluation results were output, the utterance-specific inference result evaluation unit 15B includes a teacher data creator index in the utterance-specific evaluation results, as shown in FIG. 16. Outputting the utterance-specific evaluation results for each teacher data creator makes it possible to identify difference patterns that the teacher data creator is likely to label incorrectly. This also makes it easier for the teacher data creator or their administrator to identify errors in the labeling policy. As described above, the evaluation results for each utterance include evaluation results of the training data created by the training data creator, such as the frequency of appearance of difference patterns and evaluation values for each label. Therefore, the evaluation results for each utterance may be output as the training data creator evaluation results.

[0106] The utterance-specific inference result evaluation unit 15B may display, in a ranking format, difference patterns that are likely to cause confusion, as shown in FIG. 17, instead of the evaluation values for each label shown in FIG. 16. A difference pattern that is likely to cause confusion is a pattern in which the correct label and the inferred label are different, and confusion or substitution is likely to occur. The number of difference patterns that are likely to cause confusion is, for example, the total number of utterances in which the correct label is A and the inferred label is B, and the number of utterances in which the correct label is B and the inferred label is A. The utterance-specific inference result evaluation unit 15B may also include difference patterns that are likely to cause confusion in the utterance-specific evaluation results. This allows the teacher data creator to understand difference patterns that are likely to cause mistakes (labels that are difficult to assign). Furthermore, the administrator of the teacher data creator can notice any misunderstandings in the label assignment policy of each teacher data creator.

[0107] Referring again to FIG. 12, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen based on the utterance-specific evaluation result (step S35), and outputs it to the external output interface 1.

[0108] FIG. 18 is a diagram showing an example of the confirmation screen for each utterance.

[0109] 18, the utterance-specific confirmation screen generating unit 16B, like the utterance-specific confirmation screen generating unit 16, generates an utterance-specific confirmation screen in which utterance text, a line number indicating the order of the utterance text in the call including the utterance, and the correct label and inference label of the utterance text are associated with each other. Here, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen for each teacher data creator. That is, the utterance-specific confirmation screen generating unit 16B generates a teacher data confirmation screen (utterance-specific confirmation screen) including, for each element, the correct label corresponding to the element, and the inference label of the element so that it can be confirmed for each teacher data creator.

[0110] Note that, similarly to the utterance-specific confirmation screen generation unit 16, the utterance-specific confirmation screen generation unit 16B may generate and present utterance-specific confirmation screens in order of utterance text including difference patterns that appear frequently. That is, the utterance-specific confirmation screen generation unit 16B may present utterance-specific confirmation screens in order of elements including difference patterns that appear frequently among difference patterns that are patterns in which the correct label assigned to the teacher data and the inferred label by the trained model differ. Furthermore, the utterance-specific confirmation screen generation unit 16B may present multiple utterance-specific confirmation screens generated for the same teacher data creator in a switchable manner.

[0111] As described above, the support device 10B according to this embodiment includes a label inference unit 12 and an evaluation unit 17B. The label inference unit 12 infers inferred labels, which are labels corresponding to elements constituting the training data, using a model that infers labels corresponding to elements and that has been trained using training data. The evaluation unit 17 generates an evaluation result for the training data creator based on a comparison between the correct labels of the elements constituting the training data and the inferred labels of the elements.

[0112] The support method according to this embodiment includes an inference step and an evaluation result generation step. In the inference step, an inferred label, which is a label corresponding to an element constituting the training data, is inferred using a model that is trained using the training data and that infers a label corresponding to an element. In the evaluation result generation step, an evaluation result for the training data creator is generated based on a comparison between the correct label of the element constituting the training data and the inferred label of the element.

[0113] By generating evaluation results for teacher data creators based on a comparison between the correct labels of the elements that make up the teacher data and the inferred labels of those elements, it is possible to more efficiently evaluate teacher data creators. In addition, it is possible to perform a detailed analysis of the error trends when creating teacher data for each teacher data creator, allowing for efficient training of teacher data creators on creation policies.

[0114] A computer can be suitably used to function as each of the above-described support devices 10, 10A, and 10B. Such a computer can be realized by storing a program describing the processing content for realizing the function of each of the support devices 10, 10A, and 10B in a memory unit of the computer, and having the CPU (Central Processing Unit) of the computer read and execute the program. In other words, the program can cause the computer to function as the above-described support devices 10, 10A, and 10B.

[0115] The following additional notes are provided regarding the above-described embodiments.

[0116] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Inferring inferred labels, which are labels corresponding to elements constituting the training data, using a model that infers labels corresponding to the elements and that has been trained using training data consisting of pairs of elements and correct labels corresponding to the elements; An assistance device that generates an evaluation result for the teacher data creator based on a comparison between a correct label corresponding to an element that constitutes the teacher data and an inferred label of the element.

[0117] (Additional note 2) A non-transitory storage medium storing a program executable by a computer, the program causing the computer to function as the support device described in appended claim 1.

[0118] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0119] 10,10A,10B Support device 11 Model Learning Section 12 Label inference part 13, 13B Call-specific inference result evaluation unit 14,14B Confirmation screen generation section for each call 15,15B Utterance-specific inference result evaluation section 16,16B Utterance-specific confirmation screen generation unit 17 Evaluation Section 18 Inference Error Elimination Section 21 Teacher Data Creator Evaluation Department 110 processors 120 ROM 130 RAM 140 Storage 150 Input section 160 Display section 170 Communication Interface 190 Bus

Claims

1. A support device for supporting evaluation of a teacher data creator who creates teacher data consisting of a pair of an element and a correct label corresponding to the element, an evaluation unit that presents, for each of the elements, a type of difference pattern in which the correct label corresponding to the element and the inferred label corresponding to the element are different, and which indicates confusion or interchange of the correct label and the inferred label; An assistance device, wherein the inferred label is a label inferred using an inference model trained using the training data as a label corresponding to the element.

2. 2. The support device according to claim 1, The evaluation unit generating a teacher data confirmation screen including, for each element constituting the teacher data, a correct answer label corresponding to the element and an inference label of the element so that the screen can be confirmed by each of the teacher data creators; An assistance device that generates the teacher data confirmation screen so that elements including the difference patterns that appear most frequently can be confirmed in order.

3. 2. The support device according to claim 1, The evaluation unit presents the difference patterns in a ranking format in order of the frequency of appearance of each type of difference pattern.

4. 2. The support device according to claim 1, The evaluation unit presents the type of difference pattern in a switchable manner for each teacher data creator.

5. A support method for a support device that supports evaluation of a teacher data creator who creates teacher data consisting of a pair of an element and a correct label corresponding to the element, comprising: a step of presenting, for each of the elements, a type of difference pattern in which the correct label corresponding to the element and the inferred label corresponding to the element are different, and which indicates confusion or interchange of the correct label and the inferred label; An assistance method in which the inferred label is a label inferred using an inference model trained using the training data as a label corresponding to the element.

6. A program for causing a computer to function as the support device according to any one of claims 1 to 4.

Images