Support device, support method, and program

Abstract

To provide a support device, a support method, and a program capable of efficiently evaluating training data creators.SOLUTION: A training data confirmation support device 10 includes an evaluation unit 17 which presents types of difference patterns indicating confusion or replacement of correct labels and inference labels, for each of a plurality of elements, the correct labels corresponding to the elements being different from the inference labels corresponding to the element. The inference labels are labels inferred, as labels corresponding to the elements, using an inference model trained using training data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an assistance device, an assistance method, and a program. [Background technology]

[0002] In recent years, with the aim of improving the quality of service in contact centers, systems have been proposed that perform real-time speech recognition of call contents and automatically present appropriate information to an operator by making full use of natural language processing technology.

[0003] For example, Non-Patent Document 1 discloses a technology that presents anticipated questions and answers (FAQs) to an operator in a dialogue between an operator and a customer. In this technology, the dialogue between an operator and a customer is speech-recognized, and converted into a semantically coherent spoken text by a "speech end determination" that determines whether the speaker has finished speaking. Next, a "message scene estimation" is performed to estimate which interaction scene in the dialogue the utterance corresponding to the speech text is from, such as the operator's greeting, confirmation of the customer's business, response to the business, or closing the dialogue. The dialogue is structured by the "message scene estimation". From the result of the "message scene estimation", a "FAQ search utterance determination" is performed to extract an utterance including the customer's business or an utterance by the operator confirming the customer's business. A search is performed on a database of FAQs prepared in advance using a search query based on the utterance extracted by the "FAQ search utterance determination", and the search results are presented to the operator.

[0004] The above-mentioned "end of speech determination", "interaction scene estimation" and "FAQ search utterance determination" use a model constructed by learning, using a deep neural network or the like, training data in which labels that classify utterances are assigned to speech texts. Therefore, "end of speech determination", "interaction scene estimation" and "FAQ search utterance determination" can be regarded as a sequence labeling problem in which sequential elements (utterances in a dialogue) are labeled. Non-Patent Document 2 describes a technology for estimating an interaction scene by learning, using a deep neural network including long and short-term memory, a large amount of training data in which sequential utterances are assigned labels corresponding to the interaction scenes in which the utterances are included. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] T. Hasegawa, Y. Sekiguchi, S. Yamada, and M. Tamoto, "Automatic Knowledge Support System to Support Operators' Responses," NTT Technical Review, vol. 31, no. 7, pp. 16-19, Jul. 2019. [Non-Patent Document 2] R. Masumura, S. Yamada, T. Tanaka, A. Ando, ​​H. Kamiyama, and Y. Aono, “Online Call Scene Segmentation of Contact Center Dialogues based on Role Aware Hierarchical LSTM-RNNs,” Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Nov. 2018. Summary of the Invention [Problem to be solved by the invention]

[0006] The techniques described in the above-mentioned Non-Patent Documents 1 and 2 require a large amount of training data to achieve a practical level of estimation accuracy. For example, according to Non-Patent Document 1, high estimation accuracy can be obtained by creating training data from conversation logs of a call center of about 1,000 calls and learning a model. The training data is created by an operator (training data creator) assigning a label to each spoken text while referring to the spoken text obtained by speech recognition of the spoken voice.

[0007] The training data needs to be created according to the application (for example, each industry of contact centers) of the model trained using the training data. As described above, a large amount of training data is required to obtain high estimation accuracy, so the task of creating training data to which labels are assigned is often performed by multiple workers. Here, since each worker has different experience or detailed policies for label assignment, there may be a variation in label assignment, in which different labels are assigned even for utterances of the same content. If label inconsistencies occur in the training data, this will lead to a decrease in the estimation accuracy of the model trained using the training data. Here, there is no established method for efficiently checking which training data creator created the training data that caused the variation in label assignment, and in the past, it was necessary to repeat analysis based on tacit knowledge by experts or trial and error.

[0008] Therefore, there is a demand for technology that can more efficiently evaluate training data creators.

[0009] In consideration of the above problems, an object of the present disclosure is to provide an assistance device, an assistance method, and a program that can more efficiently evaluate teacher data creators. [Means for solving the problem]

[0010] In order to solve the above problem, the support device disclosed herein is a support device that supports the evaluation of a teacher data creator who creates teacher data consisting of a pair of elements and correct labels corresponding to the elements, and includes a label inference unit that infers an inferred label, which is a label corresponding to an element that constitutes the teacher data, using a model that is learned using the teacher data and infers a label corresponding to the element, and an evaluation unit that generates an evaluation result of the teacher data creator based on a comparison between the correct label corresponding to the element that constitutes the teacher data and the inferred label of the element.

[0011] In addition, in order to solve the above problem, the assistance method disclosed herein is an assistance method in an assistance device that supports the evaluation of a teacher data creator who creates teacher data consisting of a pair of elements and correct labels corresponding to the elements, and includes a step of inferring an inferred label, which is a label corresponding to an element that constitutes the teacher data, using a model that is learned using the teacher data and infers a label corresponding to the element, and a step of generating an evaluation result of the teacher data creator based on a comparison between the correct label corresponding to the element that constitutes the teacher data and the inferred label of the element.

[0012] In order to solve the above problem, a program according to the present disclosure causes a computer to function as the above-mentioned assistance device. Effect of the Invention

[0013] According to the support device, support method, and program of the present disclosure, evaluation of teacher data creators can be carried out more efficiently. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a schematic configuration of a computer functioning as a support device according to a first embodiment of the present disclosure. [Diagram 2] 2 is a diagram illustrating an example of a functional configuration of an assistance device according to a first embodiment of the present disclosure. [Diagram 3]3 is a flowchart showing an example of the operation of the support device shown in FIG. 2. [Figure 4] 3 is a diagram showing an example of a call-specific evaluation result by the call-specific inference result evaluation unit shown in FIG. 2. FIG. [Diagram 5] 3 is a diagram showing an example of a call-specific confirmation screen generated by a call-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 6] 3 is a diagram showing another example of the call-specific confirmation screen generated by the call-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 7] 3 is a diagram showing an example of an utterance-specific evaluation result by an utterance-specific inference result evaluation unit shown in FIG. 2. [Figure 8] 3 is a diagram showing an example of an utterance-specific confirmation screen generated by an utterance-specific confirmation screen generating unit shown in FIG. 2. FIG. [Figure 9] FIG. 11 is a diagram illustrating an example of a functional configuration of an assistance device according to a second embodiment of the present disclosure. [Figure 10] 10 is a flowchart showing an example of the operation of the support device shown in FIG. [Figure 11] FIG. 13 is a diagram illustrating an example of a functional configuration of an assistance device according to a third embodiment of the present disclosure. [Figure 12] 12 is a flowchart showing an example of the operation of the support device shown in FIG. [Figure 13] FIG. 12 is a diagram showing an example of a teacher data creator evaluation result by the teacher data creator evaluation unit shown in FIG. 11. [Figure 14] 12 is a diagram showing an example of a call-specific evaluation result by the call-specific inference result evaluation unit shown in FIG. 11. FIG. [Figure 15] 12 is a diagram showing an example of a call-specific confirmation screen generated by a call-specific confirmation screen generating unit shown in FIG. 11. FIG. [Figure 16] 12 is a diagram showing an example of an utterance-specific evaluation result by the utterance-specific inference result evaluation unit shown in FIG. 11. [Figure 17] 12 is a diagram showing another example of the utterance-specific evaluation result by the utterance-specific inference result evaluation unit shown in FIG. 11. [Figure 18] 12 is a diagram showing an example of an utterance-specific confirmation screen generated by an utterance-specific confirmation screen generating unit shown in FIG. 11. FIG. [Figure 19]FIG. 13 is a diagram showing an example of a label structure made up of a plurality of items. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0016] (First embodiment) 1 is a block diagram showing a hardware configuration of an assistance device 10 according to a first embodiment of the present disclosure, which is a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0017] 1, the support device 10 has a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a storage 140, an input unit 150, a display unit 160, and a communication interface (I / F) 170. Each component is connected to each other so as to be able to communicate with each other via a bus 190. The processor 110 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types.

[0018] The processor 110 controls each of the components and executes various arithmetic processing. That is, the processor 110 reads a program from the ROM 120 or the storage 140, and executes the program using the RAM 130 as a working area. The processor 110 controls each of the components and executes various arithmetic processing according to the program stored in the ROM 120 or the storage 140. In this embodiment, the ROM 120 or the storage 140 stores a program according to the present disclosure.

[0019] The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0020] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is configured with a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system and various data.

[0021] The input unit 150 includes a pointing device such as a mouse and a keyboard, and is used to perform various input operations.

[0022] The display unit 160 is, for example, a liquid crystal display, and displays various information. The display unit 160 may function as the input unit 150 by adopting a touch panel system.

[0023] The communication interface 170 is an interface for communicating with other devices such as external devices (not shown), and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0024] Next, the functional configuration of the support device 10 according to this embodiment will be described.

[0025] FIG. 2 is a diagram showing an example of the configuration of the support device 10 according to the present embodiment. The support device 10 according to the present embodiment supports the confirmation of teacher data, such as confirmation of deviation of the label assignment criteria, by an operator who creates teacher data consisting of a pair of an element and a label (hereinafter referred to as a "correct label") assigned to the element. The correct label is a label assigned when the teacher data is created and is the subject of the confirmation. Therefore, the assigned correct label is not necessarily correct. Supporting the confirmation of teacher data also makes it easier to extract labels that need to be corrected, and the efficiency of the teacher data correction work can be improved. In the following, an example of assigning a label to a speech text obtained by speech recognition of an utterance in a dialogue between multiple speakers (operators and customers) at a contact center, as shown in FIG. 19, will be described. In FIG. 19, the speech text corresponding to the operator's utterance (hereinafter, the speech text corresponding to the utterance may be simply referred to as the "speech text") is shown in a solid line speech bubble, and the speech text of the customer is shown in a dotted line speech bubble.

[0026] In the example shown in FIG. 19, teacher data for "end of speech determination" is created by adding an end of speech label indicating whether the utterance is an end of speech to each utterance text. In addition, teacher data for "response scene estimation" is created by adding a scene label indicating a response scene in which the utterance is included to each utterance text. In addition, teacher data for "FAQ search utterance determination" is created by adding a subject label indicating that the utterance is an utterance indicating the customer's subject among utterances included in the response scene of "subject understanding" in which the customer's subject is understood, and adding a subject confirmation label indicating that the utterance is an utterance confirming the customer's subject to an utterance in which the operator confirms the customer's subject. However, the present disclosure is not limited to the example of the teacher data shown in FIG. 19, and can be applied to teacher data consisting of a set of any multiple elements and the labels of each element. In addition, the utterance text may be not only a text of an utterance in a call, but also an utterance in a text dialogue such as a chat. Furthermore, the speaker in the dialogue is not limited to a human being, but may be a robot or a virtual agent.

[0027] 2, the support device 10 according to the present embodiment includes a model learning unit 11, a label inference unit 12, a call-specific inference result evaluation unit 13, a call-specific confirmation screen generation unit 14, an utterance-specific inference result evaluation unit 15, and an utterance-specific confirmation screen generation unit 16. The call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 constitute an evaluation unit 17. The model learning unit 11, the label inference unit 12, the call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be configured by one or more processors as described above.

[0028] The model learning unit 11 receives training data consisting of a pair of spoken text (elements) and a correct label assigned to the spoken text. The model learning unit 11 uses the input training data to learn a model that infers a label corresponding to the spoken text. Any learning method can be applied as a model learning method according to the purpose of the system to which the model is applied. The model learning unit 11 outputs a model (hereinafter referred to as a "trained model") created by learning the training data to the label inference unit 12. The trained model may be prepared in advance. Therefore, the assistance device 10 does not need to include the model learning unit 11.

[0029] The label inference unit 12 receives the teacher data and the trained model created by the model learning unit 11. The teacher data input to the label inference unit 12 is the same as the teacher data used to train the trained model. The label inference unit 12 uses the trained model to infer the labels of the spoken texts (elements) that make up the teacher data (hereinafter, the labels inferred by the trained model are referred to as "inferred labels"). The label inference unit 12 outputs the inference labels of the respective spoken texts that make up the teacher data as inference results to the call-specific inference result evaluation unit 13 and the utterance-specific inference result evaluation unit 15.

[0030] The evaluation unit 17 compares and evaluates the correct labels given to the elements constituting the teacher data with the inferred labels inferred by the label inference unit 12, and outputs the evaluation result to the external output interface 1. The evaluation unit 17 also generates a teacher data confirmation screen for confirming the teacher data, which includes the elements constituting the teacher data, the correct labels given to the elements, and the inferred labels of the elements. The evaluation unit 17 outputs the generated teacher data confirmation screen to the external output interface 1.

[0031] The external output interface 1 is a device used by a worker who creates and modifies teacher data or an administrator who manages the work of the worker. The external output interface 1 presents, for example, a comparison result between a correct answer label assigned to the teacher data output from the evaluation unit 17 and an inferred label inferred by a trained model. The external output interface 1 may have any configuration as long as it has a function of communicating with the support device 10, a function of presenting (displaying) the evaluation result of the evaluation unit 17 and a teacher data confirmation screen, and a function of accepting operation input.

[0032] As described above, the evaluation unit 17 includes the call-specific inference result evaluation unit 13 , the call-specific confirmation screen generation unit 14 , the utterance-specific inference result evaluation unit 15 , and the utterance-specific confirmation screen generation unit 16 .

[0033] The call-specific inference result evaluation unit 13 receives the teacher data and the inference result of the label inference unit 12 as input. Typically, the teacher data includes a group of speech texts consisting of a plurality of speech texts in calls by a plurality of speakers, the number of which is equal to the number of calls. In other words, the teacher data includes a plurality of element groups consisting of a plurality of sequential elements. The call-specific inference result evaluation unit 13 evaluates the input teacher data and the inference result of the label inference unit 12 for each call. The call-specific inference result evaluation unit 13 outputs the evaluation result (call-specific evaluation result) to the call-specific confirmation screen generation unit 14 and the external output interface 1. The call-specific evaluation result will be described in detail later.

[0034] The call-specific confirmation screen generation unit 14 generates a teacher data confirmation screen for each call (hereinafter referred to as a "call-specific confirmation screen") based on the call-specific evaluation result output from the call-specific inference result evaluation unit 13, and outputs it to the external output interface 1. The call-specific confirmation screen will be described in detail later.

[0035] The utterance-specific inference result evaluation unit 15 receives the teacher data and the inference result of the label inference unit 12. The utterance-specific inference result evaluation unit 15 evaluates the input teacher data and the inference result of the label inference unit 12 for each utterance. The utterance-specific inference result evaluation unit 15 outputs the evaluation result (utterance-specific evaluation result) to the utterance-specific confirmation screen generation unit 16 and the external output interface 1. The utterance-specific evaluation result will be described in detail later.

[0036] The utterance-specific confirmation screen generating unit 16 generates a teacher data confirmation screen for each utterance (hereinafter referred to as an "utterance-specific confirmation screen") based on the utterance-specific evaluation result output from the utterance-specific inference result evaluating unit 15, and outputs it to the external output interface 1. The utterance-specific confirmation screen will be described in detail later.

[0037] In this embodiment, a teacher data confirmation screen is generated that includes the spoken text (elements) that constitute the teacher data, the correct answer label assigned to the spoken text, and the inferred label inferred by the trained model trained using the teacher data. Therefore, according to the support device 10 according to this embodiment, the worker can easily check the teacher data by comparing the correct answer label and the inferred label of the element on the teacher data confirmation screen, so that the efficiency of the teacher data confirmation work can be improved. In addition, by improving the efficiency of the teacher data confirmation work, it becomes easier to extract labels that need to be corrected, and the efficiency of the label correction work can also be improved.

[0038] Next, the operation of the support device 10 according to this embodiment will be described.

[0039] FIG. 3 is a flowchart showing an example of the operation of the support device 10, and is a diagram for explaining a support method by the support device 10 according to this embodiment.

[0040] The model learning unit 11 uses training data to learn a model that infers labels that distinguish spoken text (step S11).

[0041] The label inference unit 12 infers an inference label corresponding to an element of the teacher data by using the trained model trained by the model training unit 11 (step S12). As described above, the teacher data used for training the trained model is the same as the teacher data used for the teacher data inference process by the label inference unit 12.

[0042] The call-specific inference result evaluation unit 13 evaluates the teacher data and the inference result of the label inference unit 12 for each call, and outputs the evaluation result (call-specific evaluation result) (step S13). Specifically, the call-specific inference result evaluation unit 13 compares the difference between the correct label assigned to the utterance text constituting the teacher data and the inferred label inferred by the label inference unit 12 for each call. Then, the call-specific inference result evaluation unit 13 outputs the evaluation value for each call as the call-specific evaluation result, arranging the evaluation values ​​for each call in order from the call with the worst evaluation result (for example, the utterance with the evaluation value equal to or less than a threshold). That is, the call-specific inference result evaluation unit 13 outputs the evaluation result for each element group in order from the element group with the worst evaluation result (the call consisting of a plurality of utterances). The evaluation value of the call can be the matching rate, recall rate, F-value, or matching rate between the correct label and the inferred label of each utterance text constituting the teacher data.

[0043] FIG. 4 is a diagram showing an example of the call-specific evaluation result.

[0044] As shown in Fig. 4, the call-specific inference result evaluation unit 13 outputs, as a call-specific evaluation result, a call index, which is identification information for identifying a call, and an evaluation value such as a match rate for that call, in association with each other. Here, the call-specific inference result evaluation unit 13 lists the call indexes and evaluation values ​​in order of worsening evaluation result, and outputs them, for example, as text data. The call-specific evaluation result may include the start time and end time of the call.

[0045] Referring back to FIG. 3, the call-specific confirmation screen generating unit 14 generates a call-specific confirmation screen based on the call-specific evaluation result and outputs it to the external output interface 1 (step S14).

[0046] FIG. 5 is a diagram showing an example of the call-specific confirmation screen.

[0047] As shown in FIG. 5, the call-specific confirmation screen generating unit 14 generates a call-specific confirmation screen for each call, including a start time that is the time when the utterance constituting the call started, an end time that is the time when the utterance ended, an utterance text, and a correct label and an inference label of each utterance text. In this way, the call-specific confirmation screen generating unit 14 generates a teacher data confirmation screen that includes elements constituting the teacher data, a correct label of the element, and an inference label of the element. Specifically, the call-specific confirmation screen generating unit 14 generates a teacher data confirmation screen that shows the correct label and the inference label corresponding to the element constituting the teacher data in a contrastive manner (for example, as shown in FIG. 5, the correct label and the inference label corresponding to the element are shown side by side). Here, the call-specific confirmation screen generating unit 14 presents the call-specific confirmation screen in order of the worse evaluation result. For example, as shown in FIG. 5, the call-specific confirmation screen generating unit 14 may display the call-specific confirmation screen closer to the front for calls with worse evaluation results. That is, the call-specific confirmation screen generating unit 14 may generate a call-specific confirmation screen for each element so that calls with poor evaluation results can be checked in order. As described above, the call-specific confirmation screen includes the start time and end time of an utterance. Therefore, the worker can check whether the utterances overlap. Note that the start time and end time do not necessarily have to be included in the call-specific confirmation screen.

[0048] In this way, the call-specific inference result evaluation unit 13 constituting the evaluation unit 17 evaluates, for each element group, the difference between the correct label assigned to the element constituting the element group and the inference label inferred by the trained model. Also, the call-specific confirmation screen generation unit 14 constituting the evaluation unit 17 generates a teacher data confirmation screen (call-specific confirmation screen) for each element group based on the call-specific evaluation results, and presents the call-specific confirmation screen in order of the element groups with the worst evaluation results.

[0049] Furthermore, the call-specific confirmation screen generating unit 14 constituting the evaluation unit 17 may present a call-specific confirmation screen for each call in a switchable manner. In the example shown in Fig. 5, the call-specific confirmation screen generating unit 14 may switch the utterance-specific confirmation screen displayed in the foreground in response to a switching operation by the operator, for example. In this manner, the call-specific confirmation screen generating unit 14 may present the evaluation result for each element group in a switchable manner.

[0050] By presenting the call-specific confirmation screen for each call, the worker can find and correct poor quality teacher data for each call. In addition, by making the call-specific confirmation screen for each call switchable, the worker can, for example, continuously check the evaluation results for each call, thereby improving the efficiency of the teacher data confirmation work. In addition, by generating the call-specific confirmation screen so that the calls with the poorest evaluation results can be checked in order, the worker can find the tendency of poor quality teacher data for each call and grasp the main points of correction. As a result, the efficiency of the teacher data correction work can be improved. Note that, instead of presenting the call-specific confirmation screen shown in FIG. 5 in a switchable manner, the call-specific confirmation screen generating unit 14 may output a group of files of text data corresponding to the call-specific confirmation screen to the external output interface 1 by dividing them into directories based on the evaluation value for each call.

[0051] The call-specific confirmation screen is not limited to the example shown in Fig. 5. Fig. 6 is a diagram showing another example of the call-specific confirmation screen generated by the call-specific confirmation screen generating unit 14.

[0052] The call-specific confirmation screen generating unit 14 may arrange the spoken texts of the operator and the customer in a row in chronological order on the call-specific confirmation screen, as shown in Fig. 6. The call-specific confirmation screen generating unit 14 may also arrange, in association with each spoken text, the start time at which the speech started, the end time at which the speech ended, and the labels assigned to the speech (scene label, subject label, subject confirmation label, and speech end label). As shown in Fig. 6, the call-specific confirmation screen generating unit 14 may display the spoken text of the operator and the spoken text of the customer in different colors. Note that in Fig. 6, the difference in color is expressed by different hatching.

[0053] As shown in FIG. 6, the call-specific confirmation screen generating unit 14 may arrange multiple elements in a row on the call-specific confirmation screen, and may distribute and arrange the labels of multiple items on one side and the other side of the element corresponding to the label based on the structure of the labels of the multiple items.

[0054] Generally, it is easier to check and correct labels if they are placed in an area close to the spoken text. Therefore, by placing the spoken text in a line and distributing labels for multiple items on both sides of the spoken text, we can effectively use the area close to the spoken text and improve the efficiency of checking and correcting labels.

[0055] In the example shown in FIG. 6, the scene label, the subject label, and the subject confirmation label are arranged on the left side of the utterance text, and the end-of-speech label is arranged on the right side of the utterance text. When the scene label, the subject label, and the subject confirmation label are assigned to the utterance text, not only the utterance text itself but also the contents of the utterance texts before and after it are taken into consideration. In other words, the scene label, the subject label, and the subject confirmation label are labels that should take into consideration the long-term context and are determined based on the contents of multiple utterance texts including the utterance text. On the other hand, when the end-of-speech label is assigned to the utterance text, it is sufficient to mainly take into consideration only the utterance text itself. Therefore, the call-specific confirmation screen generating unit 14 may arrange the label that should take into consideration the long-term context on the left side of the utterance text, and the label that does not take into consideration the long-term context on the right side of the utterance text.

[0056] In the example shown in FIG. 6, the call-specific confirmation screen generating unit 14 places the subject label and the subject confirmation label closer to the utterance text than the scene label. Usually, the subject label or the subject confirmation label is added to the utterance text to which the scene label of "understanding the subject" is added. That is, the scene label is a label of a higher hierarchy, and the subject label / subject confirmation label is a label of a lower hierarchy. Therefore, the call-specific confirmation screen generating unit 14 may place the lower-level label of the labels of a plurality of items having a hierarchical structure closer to the utterance text. In this way, the lower the hierarchy label, the easier it is to check and correct the label by looking at the utterance text, so that the work efficiency can be improved. In addition, the end-of-speech label is mainly added with attention to the end of the utterance. Therefore, by placing the end-of-speech label on the right side of the utterance text, the worker can easily see the end of the utterance text, so that the work efficiency of checking and correcting the end-of-speech label can be improved.

[0057] Furthermore, when a label to be corrected is selected by an operator in the correction of teacher data, the call-specific confirmation screen generating unit 14 may change the display mode of the labels (upper layer labels and lower layer labels) related to the label to be corrected based on the hierarchical structure of the labels of multiple items. In the example shown in FIG. 6, it is assumed that the scene label "subject understanding" is selected as the label to be updated. In this case, the call-specific confirmation screen generating unit 14 changes the display mode by, for example, changing the display color of the subject label and the subject confirmation utterance label, which are labels in a lower layer of the scene label. This makes it easier for the operator to understand the labels related to the label to be corrected, and the work efficiency of assigning labels can be improved.

[0058] Furthermore, when updating a label in a higher level or a label in a lower level, if a contradiction occurs between related labels, the call-specific confirmation screen generating unit 14 may change the display mode of the label in question. This can eliminate the contradiction between the labels of multiple items having a hierarchical structure, and improve the accuracy of label correction.

[0059] In addition, the call-specific confirmation screen generating unit 14 may display utterance text that is not the subject of the training data, such as short utterance text such as fillers and "yes", in a different manner from other utterance text. This allows the worker to easily understand utterance text that does not need to be labeled, thereby improving work efficiency.

[0060] 3 again, the utterance-specific inference result evaluation unit 15 evaluates the teacher data and the inference result of the label inference unit 12 for each utterance, and outputs the evaluation result (utterance-specific evaluation result) (step S15). The utterance-specific inference result evaluation unit 15 compares the label of the teacher data with the label of the inference result of the label inference unit 12 for each utterance, tallying up difference patterns that are patterns in which the label of the teacher data and the label of the inference result are different, and outputs the difference patterns as utterance-specific evaluation results.

[0061] FIG. 7 is a diagram showing an example of the evaluation result for each utterance.

[0062] As shown in Figure 7, the utterance-specific inference result evaluation unit 15 outputs, as utterance-specific evaluation results, for example, a confusion matrix showing the number of occurrences of difference patterns, and evaluation values ​​for each label (precision, recall, F-score (f1-score), and number of occurrences (support)) as text data.

[0063] Referring back to FIG. 3, the utterance-specific confirmation screen generating unit 16 generates an utterance-specific confirmation screen based on the utterance-specific evaluation result (step S16), and outputs it to the external output interface 1.

[0064] FIG. 8 is a diagram showing an example of the utterance-specific confirmation screen.

[0065] As shown in FIG. 8, the utterance-specific confirmation screen generating unit 16 generates an utterance-specific confirmation screen in which an utterance text, a line number indicating the order of the utterance text in the call including the utterance, and a correct label and an inference label of the utterance text are associated with each other. In this manner, the utterance-specific confirmation screen generating unit 16 generates a teacher data confirmation screen including elements constituting the teacher data, correct labels assigned to the elements, and inference labels of the elements. Specifically, the utterance-specific confirmation screen generating unit 16 generates a teacher data confirmation screen that displays the correct labels and inference labels corresponding to the elements constituting the teacher data in a contrastive manner (for example, as shown in FIG. 8, the correct labels and inference labels corresponding to the elements are displayed side by side). The utterance-specific confirmation screen generating unit 16 generates an utterance-specific confirmation screen for each utterance in which the correct label and the inference label are different. In FIG. 8, the utterance in line number 41 surrounded by a dotted rectangle is the utterance to be displayed. As shown in FIG. 8, the utterance-specific confirmation screen generating unit 16 may mark the utterance text to be displayed (the utterance text in which the correct answer label and the inference label are different) with a predetermined mark ("**" in FIG. 8). The utterance-specific confirmation screen generating unit 16 may include utterances before and after the utterance to be displayed in the utterance-specific confirmation screen of the utterance to be displayed. That is, the utterance-specific confirmation screen generating unit 16 may generate an utterance-specific confirmation screen including an element whose correct answer label and inference label are different and elements before and after the element. FIG. 8 shows an example in which the utterance text from line number 38 to line number 44 is included in the utterance-specific confirmation screen in which the utterance text in line number 41 is displayed.

[0066] The utterance-specific confirmation screen generating unit 16 presents the utterance-specific confirmation screen for each utterance in the order of utterance text including a difference pattern that appears frequently among difference patterns in which the teacher data and the inference label are different. In other words, the utterance-specific confirmation screen generating unit 16 may present the utterance-specific confirmation screen in the order of elements including a difference pattern that appears frequently among difference patterns in which the teacher data and the inference label are different.

[0067] In this way, the utterance-specific inference result evaluation unit 15 constituting the evaluation unit 17 compares, for each element constituting the teacher data, the correct label assigned to that element with the inference label inferred by the trained model, and outputs the evaluation result. Also, the utterance-specific confirmation screen generation unit 16 constituting the evaluation unit 17 generates and presents, for each element constituting the teacher data, a teacher data confirmation screen (utterance-specific confirmation screen) in order of elements including difference patterns with the most frequent occurrences among difference patterns in which the correct label and the inference label are different.

[0068] The utterance-specific confirmation screen generating unit 16 may present a plurality of utterance-specific confirmation screens in a partially overlapping manner as shown in Fig. 8, and may switch the utterance-specific confirmation screen displayed in the foreground in response to a switching operation by the operator, for example. That is, the utterance-specific confirmation screen generating unit 16 may generate the utterance-specific confirmation screens so that elements including difference patterns with a high frequency of occurrence can be confirmed in order. In this way, only the teacher data to be confirmed can be quickly confirmed in order of the greatest impact.

[0069] By displaying the confirmation screen for each utterance, the worker can find and correct teacher data with an erroneous label for each utterance. In addition, by presenting an element with a different correct label and an inferred label and elements before and after the element, the worker can correct the label of the utterance text to be displayed while taking into consideration the contents of the utterance text (elements) before and after, so that the efficiency of the label correction work can be improved. In addition, by presenting a plurality of confirmation screens for each utterance with the same difference pattern in a switchable manner, the worker can continuously check the confirmation screens for each utterance with the same difference pattern and grasp the main points of correction for each difference pattern. As a result, the efficiency of the correction work of the teacher data can be improved. Note that, instead of presenting the confirmation screen for each utterance shown in FIG. 8 in a switchable manner, the confirmation screen generating unit 16 may output a file group of text data corresponding to the confirmation screen for each utterance to the external output interface 1 by dividing it into directories or the like.

[0070] Thus, the support device 10 according to the present embodiment includes a label inference unit 12 and an evaluation unit 17. The label inference unit 12 infers inference labels of elements constituting the teacher data using a trained model trained using the teacher data. The evaluation unit 17 generates a teacher data confirmation screen including the elements constituting the teacher data, the correct labels assigned to the elements, and the inference labels inferred by the trained model.

[0071] The teacher data correction method according to this embodiment includes a step of inferring a label (step S12) and a step of generating a teacher data confirmation screen (steps S14 and S16). In the step of inferring a label, an inferred label of an element constituting the teacher data is inferred using a trained model trained using the teacher data. In the step of generating the teacher data confirmation screen, a teacher data confirmation screen is generated including the elements constituting the teacher data, the correct labels assigned to the elements, and the inferred labels of the elements.

[0072] In this way, according to the support device 10 and support method of this embodiment, the teacher data confirmation screen, which includes the correct answer label and inference label of the element, makes it easy for the worker to confirm the teacher data, thereby making it possible to make the teacher data confirmation work more efficient.

[0073] Second embodiment Fig. 9 is a diagram showing a configuration example of a support device 10A according to a second embodiment of the present disclosure. In Fig. 9, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0074] The support device 10A according to this embodiment differs from the support device 10 according to the first embodiment in that an inference error removal unit 18 is added.

[0075] The inference error elimination unit 18 receives the evaluation results of each utterance by the utterance-specific inference result evaluation unit 15. The inference error elimination unit 18 performs an inference error elimination process to eliminate elements whose inference labels inferred by the trained model are determined to be erroneous according to a predetermined rule. Specifically, the inference error elimination unit 18 eliminates utterances whose inference labels are obviously erroneous from the evaluation results of the utterance-specific inference result evaluation unit 15. An obviously erroneous utterance is, for example, an utterance in which one scene is composed of only one utterance, or an utterance in which a closing label indicating the end of the call or a label indicating a response to a customer's request is added to the utterance text, even though it is the beginning of the call. The criteria for judging an obviously erroneous utterance are manually determined in advance.

[0076] Next, the operation of the support device 10A according to this embodiment will be described. Fig. 10 is a flowchart showing an example of the operation of the support device 10A. In Fig. 10, the same processes as those in Fig. 3 are given the same reference numerals, and the description will be omitted.

[0077] When the utterance-specific inference result evaluation unit 15 outputs the utterance-specific evaluation result (step S15), the inference error elimination unit 18 excludes utterances in which the inference labels inferred by the trained model are clearly incorrect from the utterance-specific evaluation results (step S21).

[0078] In this embodiment, the inference error elimination unit 18 is described using an example in which it excludes obviously erroneous utterances from the utterance-specific evaluation results, but the present disclosure is not limited to this. In short, the inference error elimination unit 18 only needs to exclude obviously erroneous utterances from the evaluation results and the teacher data confirmation screen. Therefore, the inference error elimination unit 18 may be provided, for example, between the label inference unit 12 and the call-specific inference result evaluation unit 13 and the utterance-specific inference result evaluation unit 15.

[0079] As described above, in this embodiment, the assistance device 10A further includes an inference error exclusion unit 18 that excludes elements for which the inference label inferred by the trained model is determined to be incorrect according to a predetermined rule.

[0080] As a result, obvious errors are eliminated, reducing the amount of training data that workers need to check, and improving the efficiency of the training data correction work.

[0081] (Third embodiment) Fig. 11 is a diagram showing an example of a functional configuration of a support device 10B according to a third embodiment of the present disclosure. The support device 10B according to this embodiment supports the evaluation of a teacher data creator who creates teacher data by adding labels to elements constituting the teacher data. In Fig. 11, the same components as those in Fig. 2 are given the same reference numerals, and the description thereof will be omitted.

[0082] 11, the support device 10B according to this embodiment includes a model learning unit 11, a label inference unit 12, a call-specific inference result evaluation unit 13B, a call-specific confirmation screen generation unit 14B, an utterance-specific inference result evaluation unit 15B, an utterance-specific confirmation screen generation unit 16B, and a teacher data creator evaluation unit 21. The call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, the utterance-specific confirmation screen generation unit 16B, and the teacher data creator evaluation unit 21 configure an evaluation unit 17B. That is, the support device 10B of this embodiment differs from the support device 10 of the first embodiment in that the call-specific inference result evaluation unit 13, the call-specific confirmation screen generation unit 14, the utterance-specific inference result evaluation unit 15, and the utterance-specific confirmation screen generation unit 16 have been changed to call-specific inference result evaluation unit 13B, call-specific confirmation screen generation unit 14B, utterance-specific inference result evaluation unit 15B, and utterance-specific confirmation screen generation unit 16B, respectively, and in that a teacher data creator evaluation unit 21 has been added.

[0083] The evaluation unit 17B generates an evaluation result of the teacher data creator based on a comparison between the correct labels of the elements constituting the teacher data and the inference labels of the elements inferred by the label inference unit 12. As described above, the call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, the utterance-specific confirmation screen generation unit 16B, and the teacher data creator evaluation unit 21 constitute the evaluation unit 17B.

[0084] The call-specific inference result evaluation unit 13B, the call-specific confirmation screen generation unit 14B, the utterance-specific inference result evaluation unit 15B, and the utterance-specific confirmation screen generation unit 16B receive input of teacher data creator information, which is information identifying the teacher data creator who created the teacher data used to create the trained model. As described above, a large amount of teacher data is required to create a model with estimation accuracy that is practical. Therefore, teacher data is usually created by multiple teacher data workers. The teacher data creator information is information that identifies each of the multiple teacher data creators who created the teacher data.

[0085] The call-specific inference result evaluation unit 13B, like the call-specific inference result evaluation unit 13, evaluates the teacher data and the inference result of the label inference unit 12 for each call, and outputs the evaluation result (call-specific evaluation result) to the call-specific confirmation screen generation unit 14B and the external output interface 1. Here, the call-specific inference result evaluation unit 13B generates a call-specific evaluation result for each teacher data creator based on the teacher data creator information. That is, the call-specific inference result evaluation unit 13B constituting the evaluation unit 17B generates an evaluation result for each element group by comparing the correct label and the inference label of the element constituting the element group for each teacher data creator. Although details will be described later, the call-specific inference result evaluation unit 13B may switchably present the call-specific evaluation result generated for each teacher data creator.

[0086] The call-specific confirmation screen generating unit 14B, like the call-specific confirmation screen generating unit 14, generates a teacher data confirmation screen (call-specific confirmation screen) for each call based on the call-specific evaluation result output from the call-specific inference result evaluating unit 13B, and outputs it to the external output interface 1. Here, the call-specific confirmation screen generating unit 14B generates a call-specific confirmation screen for each teacher data creator based on the teacher data creator information. That is, the call-specific confirmation screen generating unit 14B constituting the evaluating unit 17B generates a teacher data confirmation screen for each element group, including the elements constituting the element group, the correct answer labels of the elements, and the inference labels of the elements, for each teacher data creator. Although details will be described later, the call-specific confirmation screen generating unit 14B may present teacher data confirmation screens generated for the same teacher data creator in a switchable manner.

[0087] Like the utterance-specific inference result evaluation unit 15, the utterance-specific inference result evaluation unit 15B evaluates the teacher data and the inference result of the label inference unit 12 for each utterance, and outputs the evaluation result (utterance-specific evaluation result) to the utterance-specific confirmation screen generation unit 16B and the external output interface 1. That is, the utterance-specific inference result evaluation unit 15B constituting the evaluation unit 17B generates evaluation results for each teacher data creator based on a comparison between the correct label and the inference label for each element constituting the teacher data.

[0088] Like the utterance-specific confirmation screen generating unit 16, the utterance-specific confirmation screen generating unit 16B generates a teacher data confirmation screen (utterance-specific confirmation screen) for each utterance based on the utterance-specific evaluation result output from the utterance-specific inference result evaluating unit 15B, and outputs it to the external output interface 1. Here, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen for each teacher data creator based on the teacher data creator information. That is, the utterance-specific confirmation screen generating unit 16B constituting the evaluating unit 17B generates a teacher data confirmation screen including elements constituting the teacher data, the correct answer labels of the elements, and the inference labels of the elements, for each teacher data creator. Although details will be described later, the utterance-specific confirmation screen generating unit 16B may generate an utterance-specific confirmation screen (a screen on which the evaluation result for each element group can be confirmed) so that it can be switched for each teacher data creator.

[0089] The teacher data creator evaluation unit 21 receives the teacher data, the inference result by the label inference unit 12, and the teacher data creator information. The teacher data creator evaluation unit 21 generates an evaluation result of the teacher data creator (hereinafter referred to as the "teacher data creator evaluation result") based on a comparison between the correct labels of the elements constituting the teacher data and the inferred labels of the elements, and outputs the evaluation result to the external output interface 1.

[0090] In this embodiment, the evaluation result of the teacher data creator is generated based on a comparison between the correct label given to the element constituting the teacher data and the inferred label of the element, so that the evaluation of the teacher data creator can be performed more efficiently. In addition, it is possible to perform a detailed analysis of the tendency of errors when creating teacher data for each teacher data creator, and it is possible to efficiently educate the teacher data creator on the policy for creating teacher data.

[0091] Next, the operation of the support device 10B according to this embodiment will be described.

[0092] Fig. 12 is a flowchart showing an example of the operation of the support device 10B, and is a diagram for explaining a support method by the support device 10B according to the present embodiment. In Fig. 12, the same processes as those in Fig. 3 are given the same reference numerals, and the explanation thereof will be omitted.

[0093] When the label inference unit 12 infers the inferred labels of the elements that make up the teacher data (step S12), the teacher data creator evaluation unit 21 generates a teacher data creator evaluation result based on a comparison between the correct label of the element that makes up the teacher data and the inferred label of that element, and outputs it to the external output interface 1 (step S31).

[0094] FIG. 13 is a diagram showing an example of the teacher data creator evaluation result.

[0095] As shown in FIG. 13, the teacher data creator evaluation unit 21 outputs, as a teacher data creator evaluation result, a teacher data creator index, which is identification information for identifying a teacher data creator, and an evaluation value of teacher data created by the teacher data creator in association with each other. The evaluation value of teacher data is, for example, an average value of values ​​such as the precision rate, recall rate, F value, or matching rate of an inferred label for a correct answer label of a plurality of teacher data created by the teacher data creator. That is, the teacher data creator evaluation unit 21 generates an evaluation result based on a comparison between a correct answer label corresponding to an element constituting the element group and an inferred label for each element group so that the evaluation result can be confirmed for each teacher data creator. It is considered that a teacher data creator who has created a teacher data with a high evaluation value is likely to have assigned an appropriate label. On the other hand, it is considered that a teacher data creator who has created a teacher data with a low evaluation value is likely to have not been able to assign an appropriate label and to require training such as learning a policy for assigning a label. The teacher data creator evaluation unit 21 outputs the teacher data index and the evaluation value in order of, for example, the worst evaluation value. This makes it easy to identify training data creators who are likely to create low-quality training data and require training such as learning labeling policies.

[0096] Referring again to FIG. 12, the call-specific inference result evaluation unit 13B evaluates the correct label of the teacher data and the inference result of the label inference unit 12 for each call, and outputs the call-specific evaluation result (step S32).

[0097] FIG. 14 is a diagram showing an example of the call-specific inference result output by the call-specific inference result evaluation unit 13B.

[0098] 14, the call-specific inference result evaluation unit 13B, like the call-specific inference result evaluation unit 13, outputs a call index and an evaluation value such as a match rate for that call in association with each other as a call-specific evaluation result. Also, like the call-specific inference result evaluation unit 13, the call-specific inference result evaluation unit 13B may list the call index and the evaluation value in order of worse evaluation result, and output them as text data, for example. The call-specific evaluation result may include the start time and end time of the call.

[0099] The call-specific inference result evaluation unit 13B generates call-specific evaluation results for each teacher data creator, as shown in Fig. 14. The call-specific inference result evaluation unit 13 may present the call-specific evaluation results for each teacher data creator in a switchable manner. By generating call-specific evaluation results for each teacher data creator, it is possible to easily grasp the labeling tendency for each teacher data creator.

[0100] Referring back to FIG. 12, the call-specific confirmation screen generating unit 14B generates a call-specific confirmation screen based on the call-specific evaluation result (step S33), and outputs it to the external output interface 1.

[0101] FIG. 15 is a diagram showing an example of the call-specific confirmation screen.

[0102] As shown in FIG. 15, the call-specific confirmation screen generating unit 14B generates a call-specific confirmation screen for each call, similar to the call-specific confirmation screen generating unit 14, including the start time of the utterance constituting the call, the end time of the utterance, the utterance text, and the correct label and inference label of the utterance text. Here, the call-specific confirmation screen generating unit 14B generates a call-specific confirmation screen for each teacher data creator. In order to indicate for which teacher data creator the call-specific confirmation screen has been generated, the call-specific confirmation screen generating unit 14B includes a teacher data creator index in the call-specific confirmation screen, as shown in FIG. 15. The call-specific confirmation screen generating unit 14B may superimpose and present call-specific confirmation screens generated for the same teacher data creator in a switchable manner, as shown in FIG. 15. That is, the call-specific confirmation screen generating unit 14B may generate a teacher data confirmation screen (call-specific confirmation screen) for each teacher data creator, which includes elements constituting the element group, correct labels corresponding to the elements, and inference labels for the elements, and is switchable for each element group. In this case, the call-specific confirmation screen generating unit 14B may display the call-specific confirmation screen in the foreground for a call with a worse evaluation result.

[0103] Referring again to FIG. 12, the utterance-specific inference result evaluation unit 15B evaluates the teacher data and the inference result of the label inference unit 12 for each utterance, and outputs the evaluation result (utterance-specific evaluation result) (step S34).

[0104] FIG. 16 is a diagram showing an example of the evaluation result for each utterance.

[0105] As shown in FIG. 16, the utterance-specific inference result evaluation unit 15B outputs, as the utterance-specific evaluation result, for example, a result showing the occurrence frequency of difference patterns in a confusion matrix, and an evaluation value for each label (precision, recall, F-score (f1-score), and occurrence frequency (support)) in the form of text data. Here, the utterance-specific inference result evaluation unit 15B outputs the utterance-specific evaluation result for each teacher data creator. In order to indicate for which teacher data creator the utterance-specific evaluation result has been output, the utterance-specific inference result evaluation unit 15B includes a teacher data creator index in the utterance-specific evaluation result, as shown in FIG. 16. By outputting the utterance-specific evaluation result for each teacher data creator, it is possible to confirm difference patterns for which the teacher data creator is likely to mislabel. In addition, it becomes easier for the teacher data creator or its administrator to grasp errors in the policy of labeling. As described above, the evaluation result for each utterance includes the evaluation result of the teacher data created by the teacher data creator, such as the frequency of occurrence of difference patterns and the evaluation value for each label. Therefore, the evaluation result for each utterance may be output as the teacher data creator evaluation result.

[0106] The utterance-specific inference result evaluation unit 15B may display difference patterns that are likely to cause confusion in a ranking format as shown in FIG. 17, instead of the evaluation value for each label shown in FIG. 16. A difference pattern that is likely to cause confusion is a pattern in which the correct label and the inference label are different, and confusion or replacement is likely to occur. The number of difference patterns that are likely to cause confusion is, for example, the total number of utterances in which the correct label is A and the inference label is B, and the number of utterances in which the correct label is B and the inference label is A. Furthermore, the utterance-specific inference result evaluation unit 15B may include difference patterns that are likely to cause confusion in the utterance-specific evaluation result. In this way, the teacher data creator can grasp difference patterns that are likely to cause mistakes (labels that are difficult to assign). In addition, the administrator of the teacher data creator can notice a misrecognition of the label assignment policy for each teacher data creator.

[0107] Referring again to FIG. 12, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen based on the utterance-specific evaluation result (step S35), and outputs it to the external output interface 1.

[0108] FIG. 18 is a diagram showing an example of the utterance-specific confirmation screen.

[0109] 18, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen in which an utterance text, a line number indicating the order of the utterance text in a call including the utterance, and a correct answer label and an inference label of the utterance text are associated with each other, similar to the utterance-specific confirmation screen generating unit 16. Here, the utterance-specific confirmation screen generating unit 16B generates an utterance-specific confirmation screen for each teacher data creator. That is, the utterance-specific confirmation screen generating unit 16B generates a teacher data confirmation screen (utterance-specific confirmation screen) including, for each element, an element, a correct answer label corresponding to the element, and an inference label of the element so that the element can be confirmed for each teacher data creator.

[0110] Note that, like the utterance-specific confirmation screen generating unit 16, the utterance-specific confirmation screen generating unit 16B may generate and present utterance-specific confirmation screens in the order of utterance texts including difference patterns with a high frequency of occurrence. That is, the utterance-specific confirmation screen generating unit 16B may present utterance-specific confirmation screens in the order of elements including difference patterns with a high frequency of occurrence among difference patterns in which the correct answer label assigned to the teacher data is different from the inference label by the trained model. Furthermore, the utterance-specific confirmation screen generating unit 16B may present multiple utterance-specific confirmation screens generated for the same teacher data creator in a switchable manner.

[0111] Thus, the support device 10B according to this embodiment includes a label inference unit 12 and an evaluation unit 17B. The label inference unit 12 infers inferred labels, which are labels corresponding to elements constituting the teacher data, using a model for inferring labels corresponding to elements, which is learned using the teacher data. The evaluation unit 17 generates an evaluation result of the teacher data creator based on a comparison between the correct labels of the elements constituting the teacher data and the inferred labels of the elements.

[0112] The support method according to the present embodiment includes an inference step and an evaluation result generating step. In the inference step, an inference label, which is a label corresponding to an element constituting the training data, is inferred using a model that is trained using the training data and infers a label corresponding to an element. In the evaluation result generating step, an evaluation result of the training data creator is generated based on a comparison between the correct label of the element constituting the training data and the inference label of the element.

[0113] By generating an evaluation result for the teacher data creator based on a comparison between the correct answer label of the element that constitutes the teacher data and the inferred label of that element, the teacher data creator can be evaluated more efficiently. In addition, it is possible to perform a detailed analysis of the tendency of errors made when creating teacher data for each teacher data creator, and it is possible to efficiently educate the teacher data creator on the creation policy.

[0114] A computer can be suitably used to function as each part of the above-mentioned support devices 10, 10A, and 10B. Such a computer can be realized by storing a program describing the processing contents for realizing the functions of each part of the support devices 10, 10A, and 10B in a storage unit of the computer, and reading and executing the program by a CPU (Central Processing Unit) of the computer. In other words, the program can make the computer function as the above-mentioned support devices 10, 10A, and 10B.

[0115] The following supplementary notes are further disclosed regarding the above embodiment.

[0116] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, Inferring an inferred label, which is a label corresponding to an element constituting the training data, using a model for inferring a label corresponding to the element, the model being trained using training data consisting of a pair of an element and a correct label corresponding to the element; An assistance device that generates an evaluation result for the creator of the training data based on a comparison between a correct answer label corresponding to an element constituting the training data and an inferred label of the element.

[0117] (Additional note 2) A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to function as the support device described in appended claim 1.

[0118] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0119] 10,10A,10B Support device 11 Model Learning Section 12 Label inference part 13, 13B Call-specific inference result evaluation unit 14,14B Confirmation screen generation section for each call 15, 15B Utterance-specific inference result evaluation section 16,16B Confirmation screen generation section for each utterance 17 Evaluation Section 18. Inference Error Elimination Section 21 Teacher Data Creator Evaluation Department 110 Processor 120 ROM 130 RAM 140 Storage 150 Input section 160 Display section 170 Communication Interface 190 Bus

Claims

1. A support device for supporting an evaluation of a teacher data creator who creates teacher data consisting of a pair of an element and a correct label corresponding to the element, comprising: an evaluation unit that presents, for each of the elements, a type of difference pattern in which the correct label corresponding to the element and the inferred label corresponding to the element are different, and which indicates confusion or replacement of the correct label and the inferred label; The inferred label is a label corresponding to the element, which is inferred using an inference model trained using the training data.

2. 2. The support device according to claim 1, The evaluation unit is A teacher data confirmation screen including a correct answer label corresponding to each element constituting the teacher data and an inference label of the element is generated so as to be able to be confirmed by each teacher data creator; The support device generates the teacher data confirmation screen so that elements including the difference patterns that appear most frequently among the difference patterns can be confirmed in order.

3. 2. The support device according to claim 1, The evaluation unit presents the difference patterns in a ranking format in order of the frequency of occurrence of each type of difference pattern.

4. 2. The support device according to claim 1, The support device, wherein the evaluation unit presents the type of difference pattern in a switchable manner for each teacher data creator.

5. A method for supporting an evaluation of a training data creator who creates training data consisting of a pair of an element and a correct label corresponding to the element, comprising: presenting, for each of the elements, a type of difference pattern in which the correct label corresponding to the element is different from the inferred label corresponding to the element, and which indicates confusion or replacement of the correct label and the inferred label; An assistance method, wherein the inferred label is a label inferred using an inference model trained using the training data as a label corresponding to the element.

6. A program for causing a computer to function as the support device according to any one of claims 1 to 4.

Images