Presentation device, presentation method, and program

Abstract

To provide a presentation device, a presentation method, and a program capable of evaluating the quality of dialogue contents based on scores according to a certain standard.SOLUTION: According to an embodiment, the presentation device includes a dialogue acquisition unit and a response evaluation unit. The dialogue acquisition unit acquires dialogue data representing the contents of a dialogue between an evaluated person and an interlocutor. The response evaluation unit outputs a response evaluation instruction to have a natural language processing device evaluate the responses of the evaluated person based on the dialogue data. The response evaluation unit acquires evaluation results according to the response evaluation instruction from the natural language processing device. The response evaluation instruction includes an instruction text for scoring each of multiple evaluation items related to the responses of the evaluated person and a standard presentation text that presents multiple evaluation criteria for each of the multiple evaluation items.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a presentation device, a presentation method, and a program. [Background technology]

[0002] At call centers, there is a desire to evaluate the content of how operators respond to customers. However, evaluating the content of responses takes time and is necessarily a qualitative judgment, making it difficult for evaluators to evaluate all of the recording data for each response. It is also difficult to standardize evaluation criteria among evaluators, and it is not uncommon for evaluation results to differ depending on the evaluator for the same response. Patent Document 1 discloses a technology for evaluating the responses of call center operators according to quantitative criteria such as the number of calls and call duration. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-095758 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a presentation device, a presentation method, and a program that can evaluate the quality of dialogue content by a score according to a certain standard. [Means for solving the problem]

[0005] According to an embodiment of the present invention, the presentation device has a dialogue acquisition unit and a response evaluation unit. The dialogue acquisition unit acquires dialogue data representing the content of the dialogue between the person being evaluated and the interlocutor. The response evaluation unit outputs a response evaluation instruction to cause the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data. The response evaluation unit acquires a response evaluation result in accordance with the response evaluation instruction from the natural language processing device. The response evaluation instruction includes an instruction statement to score each of multiple evaluation items related to the response of the person being evaluated, and a criterion presentation statement that presents multiple evaluation criteria for each of the multiple evaluation items. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a schematic block diagram showing the configuration of an evaluation system according to a first embodiment. [Figure 2] FIG. 2 is a schematic block diagram showing the configuration of a management device according to the first embodiment. [Figure 3] 10 is an example of a template of a response evaluation instruction prompt according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of an evaluation screen for an operator according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of an evaluation screen for a manager according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of an evaluation screen for an individual response according to the first embodiment. [Figure 7] 4 is a flowchart showing an evaluation method performed by the management device according to the first embodiment. [Figure 8] FIG. 10 is a schematic block diagram showing the configuration of a management device according to a second embodiment. [Figure 9] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] A presentation device, a presentation method, and a program according to an embodiment will be described below with reference to the accompanying drawings. (First embodiment) 1 is a schematic block diagram showing the configuration of an evaluation system 1 according to a first embodiment. The evaluation system 1 according to the first embodiment evaluates calls made by operators at a call center. The evaluation system 1 includes a plurality of operator terminals 10, an administrator terminal 20, a management device 30, and a natural language processing device 50.

[0008] The operator terminal 10 is a terminal operated by an operator at a call center. The operator terminal 10 is, for example, a telephone, and enables a call between a customer at the call center and an operator. The operator terminal 10 is provided with an input / output device 11 having a microphone and a speaker. Examples of the input / output device 11 include a headset and a handset. The manager terminal 20 is a terminal operated by a manager who manages operators.

[0009] The management device 30 controls a plurality of operator terminals 10. For example, the management device 30 may be a CTI (Computer Telephony Integration) system. The operator terminals 10 and the management device 30 are connected to each other, for example, by a LAN (Local Area Network). The management device 30 records calls made by each operator terminal 10 and generates call voice data for each operator. Based on the call voice data, the management device 30 transmits a prompt to the natural language processing device 50 to cause the operator to evaluate the call content. The management device 30 receives the evaluation result from the natural language processing device 50 and generates information representing the operator's evaluation. The management device 30 is an example of a presentation device that presents the call evaluation result to an operator or a manager. The management device 30 communicates with the natural language processing device 50 via the Internet or the like.

[0010] The natural language processing device 50 receives input of natural language sentences called prompts via the Internet or the like, and generates natural language sentences in response to the input prompts. Note that the natural language processing device 50 may input and output not only sentences written in natural languages, but also sentences written in artificial languages. The natural language processing device 50 according to the first embodiment has a text-to-text large-scale language model, inputs prompts to the large-scale language model, and outputs sentences generated by the large-scale language model in response to the prompts.

[0011] 2 is a schematic block diagram showing the configuration of a management device 30 according to the first embodiment. The management device 30 includes a dialogue acquisition unit 31, a text conversion unit 32, a response evaluation unit 33, an extraction unit 34, a personal evaluation unit 35, an output unit 36, and a storage unit 37.

[0012] The dialogue acquisition unit 31 acquires, from the operator terminal 10, voice data representing the content of the dialogue between a customer and an operator. The voice data is stereo voice data in which the voice of the customer and the voice of the operator are recorded separately on two channels. In other words, the operator terminal 10 records the voice input to the microphone of the input / output device 11 and the voice received from the outside via voice communication separately on two channels. Which of the voice of the customer and the voice of the operator is to be recorded on the left and which on the right may be set in advance in each operator terminal 10 by the management device 30. The dialogue acquisition unit 31 can identify which operator's dialogue the acquired voice data represents, based on the login information of the operator terminal 10 from which the voice data was acquired.

[0013] The text conversion unit 32 converts the voice data acquired by the dialogue acquisition unit 31 into text data. In other words, the text conversion unit 32 generates text data by performing voice recognition processing on the voice data. Hereinafter, the text data representing the dialogue content generated by the text conversion unit 32 will be referred to as dialogue text data. At this time, the text conversion unit 32 generates dialogue text data in which a label indicating the speaker and the time of utterance are attached to each utterance included in the voice data. For example, the format of the dialogue text data generated by the text conversion unit 32 may be as shown below. Note that in the following sentences, "{}" represents a variable.

[0014] "{Start time of speech} - {End time of speech}, {Speaker}: {Content of speech}" In other embodiments, the dialogue text data may not include information on the speech start time and speech end time. On the other hand, if the dialogue text data includes time information, the natural language processing device 50 can evaluate the speech speed and timing. The dialogue text data may be associated with non-verbal information such as speaking rate, speech interval from the previous utterance, various statistics (average, variance, etc.) of acoustic features (fundamental frequency, volume, etc.) obtained by acoustic analysis of the voice data, and emotional values ​​obtained by voice emotion analysis. By using this non-verbal information obtained from the voice, the natural language processing device 50 can improve the evaluation accuracy of evaluation items that are difficult to infer from text data alone, such as the impression of the conversation and attitude.

[0015] For example, the text conversion unit 32 may generate dialogue text data in the following procedure. The text conversion unit 32 performs speech recognition processing on each channel of the voice data. The text conversion unit 32 generates text data of each utterance by an operator according to the above format from text data of the utterance content obtained from a channel assigned to the operator's voice, the label of the operator who is the speaker, and the utterance start time and end time. The text conversion unit 32 generates text data of each utterance by a customer according to the above format from text data of the utterance content obtained from a channel assigned to the customer's voice, the label of the customer who is the speaker, and the utterance start time and end time. The text conversion unit 32 generates dialogue text data by chronologically arranging the text data of each utterance. The text conversion unit 32 is an example of a speaker separation unit that generates dialogue sentence data from dialogue data, associating character strings representing the utterance content with the speaker of the utterance content.

[0016] The response evaluation unit 33 generates a response evaluation instruction prompt based on the dialogue text data generated by the text generation unit 32, and transmits the response evaluation instruction prompt to the natural language processing device 50. The response evaluation instruction prompt includes the following information: - Instructions to score each of multiple evaluation items related to the operator's response -Explanation of each of the multiple evaluation items -Explanation of multiple evaluation criteria for each of the multiple evaluation items -Examples of dialogues related to multiple evaluation criteria Evaluation result format

[0017] For example, the response evaluation unit 33 may store a response evaluation instruction prompt template in advance and generate the response evaluation instruction prompt by embedding dialogue text data in variables included in the template. Fig. 3 shows an example of a response evaluation instruction prompt template according to the first embodiment. As shown in Fig. 3, the template of the response evaluation instruction prompt may be, for example, as follows: Note that in Fig. 3 and the following text, "{}" represents a variable.

[0018] “#Instructions You are a consultant who helps improve the quality of call center responses. Please read the following dialogue and rate the quality of the operator's response for each evaluation item on a scale of 1 to 10. Please also explain in detail the reasons for your rating using the dialogue. #Evaluation items 1. Emotionally sensitive service 1A. Demonstrates specific understanding of customer sentiment Good example: We understand that you must be worried about missing our products, which you use every day. 1B.…… example:…… 2.…… 3.…… #format <Evaluation items> Rating: <Rating value from 1 to 10> Reason: <reason> #Dialogue {Dialogue text data}" The evaluation items or evaluation criteria are not limited to evaluation of the operator's response itself, but may also be evaluations of the customer's impression or reaction to the operator's response, such as "customer satisfaction with the operator's proposal," "customer's reliability regarding the product / service inquired about," etc. When one evaluation item contains a mixture of evaluation criteria related to the evaluation of the operator's response and evaluation criteria related to the customer's reaction to the operator's response, the natural language processing device 50 calculates the evaluation value of the evaluation item taking both into consideration, thereby making it possible to make an evaluation based on both the way the operator responded and how the customer felt about the response.

[0019] The response evaluation unit 33 acquires a sentence generated in response to the response evaluation instruction prompt by the natural language processing device 50. The sentence acquired by the response evaluation unit 33 is a sentence that conforms to the format included in the response evaluation instruction prompt.

[0020] The extraction unit 34 extracts the evaluation value and the reason for each evaluation item from the sentence acquired by the response evaluation unit 33 according to the format specified in the response evaluation instruction prompt. For each interaction, the extraction unit 34 associates the date and time of the interaction, the ID of the corresponding agent, and the evaluation value and reason for each evaluation item, and records the resulting data as interaction evaluation data in the storage unit 37. By extracting the evaluation value from the response evaluation result data, the extraction unit 34 can treat the evaluation value included in the response evaluation result data as a numerical value.

[0021] The individual evaluation unit 35 generates an individual evaluation instruction prompt for causing an evaluation of the agent to be evaluated from a plurality of pieces of dialogue evaluation data associated with the agent to be evaluated, and transmits the individual evaluation instruction prompt to the natural language processing device 50. The individual evaluation instruction prompt includes the following information: - Instructions to generate advice for operators based on response evaluation result data Evaluation result format

[0022] For example, the individual evaluation unit 35 may store a template of an individual evaluation instruction prompt in advance and generate the individual evaluation instruction prompt by embedding dialogue text data in variables included in the template. The template of the individual evaluation instruction prompt may be, for example, as follows:

[0023] “#Instructions You are a consultant who helps improve the quality of call center responses. Read the evaluation of the response of the agent you are evaluating and give advice to this agent to improve the quality of their response, including specific evaluation points for each response process and the response as a whole. #Response process 1. Listen to your customers 2. Organize your customer's story 3. Present the big picture 4. Present a specific response policy 3. Make an appointment to meet again #format <Response process> Good points: <Advice> Improvements: <Advice> #Service evaluation {Response evaluation result data} {Response evaluation result data} ……"

[0024] The individual evaluation instruction prompt is accompanied by a plurality of pieces of response evaluation result data relating to the evaluation period. The natural language processing device 50 generally has a limit on the number of characters in a prompt that it can accept, and may not be able to process long prompts. Therefore, the individual evaluation unit 35 can reduce the number of characters in the prompt by attaching response evaluation result data with a relatively small number of characters rather than attaching the entire dialogue text data relating to the evaluation period to the prompt. In particular, the response evaluation result data has a specified format as shown in Figure 3, so it can be said that there is little variation in the number of characters. In another embodiment, instead of the response evaluation result data itself, the individual evaluation instruction prompt may include, for example, data that has been previously summarized using the natural language processing device 50. This allows the individual evaluation instruction prompt to be compressed into shorter text that includes information important for evaluation and omits unnecessary information. Furthermore, in another embodiment, instead of the response evaluation result data for each response, the individual evaluation instruction prompt may include data that has been previously summarized using the natural language processing device 50 from multiple response result data. Furthermore, in another embodiment, instead of the response evaluation result data, the individual evaluation instruction prompt may include statistics of multiple response evaluation result data. By using such summaries, the management device 30 can further improve the accuracy of the response while reducing the number of characters in the prompt.

[0025] The personal evaluation unit 35 acquires the sentence generated by the natural language processing device 50 in response to the personal evaluation instruction prompt. The extraction unit 34 extracts advice from the sentence acquired by the individual evaluation unit 35 in accordance with the format specified in the individual evaluation instruction prompt. The extraction unit 34 associates the evaluation period, the operator ID, and the advice for each operator, and records them in the storage unit 37 as individual evaluation result data.

[0026] The output unit 36 ​​generates an evaluation screen based on the response evaluation result data acquired by the response evaluation unit 33 and the personal evaluation result data acquired by the personal evaluation unit 35, and outputs it to the operator terminal 10 or the manager terminal 20.

[0027] 4 is a diagram showing an example of an evaluation screen for operators according to the first embodiment. The evaluation screen for operators displays a graph showing scores for each evaluation item for the responses of the operator to be evaluated during the evaluation period, advice for the responses during the evaluation period, evaluation points for each response process, and a list of responses during the evaluation period. The output unit 36 ​​can generate a graph showing scores for the responses during the evaluation period by calculating the average score for each evaluation item from multiple response evaluation result data for the agent to be evaluated during the evaluation period.

[0028] 5 is a diagram showing an example of an evaluation screen for a manager according to the first embodiment. The evaluation screen for a manager displays a graph showing the scores for each evaluation item for the entire team managed by the manager during an evaluation period, trends for the entire team, advice for the agents belonging to the team, changes in the scores of the agents belonging to the team, and a list of responses during the evaluation period.

[0029] The list of responses during the evaluation period shown in Figure 4 or 5 displays the evaluation value, date and time, summary, and details button for each response. The list of responses may also include the operator's name. When the details button is pressed, the output unit 36 ​​outputs an evaluation screen showing response evaluation result data to the operator terminal 10 or the administrator terminal 20. 6 is a diagram showing an example of an evaluation screen for an individual call according to the first embodiment. The evaluation screen for a call displays a graph showing the score for each evaluation item for the selected call, an overall evaluation and advice for the call, and the score for each evaluation item and the reason for the score. In Figure 4-6, four examples of evaluation items are given: communication skills, efficient behavior, problem-solving ability, and gaining trust. However, these are merely examples and are not limited to these.

[0030] FIG. 7 is a flowchart showing an evaluation method performed by the management device 30 according to the first embodiment. The dialogue acquisition unit 31 of the management device 30 acquires voice data from the operator terminal 10 for each dialogue between a customer and an operator. The text conversion unit 32 generates dialogue text data consisting of a pair of a speaker and the content of the utterance from the voice data acquired by the dialogue acquisition unit 31, and records the data in the storage unit 37 in association with the operator ID and the date and time of the response.

[0031] The response evaluation unit 33 of the management device 30 extracts the response dates and times that belong to the evaluation period from the dialogue text data recorded in the storage unit 37 (step S1). The response evaluation unit 33 selects the extracted dialogue text data one by one (step S2) and executes the following processes from step S3 to step S7.

[0032] The response evaluation unit 33 generates a response evaluation instruction prompt for the selected dialogue text data according to a predetermined template (step S3). The response evaluation unit 33 transmits the generated response evaluation instruction prompt to the natural language processing device 50 (step S4). The response evaluation unit 33 receives a sentence generated based on the response evaluation instruction prompt from the natural language processing device 50 (step S5). The extraction unit 34 extracts an evaluation value and a reason for each evaluation item from the received sentence (step S6). The extraction unit 34 associates the extracted evaluation value and reason for each evaluation item with the agent ID and response date and time related to the dialogue text data selected in step S2, and records them in the storage unit 37 as response evaluation result data (step S7).

[0033] When the management device 30 executes the processes of steps S3 to S7 for all the dialogue text data extracted in step S1, the individual evaluation unit 35 selects an operator one by one (step S8) and executes the processes of steps S9 to S14 below.

[0034] The individual evaluation unit 35 reads all response evaluation result data within the evaluation period associated with the agent selected in step S8 from the storage unit 37 (step S9). The individual evaluation unit 35 generates an individual evaluation instruction prompt including the response evaluation result data read in step S9 according to a predetermined template (step S10). The individual evaluation unit 35 transmits the generated individual evaluation instruction prompt to the natural language processing device 50 (step S11). The individual evaluation unit 35 receives text generated based on the individual evaluation instruction prompt from the natural language processing device 50 (step S12). The extraction unit 34 extracts advice for the entire response and for each response process from the received text (step S13). The extraction unit 34 associates the information extracted in step S13 with the ID of the agent selected in step S8 and records the information extracted in step S13 as individual evaluation result data in the storage unit 37 (step S14).

[0035] The output unit 36 ​​receives a request to view the evaluation results from the operator terminal 10 or the administrator terminal 20 (step S15). The output unit 36 ​​reads out, from the storage unit 37, the individual evaluation result data associated with the operator related to the viewing request or the operator related to the team under the management of the administrator (step S16). The output unit 36 ​​generates an evaluation screen based on the read individual evaluation result data, and transmits the display data of the evaluation screen to the operator terminal 10 or the administrator terminal 20 that sent the viewing request (step S17).

[0036] As described above, the evaluation system 1 according to the first embodiment functions as follows: The dialogue acquisition unit 31 acquires dialogue data representing the content of the dialogue between an agent and a customer. The response evaluation unit 33 outputs a response evaluation instruction prompt to cause the natural language processing device 50 to evaluate the agent's response based on the dialogue data, and acquires the response evaluation result from the natural language processing device 50. The response evaluation instruction prompt includes an instruction statement for scoring each of a plurality of evaluation items related to the agent's response, and a criterion presentation statement presenting a plurality of evaluation criteria for each of the plurality of evaluation items. The evaluation system 1 presents multiple evaluation criteria for each evaluation item, which allows the natural language processing device 50 to reduce fluctuations in evaluation for the evaluation items and perform scoring according to certain criteria.

[0037] The response evaluation prompt according to the first embodiment includes example sentences that illustrate, for each of a plurality of evaluation criteria, dialogues that satisfy the evaluation criteria or dialogues that do not satisfy the evaluation criteria. This allows the natural language processing device 50 to create sentences to be evaluated in accordance with the examples, thereby preventing variations in evaluation.

[0038] The response evaluation instruction prompt according to the first embodiment includes a format suggestion sentence that suggests the format of the evaluation results. The extraction unit 34 of the evaluation system 1 extracts the scoring results of multiple evaluation items from the evaluation results according to the format. The natural language processing device 50 can generate free sentences, but by specifying a format, it can extract the scoring results as numerical values ​​that can be used for calculating average values, etc.

[0039] (Second embodiment) The evaluation system 1 according to the first embodiment evaluates each dialogue in accordance with the response evaluation instruction prompt shown in FIG. 3. However, since the difficulty of the response varies depending on the dialogue, it may not always be appropriate to perform a uniform evaluation. For example, if the customer's purpose is clear and the customer is calm, the difficulty of the operator's response may be low, whereas if the customer's purpose is unclear or the customer is very angry, the difficulty of the operator's response may be high. The evaluation system 1 according to the second embodiment realizes evaluation according to the difficulty of the response.

[0040] 8 is a schematic block diagram showing the configuration of a management device 30 according to the second embodiment. The management device 30 according to the second embodiment further includes a difficulty evaluation unit 38 in addition to the configuration of the first embodiment.

[0041] The difficulty evaluation unit 38 generates a difficulty evaluation instruction prompt based on the dialogue text data, and transmits the difficulty evaluation instruction prompt to the natural language processing device 50. The difficulty evaluation instruction prompt includes the following information. - Instructions to rate the difficulty of the operator's response in a dialogue as a numerical value -Explanation of the difficulty evaluation criteria Evaluation result format

[0042] For example, the difficulty evaluation unit 38 may store a template of a difficulty evaluation instruction prompt in advance and generate the difficulty evaluation instruction prompt by embedding dialogue text data in variables included in the template. The template of the difficulty evaluation instruction prompt may be, for example, as follows:

[0043] “#Instructions Read the following dialogue and rate the difficulty of the operator's response on a scale of 1 to 3. #Evaluation criteria 1. If the customer's response to the operator's utterance is not logical, the difficulty level is high. example:…… 2. If the customer speaks strongly, it is more difficult. example:…… 3.…… #format Difficulty: <Difficulty of 1 to 3 points> <Reason> #Dialogue {Dialogue text data}"

[0044] The difficulty evaluation unit 38 acquires a sentence generated in response to the difficulty evaluation instruction prompt by the natural language processing device 50. The sentence acquired by the difficulty evaluation unit 38 is a sentence that conforms to the format included in the difficulty evaluation instruction prompt.

[0045] The response evaluation unit 33 may generate a response evaluation prompt according to a template corresponding to the evaluation by the difficulty evaluation unit 38. For example, the response evaluation unit 33 may prepare a template for difficulty level 1, a template for difficulty level 2, and a template for difficulty level 3, and generate a response evaluation prompt according to the template corresponding to the difficulty level identified by the difficulty evaluation unit 38. Each template has different sentences that present multiple evaluation criteria for at least multiple evaluation items. This allows the response evaluation unit 33 to perform evaluation according to the difficulty level.

[0046] Furthermore, the individual evaluation unit 35 may calculate the average evaluation score by adjusting the evaluation score for each response according to the level of difficulty. For example, the individual evaluation unit 35 may apply a large weight to a response with a high level of difficulty when a high evaluation score is assigned to the response, and a small weight to a response with a low level of difficulty when a low evaluation score is assigned to the response. This allows the individual evaluation unit 35 to highly evaluate a response with a high level of difficulty for being properly performed, while preventing an inability to handle a high level of difficulty from immediately giving a low evaluation. For example, the individual evaluation unit 35 may apply a small weight to a response with a low level of difficulty when a high evaluation score is assigned to the response, and a large weight to a response with a low level of difficulty when a low evaluation score is assigned to the response. This allows the individual evaluation unit 35 to prevent an immediate high evaluation for a response with a low level of difficulty that was properly performed, and to give a low evaluation for an inappropriate response despite the low level of difficulty.

[0047] In addition, when creating an evaluation screen for managers, the output unit 36 ​​extracts evaluations of responses judged to be difficult for operators within the team and presents them in a comparable manner, thereby allowing managers to evaluate the response capabilities of each operator.

[0048] (Third embodiment) The management device 30 according to the above-described embodiment evaluates responses according to a prepared template. However, in order to evaluate whether a response conforms to a business manual, for example, the contents of the business manual must be presented to the natural language processing device 50, and the prompt may exceed the number of characters that can be processed by the natural language processing device 50. Therefore, the management device 30 according to the third embodiment generates a response evaluation instruction prompt that causes an evaluation to be performed based on documents such as business manuals narrowed down by RAG (Retrieval-Augmented Generation).

[0049] The response evaluation unit 33 according to the third embodiment can cause the natural language processing device 50 to score whether the response is in accordance with the business manual, for example, by the following procedure. The response evaluation unit 33 transmits a prompt to the natural language processing device 50 for each piece of business manual data prepared in advance, causing the natural language processing device 50 to evaluate whether the dialogue is in accordance with the business manual. The template of the prompt may be, for example, as follows:

[0050] “#Instructions Please determine whether the following dialogue is in accordance with the business manual. If the dialogue is in accordance with the business manual, return 1 as the evaluation value. If the dialogue is not in accordance with the business manual, return -1. If the dialogue cannot be evaluated in the business manual, return 0. #format <Evaluation value from -1 to 1> <Reason> #Dialogue {Dialogue text data} #Business Manual {Operations manual data}"

[0051] The response evaluation unit 33 performs scoring based on the evaluation value obtained for each piece of business manual data. For example, the response evaluation unit 33 may perform scoring based on the ratio between the number of evaluation results with an evaluation value of 1 and the number of evaluation results with an evaluation value of -1. The response evaluation unit 33 may also cause the natural language processing device 50 to perform scoring by sending a prompt based on the evaluation result to the natural language processing device 50. Note that the response evaluation unit 33 according to another embodiment may perform a single evaluation based on the business manuals narrowed down by RAG, rather than evaluating each of a plurality of business manuals.

[0052] (Fourth embodiment) An agent may become mentally exhausted by handling calls. The evaluation system 1 according to the fourth embodiment not only evaluates the content of the agent's handling but also determines the agent's mental state.

[0053] The individual evaluation unit 35 of the management device 30 according to the fourth embodiment evaluates the mental state of an agent. One method for evaluating the mental state of an agent is to determine whether the agent is currently unable to perform tasks that the agent was previously able to perform. For example, the individual evaluation unit 35 according to the fourth embodiment compares the scores for each evaluation item related to responses during the evaluation period with the scores for each evaluation item related to responses prior to the evaluation period, and evaluates the agent's mental state as poor if the scores for the evaluation period are lower than the past scores by a difference equal to or greater than a predetermined threshold. The individual evaluation unit 35 may include response evaluation result data for the evaluation period and response evaluation result data for the prior evaluation period, and may cause the natural language processing device 50 to evaluate the agent's mental state by a prompt that prompts the agent to evaluate the agent's mental state.

[0054] The output unit 36 ​​according to the fourth embodiment displays the evaluation result of the mental state of the operator and the reason on the evaluation screen for the manager, thereby enabling the manager to follow up on the operator in consideration of the mental state.

[0055] (Summary) According to at least one embodiment described above, a presentation device such as the management device 30 includes a dialogue acquisition unit 31 and a response evaluation unit 33. The dialogue acquisition unit 31 acquires dialogue data representing the content of a dialogue between the person being evaluated and the interlocutor. The response evaluation unit 33 outputs a response evaluation instruction to the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data, and acquires a response evaluation result in accordance with the response evaluation instruction from the natural language processing device. The response evaluation instruction includes an instruction statement to score each of multiple evaluation items related to the response of the person being evaluated, and a criterion presentation statement presenting multiple evaluation criteria for each of the multiple evaluation items. This allows the presentation device to evaluate the quality of the dialogue content using a score based on certain criteria.

[0056] The evaluation system 1 according to the above embodiment evaluates the way operators at a call center interact with customers, but is not limited to this. For example, the evaluation system 1 according to other embodiments may evaluate other types of interactions, such as the way a sales representative approaches customers or the way a company's reception staff handles customers.

[0057] Furthermore, the natural language processing apparatus 50 according to the embodiment described above performs natural language processing based on a large-scale text-to-text language model, but is not limited to this. The natural language processing apparatus 50 may be any apparatus that receives an input of a sentence in a natural language and generates a sentence in the natural language corresponding to the input sentence.

[0058] FIG. 9 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91 , a main memory 92 , a storage 93 , and an interface 94 . The above-mentioned management device 30 is implemented in a computer 90. The operations of each of the above-mentioned processing units are stored in the form of a program in a storage 93. A processor 91 reads the program from the storage 93, loads it into a main memory 92, and executes the above-mentioned processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to each of the above-mentioned storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0059] The program may be for realizing some of the functions to be performed by the computer 90. For example, the program may be combined with other programs already stored in storage or implemented in other devices to perform the functions. In another embodiment, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 91 may be realized by the integrated circuit. Such an integrated circuit is also an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

[0060] Examples of storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0061] The program may also be a program for realizing some of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in storage 93.

[0062] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0063] 1...Evaluation system 10...Operator terminal 11...Input / output device 20...Administrator terminal 30...Management device 31...Dialogue acquisition unit 32...Text conversion unit 33...Response evaluation unit 34...Extraction unit 35...Personal evaluation unit 36...Output unit 37...Storage unit 38...Difficulty evaluation unit 50...Natural language processing device 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface

Claims

1. a dialogue acquisition unit that acquires dialogue data representing the content of a dialogue between an evaluee and a dialogue partner; a response evaluation unit that outputs a response evaluation instruction to a natural language processing device to cause the natural language processing device to evaluate the response of the person being evaluated with respect to the dialogue data, and obtains a response evaluation result in accordance with the response evaluation instruction from the natural language processing device; Equipped with The response evaluation instruction is An instruction to score each of a plurality of evaluation items related to the response of the person being evaluated; A criterion presentation statement presenting a plurality of evaluation criteria for each of the plurality of evaluation items; Including, Presentation device.

2. the instruction to evaluate the response includes, for each of the plurality of evaluation criteria, example sentences that illustrate a dialogue that satisfies the evaluation criteria or a dialogue that does not satisfy the evaluation criteria; The presentation device according to claim 1 .

3. the instruction for response evaluation includes a format suggestion statement for suggesting a format of the response evaluation result; an extraction unit that extracts, from the response evaluation result, the scoring results of the plurality of evaluation items in accordance with the format; The presentation device according to claim 1 .

4. a difficulty evaluation unit that outputs a difficulty evaluation instruction to a natural language processing device to evaluate the difficulty of the response by the person being evaluated for the dialogue data, and obtains a difficulty evaluation result from the natural language processing device in accordance with the difficulty evaluation instruction; Equipped with the standard statement included in the response evaluation instruction varies depending on the difficulty evaluation result. The presentation device according to claim 1 .

5. The instruction sentence of the response evaluation instruction includes an instruction to present the basis for the scoring from the content of the conversation. The presentation device according to claim 1 .

6. a personal evaluation unit that generates personal evaluation data indicating an evaluation of the person being evaluated from a plurality of response evaluation results corresponding to a plurality of dialogue data by the person being evaluated; The presentation device of claim 1 .

7. The dialogue data is recording data of a dialogue between the assessee and the interlocutor, the presentation device includes a speaker separation unit that generates dialogue sentence data from the dialogue data, in which a character string representing an utterance content is associated with a speaker of the utterance content; the instruction sentence of the response evaluation instruction includes an instruction to perform the scoring on the dialogue sentence data. The presentation device according to claim 1 .

8. The dialogue data is stereo audio data in which the speech of the subject and the speech of the interlocutor are recorded separately on the left and right. The presentation device according to claim 7 .

9. the natural language processing device has a large-scale language model; the response evaluation instruction is a prompt to be input to the natural language processing device; The presentation device according to any one of claims 1 to 8.

10. The computer acquiring dialogue data representing the content of a dialogue between an assessee and an interlocutor; a step of outputting a response evaluation instruction to a natural language processing device to cause the natural language processing device to evaluate the response of the evaluatee with respect to the dialogue data; acquiring an answer evaluation result according to the answer evaluation instruction by the natural language processing device; Equipped with The response evaluation instruction is An instruction to score each of a plurality of evaluation items related to the response of the person being evaluated; A criterion presentation statement presenting a plurality of evaluation criteria for each of the plurality of evaluation items; Including, Presentation method.

11. On the computer, acquiring dialogue data representing the content of a dialogue between an assessee and an interlocutor; a step of outputting a response evaluation instruction to a natural language processing device to cause the natural language processing device to evaluate the response of the evaluatee with respect to the dialogue data; acquiring an answer evaluation result according to the answer evaluation instruction by the natural language processing device; and The response evaluation instruction is An instruction to score each of a plurality of evaluation items related to the response of the person being evaluated; A criterion presentation statement presenting a plurality of evaluation criteria for each of the plurality of evaluation items; Including, program.

Images