Presentation device, presentation method, and program

Abstract

A presentation device according to an embodiment of the present invention comprises a dialogue acquisition unit and a handling evaluation unit. The dialogue acquisition unit acquires dialogue data which indicates the content of a dialogue between an evaluation subject and an interlocutor. The handling evaluation unit outputs a handling evaluation instruction for causing a natural language processing device to evaluate handling by the evaluation subject, in relation to the dialogue data. The handling evaluation unit acquires a handling evaluation result produced by the natural language processing device in accordance with the handling evaluation instruction. The handling evaluation instruction includes: an instruction sentence which is for performing scoring of each of a plurality of evaluation items relating to the handling by the evaluation subject; and a criteria presentation sentence which presents a plurality of evaluation criteria for each of the plurality of evaluation items.

Description

Presentation device, presentation method, and program

[0001] This application claims priority to Japanese Patent Application No. 2024-084384, filed May 23, 2024, the contents of which are incorporated herein by reference.

[0002] In call centers, there is a desire to evaluate the content of an operator's response to a customer. However, evaluating the content of a response takes time and is necessarily a qualitative judgment, making it difficult for an evaluator to evaluate all of the recording data of each response. In addition, it is difficult to standardize evaluation criteria among evaluators, and it is not uncommon for evaluation results to differ depending on the evaluator for the same response. Patent Document 1 discloses a technology for evaluating the response of an operator at a call center according to quantitative criteria such as the number of calls and the duration of the call.

[0003] Japanese Patent Application Laid-Open No. 2015-095758

[0004] The problem to be solved by the present invention is to provide a presentation device, a presentation method, and a program that can evaluate the quality of dialogue content using a score according to a certain standard.

[0005] According to an embodiment of the present invention, the presentation device has a dialogue acquisition unit and a response evaluation unit. The dialogue acquisition unit acquires dialogue data representing the content of the dialogue between the person being evaluated and the interlocutor. The response evaluation unit outputs a response evaluation instruction to cause the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data. The response evaluation unit acquires a response evaluation result in accordance with the response evaluation instruction from the natural language processing device. The response evaluation instruction includes an instruction statement to score each of multiple evaluation items related to the response of the person being evaluated, and a criterion presentation statement that presents multiple evaluation criteria for each of the multiple evaluation items.

[0006] FIG. 1 is a schematic block diagram showing the configuration of an evaluation system according to a first embodiment. FIG. 2 is a schematic block diagram showing the configuration of a management device according to the first embodiment. FIG. 3 is an example of a template for a response evaluation instruction prompt according to the first embodiment. FIG. 4 is a diagram showing an example of an evaluation screen for an operator according to the first embodiment. FIG. 5 is a diagram showing an example of an evaluation screen for a manager according to the first embodiment. FIG. 6 is a diagram showing an example of an evaluation screen for an individual response according to the first embodiment. FIG. 7 is a flowchart showing an evaluation method by the management device according to the first embodiment. FIG. 8 is a schematic block diagram showing the configuration of a management device according to a second embodiment. FIG. 9 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0007] Hereinafter, a presentation device, a presentation method, and a program according to embodiments will be described with reference to the drawings. (First Embodiment) Fig. 1 is a schematic block diagram showing the configuration of an evaluation system 1 according to a first embodiment. The evaluation system 1 according to the first embodiment evaluates calls made by operators at a call center. The evaluation system 1 includes a plurality of operator terminals 10, an administrator terminal 20, a management device 30, and a natural language processing device 50.

[0008] The operator terminal 10 is a terminal operated by an operator at a call center. The operator terminal 10 is, for example, a telephone, and enables calls between customers at the call center and operators. The operator terminal 10 is provided with an input / output device 11 having a microphone and a speaker. Examples of the input / output device 11 include a headset and a handset. The administrator terminal 20 is a terminal operated by an administrator who manages the operators.

[0009] The management device 30 controls a plurality of operator terminals 10. For example, the management device 30 may be a CTI (Computer Telephony Integration) system. The operator terminals 10 and the management device 30 are connected to each other, for example, by a LAN (Local Area Network). The management device 30 records calls made by each operator terminal 10 and generates call voice data for each operator. Based on the call voice data, the management device 30 transmits a prompt to the natural language processing device 50 to cause the operator to evaluate the content of the call. The management device 30 receives the evaluation result from the natural language processing device 50 and generates information representing the operator's evaluation. The management device 30 is an example of a presentation device that presents the call evaluation result to an operator or administrator. The management device 30 communicates with the natural language processing device 50 via the Internet or the like.

[0010] The natural language processing device 50 receives input of natural language sentences called prompts via the Internet or the like, and generates natural language sentences in response to the input prompts. Note that the natural language processing device 50 may input and output not only sentences written in natural languages, but also sentences written in artificial languages. The natural language processing device 50 according to the first embodiment has a text-to-text large-scale language model, inputs prompts to the large-scale language model, and outputs sentences generated by the large-scale language model in response to the prompts.

[0011] 2 is a schematic block diagram showing the configuration of the management device 30 according to the first embodiment. The management device 30 includes a dialogue acquisition unit 31, a text conversion unit 32, a response evaluation unit 33, an extraction unit 34, a personal evaluation unit 35, an output unit 36, and a storage unit 37.

[0012] The dialogue acquisition unit 31 acquires, from the operator terminal 10, voice data representing the content of a conversation between a customer and an operator. The voice data is stereo voice data in which the voice of the customer and the voice of the operator are recorded separately on two channels. In other words, the operator terminal 10 records the voice input to the microphone of the input / output device 11 and the voice received from the outside via voice communication separately on two channels. Which of the voice of the customer and the voice of the operator is to be recorded on the left and which on the right may be set in advance in each operator terminal 10 by the management device 30. The dialogue acquisition unit 31 can identify which operator's dialogue the acquired voice data represents, based on the login information of the operator terminal 10 from which the voice data was acquired.

[0013] The text conversion unit 32 converts the voice data acquired by the dialogue acquisition unit 31 into text data. In other words, the text conversion unit 32 generates text data by performing voice recognition processing on the voice data. Hereinafter, the text data representing the dialogue content generated by the text conversion unit 32 will be referred to as dialogue text data. At this time, the text conversion unit 32 generates dialogue text data in which a label indicating the speaker and the time of utterance are attached to each utterance included in the voice data. For example, the format of the dialogue text data generated by the text conversion unit 32 may be as shown below. Note that in the following sentences, "{}" represent variables.

[0014] "{Utterance start time} - {Utterance end time}, {Speaker}: {Utterance content}" Note that in other embodiments, the dialogue text data may not include information on the utterance start time and utterance end time. On the other hand, if the dialogue text data includes time information, the natural language processing device 50 can evaluate the speed and timing of the utterance. The dialogue text data may be associated with non-verbal information such as the speaking rate, the interval between the utterance and the previous utterance, various statistics (e.g., mean, variance) of acoustic features (e.g., fundamental frequency, volume) obtained by acoustic analysis of the speech data, and emotional values ​​obtained by speech emotion analysis. By using this non-verbal information obtained from speech, the natural language processing device 50 can improve the evaluation accuracy of evaluation items that are difficult to infer from text data alone, such as the impression of the conversation and attitude.

[0015] For example, the text conversion unit 32 may generate dialogue text data in the following procedure. The text conversion unit 32 performs speech recognition processing on each channel of the voice data. The text conversion unit 32 generates text data of each utterance by an operator according to the above format from text data of the utterance content obtained from a channel assigned to the operator's voice, a label of the operator who is the speaker, and the utterance start time and end time. The text conversion unit 32 generates text data of each utterance by a customer according to the above format from text data of the utterance content obtained from a channel assigned to the customer's voice, a label of the customer who is the speaker, and the utterance start time and end time. The text conversion unit 32 generates dialogue text data by chronologically arranging the text data of each utterance. The text conversion unit 32 is an example of a speaker separation unit that generates dialogue sentence data from dialogue data, associating character strings representing the utterance content with the speaker of the utterance content.

[0016] The response evaluation unit 33 generates a response evaluation instruction prompt based on the dialogue text data generated by the text conversion unit 32, and transmits the response evaluation instruction prompt to the natural language processing device 50. The response evaluation instruction prompt includes the following information: Instructions to score each of a plurality of evaluation items related to the agent's response Explanation of each of the plurality of evaluation items Explanation of a plurality of evaluation criteria related to each of the plurality of evaluation items Example of a dialogue related to each of the plurality of evaluation criteria Format of the evaluation result

[0017] For example, the response evaluation unit 33 may store a response evaluation instruction prompt template in advance and generate the response evaluation instruction prompt by embedding dialogue text data in variables included in the template. FIG. 3 shows an example of a response evaluation instruction prompt template according to the first embodiment. As shown in FIG. 3, the response evaluation instruction prompt template may be, for example, as follows: Note that in FIG. 3 and the following text, "{}" represents a variable.

[0018] 」 English: "#Instructions You are a consultant who supports call center staff in improving the quality of customer service. Read the following dialogue and rate the quality of the operator's service for each evaluation item on a scale of 1 to 10. Please also explain in detail the reasons for your rating using the dialogue. #Evaluation items 1. Response that is considerate of the customer's feelings 1A. Conveys specific understanding of the customer's feelings Good example: We understand your anxiety at being unable to provide our product, which you have been using daily. 1B. ...... Example: ...... 2. ...... 3. ...... #Format <Evaluation item> Evaluation value: <Evaluation value from 1 to 10> Reason: <Reason> #Dialogue {Dialogue text data}" Note that the evaluation items or evaluation criteria are not limited to evaluating the operator's service itself; they can also evaluate the customer's impression or reaction to the operator's service, such as "customer satisfaction with the operator's suggestions" or "customer confidence in the product or service they inquired about." In addition, when an evaluation item contains a mixture of evaluation criteria related to the evaluation of the operator's response and evaluation criteria related to the customer's reaction to the operator's response, the natural language processing device 50 calculates the evaluation value of the evaluation item by taking both into consideration, thereby making it possible to make an evaluation based on both the operator's response and how the customer felt about the response.

[0019] The response evaluation unit 33 acquires a sentence generated in response to the response evaluation instruction prompt by the natural language processing device 50. The sentence acquired by the response evaluation unit 33 is a sentence that conforms to the format included in the response evaluation instruction prompt.

[0020] The extraction unit 34 extracts the evaluation value and the reason for each evaluation item from the sentence acquired by the response evaluation unit 33 in accordance with the format specified in the response evaluation instruction prompt. For each interaction, the extraction unit 34 associates the date and time of the interaction, the ID of the corresponding agent, and the evaluation value and reason for each evaluation item, and records the resulting data as interaction evaluation data in the storage unit 37. By extracting the evaluation value from the response evaluation result data, the extraction unit 34 can treat the evaluation value included in the response evaluation result data as a numerical value.

[0021] The personal evaluation unit 35 generates a personal evaluation instruction prompt for performing an evaluation of the agent from a plurality of pieces of interaction evaluation data associated with the agent to be evaluated, and transmits the personal evaluation instruction prompt to the natural language processing device 50. The personal evaluation instruction prompt includes the following information: Instruction to generate advice for the agent from the response evaluation result data Format of the evaluation result

[0022] For example, the individual evaluation unit 35 may store a template of an individual evaluation instruction prompt in advance and generate the individual evaluation instruction prompt by embedding dialogue text data in variables included in the template. The template of the individual evaluation instruction prompt may be, for example, as follows:

[0023] "#Instructions You are a consultant who supports call center staff in improving the quality of customer service. Read the evaluation of the operator's service and provide advice to improve the quality of the service, including specific evaluation points for each service process and the overall service. #Service process 1. Listen to the customer 2. Organize what the customer has said 3. Present the overall picture 4. Present a specific response policy 3. Make an appointment for the next service #Format <Service process> Good points: <Advice> Areas for improvement: <Advice> #Service evaluation {Service evaluation result data} {Service evaluation result data} ..."

[0024] The individual evaluation instruction prompt is accompanied by multiple pieces of response evaluation result data relating to the evaluation period. The natural language processing device 50 generally has a limit on the number of characters in a prompt, and may not be able to process long prompts. Therefore, the individual evaluation unit 35 can reduce the number of characters in the prompt by attaching response evaluation result data with a relatively small number of characters rather than attaching the entire dialogue text data relating to the evaluation period to the prompt. In particular, the response evaluation result data has a specified format, as shown in FIG. 3 , so that the number of characters is less likely to vary. In another embodiment, instead of the response evaluation result data itself, for example, data that has been previously summarized using the natural language processing device 50 may be included in the individual evaluation instruction prompt. This allows the individual evaluation instruction prompt to be compressed into shorter text that includes information important for evaluation and omits unnecessary information. Furthermore, in another embodiment, instead of the response evaluation result data for each response, the individual evaluation instruction prompt may include data that has been previously summarized using the natural language processing device 50 from multiple pieces of response result data. In another embodiment, instead of the response evaluation result data, statistics of multiple pieces of response evaluation result data may be included in the individual evaluation instruction prompt. By providing such summaries, the management device 30 can further improve the accuracy of the answers while reducing the number of characters in the prompt.

[0025] The individual evaluation unit 35 acquires the sentence generated by the natural language processing device 50 in response to the individual evaluation instruction prompt. The extraction unit 34 extracts advice from the sentence acquired by the individual evaluation unit 35 in accordance with the format specified in the individual evaluation instruction prompt. The extraction unit 34 associates the evaluation period, the operator ID, and the advice for each operator, and records them in the storage unit 37 as individual evaluation result data.

[0026] The output unit 36 ​​generates an evaluation screen based on the response evaluation result data acquired by the response evaluation unit 33 and the personal evaluation result data acquired by the personal evaluation unit 35, and outputs it to the operator terminal 10 or the administrator terminal 20.

[0027] 4 is a diagram showing an example of an evaluation screen for operators according to the first embodiment. The evaluation screen for operators displays a graph showing scores for each evaluation item for the responses of the operator to be evaluated during the evaluation period, advice for the responses during the evaluation period, evaluation points for each response process, and a list of responses during the evaluation period. The output unit 36 ​​can generate a graph showing scores for the responses during the evaluation period by calculating the average score for each evaluation item from multiple response evaluation result data for the agent to be evaluated during the evaluation period.

[0028] 5 is a diagram showing an example of an evaluation screen for a manager according to the first embodiment. The evaluation screen for a manager displays a graph showing the scores for each evaluation item for the entire team managed by the manager during an evaluation period, trends for the entire team, advice for the agents belonging to the team, changes in the scores of the agents belonging to the team, and a list of responses during the evaluation period.

[0029] The list of responses during the evaluation period shown in FIG. 4 or FIG. 5 displays an evaluation value, date and time, a summary, and a Details button for each response. The list of responses may also include the operator's name. When the Details button is pressed, the output unit 36 ​​outputs an evaluation screen showing response evaluation result data to the operator terminal 10 or the administrator terminal 20. FIG. 6 is a diagram showing an example of an evaluation screen for an individual response according to the first embodiment. The response evaluation screen displays a graph showing the scores for each evaluation item for the selected response, an overall evaluation and advice for the response, and the score for each evaluation item and the reasons for the score. Note that in FIGS. 4-6, four items, communication skills, efficient behavior, problem-solving ability, and gaining trust, are listed as examples of evaluation items, but these are merely examples and are not limited to these.

[0030] 7 is a flowchart showing an evaluation method by the management device 30 according to the first embodiment. The dialogue acquisition unit 31 of the management device 30 acquires voice data from the operator terminal 10 for each dialogue between a customer and an agent. The text conversion unit 32 generates dialogue text data consisting of pairs of speakers and spoken content from the voice data acquired by the dialogue acquisition unit 31, and records the data in the storage unit 37 in association with the operator ID and the date and time of the response.

[0031] The response evaluation unit 33 of the management device 30 extracts the response dates and times that belong to the evaluation period from the dialogue text data recorded in the storage unit 37 (step S1). The response evaluation unit 33 selects each of the extracted dialogue text data (step S2) and executes the following processes from step S3 to step S7.

[0032] The response evaluation unit 33 generates a response evaluation instruction prompt for the selected dialogue text data according to a predetermined template (step S3). The response evaluation unit 33 transmits the generated response evaluation instruction prompt to the natural language processing device 50 (step S4). The response evaluation unit 33 receives a sentence generated based on the response evaluation instruction prompt from the natural language processing device 50 (step S5). The extraction unit 34 extracts an evaluation value and a reason for each evaluation item from the received sentence (step S6). The extraction unit 34 associates the extracted evaluation value and reason for each evaluation item with the agent ID and response date and time related to the dialogue text data selected in step S2 and records them in the storage unit 37 as response evaluation result data (step S7).

[0033] When the management device 30 executes the processes of steps S3 to S7 for all the dialogue text data extracted in step S1, the personal evaluation unit 35 selects operators one by one (step S8) and executes the processes of steps S9 to S14 below.

[0034] The individual evaluation unit 35 reads all response evaluation result data within the evaluation period associated with the agent selected in step S8 from the storage unit 37 (step S9). The individual evaluation unit 35 generates an individual evaluation instruction prompt including the response evaluation result data read in step S9 according to a predetermined template (step S10). The individual evaluation unit 35 transmits the generated individual evaluation instruction prompt to the natural language processing device 50 (step S11). The individual evaluation unit 35 receives a sentence generated based on the individual evaluation instruction prompt from the natural language processing device 50 (step S12). The extraction unit 34 extracts advice for the entire response and each response process from the received sentence (step S13). The extraction unit 34 associates the information extracted in step S13 with the ID of the agent selected in step S8 and records the information extracted in step S13 as individual evaluation result data in the storage unit 37 (step S14).

[0035] The output unit 36 ​​receives a request to view the evaluation results from the operator terminal 10 or the manager terminal 20 (step S15). The output unit 36 ​​reads out from the storage unit 37 the individual evaluation result data associated with the operator related to the viewing request or the operator related to the team under the manager's jurisdiction (step S16). The output unit 36 ​​generates an evaluation screen based on the read individual evaluation result data and transmits the display data of the evaluation screen to the operator terminal 10 or the manager terminal 20 that sent the viewing request (step S17).

[0036] As described above, the evaluation system 1 according to the first embodiment functions as follows. The dialogue acquisition unit 31 acquires dialogue data representing the content of a dialogue between an agent and a customer. The response evaluation unit 33 outputs a response evaluation instruction prompt to the natural language processing device 50 to evaluate the agent's response based on the dialogue data, and acquires the response evaluation results from the natural language processing device 50. The response evaluation instruction prompt includes an instruction statement to perform scoring for each of multiple evaluation items related to the agent's response, and a criterion presentation statement presenting multiple evaluation criteria for each of the multiple evaluation items. By presenting multiple evaluation criteria for each evaluation item, the evaluation system 1 enables the natural language processing device 50 to reduce fluctuations in evaluation for the evaluation items and perform scoring according to certain criteria.

[0037] The response evaluation prompt according to the first embodiment includes example sentences that illustrate, for each of a plurality of evaluation criteria, dialogues that satisfy the evaluation criteria or dialogues that do not satisfy the evaluation criteria. This allows the natural language processing device 50 to create evaluation sentences based on the examples, thereby preventing variations in evaluation.

[0038] The response evaluation instruction prompt according to the first embodiment includes a format suggestion sentence that suggests a format for the evaluation results. The extraction unit 34 of the evaluation system 1 extracts the scoring results for multiple evaluation items from the evaluation results according to the format. The natural language processing device 50 can generate free sentences, but by specifying a format, it can extract the scoring results as numerical values ​​that can be used for calculating average values, etc.

[0039] Second Embodiment The evaluation system 1 according to the first embodiment evaluates each dialogue in accordance with the response evaluation instruction prompt shown in FIG. 3 . However, because the level of difficulty of a dialogue varies depending on the dialogue, it may not always be appropriate to perform a uniform evaluation. For example, if the customer's purpose is clear and the customer is calm, the level of difficulty for the operator to handle the dialogue may be low, whereas if the customer's purpose is unclear or the customer is very angry, the level of difficulty for the operator to handle the dialogue may be high. The evaluation system 1 according to the second embodiment realizes evaluation according to the level of difficulty of a dialogue.

[0040] 8 is a schematic block diagram showing the configuration of a management device 30 according to the second embodiment. The management device 30 according to the second embodiment further includes a difficulty evaluation unit 38 in addition to the configuration of the first embodiment.

[0041] The difficulty evaluation unit 38 generates a difficulty evaluation instruction prompt based on the dialogue text data and transmits the difficulty evaluation instruction prompt to the natural language processing device 50. The difficulty evaluation instruction prompt includes the following information: An instruction to evaluate the difficulty of the operator's response in the dialogue as a numerical value An explanation of the difficulty evaluation criteria The format of the evaluation result

[0042] For example, the difficulty evaluation unit 38 may store a template of a difficulty evaluation instruction prompt in advance and generate the difficulty evaluation instruction prompt by embedding dialogue text data in variables included in the template. The template of the difficulty evaluation instruction prompt may be, for example, as follows:

[0043] "#Instructions Please read the following dialogue and rate the difficulty of the operator's response on a scale of 1 to 3. #Evaluation criteria 1. If the customer's response to the operator's utterance is not a logical response, the difficulty is high. Example: ... 2. If the customer speaks forcefully, the difficulty is high. Example: ... 3. ... #Format Difficulty: <Difficulty of 1 to 3 points> <Reason> #Dialogue {Dialogue text data}"

[0044] The difficulty evaluation unit 38 acquires a sentence generated in response to the difficulty evaluation instruction prompt by the natural language processing device 50. The sentence acquired by the difficulty evaluation unit 38 is a sentence that conforms to the format included in the difficulty evaluation instruction prompt.

[0045] The response evaluation unit 33 may generate a response evaluation prompt according to a template corresponding to the evaluation by the difficulty evaluation unit 38. For example, the response evaluation unit 33 may prepare a template for difficulty level 1, a template for difficulty level 2, and a template for difficulty level 3, and generate a response evaluation prompt according to the template corresponding to the difficulty level identified by the difficulty evaluation unit 38. Each template has different sentences that present multiple evaluation criteria for at least multiple evaluation items. This allows the response evaluation unit 33 to perform an evaluation according to the difficulty level.

[0046] Furthermore, the individual evaluation unit 35 may calculate the average evaluation score by adjusting the evaluation score of each response according to the level of difficulty. For example, the individual evaluation unit 35 may apply a large weight to a response with a high level of difficulty when a high evaluation score is assigned to the response, and a small weight to a response with a low level of difficulty when a low evaluation score is assigned to the response. This allows the individual evaluation unit 35 to highly evaluate a response with a high level of difficulty for being properly performed, while preventing a low evaluation from being immediately given even if the response was not handled properly. For example, the individual evaluation unit 35 may apply a small weight to a response with a low level of difficulty when a high evaluation score is assigned to the response, and a large weight to a response with a low level of difficulty when a low evaluation score is assigned to the response. This allows the individual evaluation unit 35 to prevent a high evaluation from being immediately given just because a response with a low level of difficulty was properly performed, and to give a low evaluation for an inappropriate response despite the response being low in difficulty.

[0047] In addition, when creating an evaluation screen for managers, the output unit 36 ​​extracts evaluations of responses judged to be difficult for operators within the team and presents them in a comparable manner, thereby allowing managers to evaluate the response capabilities of each operator.

[0048] Third Embodiment The management device 30 according to the above-described embodiment evaluates responses according to a prepared template. On the other hand, in order to evaluate whether a response conforms to a business manual, for example, the contents of the business manual must be presented to the natural language processing device 50, and the prompt may exceed the number of characters that can be processed by the natural language processing device 50. Therefore, the management device 30 according to the third embodiment generates a response evaluation instruction prompt that causes an evaluation to be performed based on documents such as business manuals narrowed down by RAG (Retrieval-Augmented Generation).

[0049] The response evaluation unit 33 according to the third embodiment can cause the natural language processing device 50 to score whether the response is in accordance with the business manual, for example, by the following procedure. The response evaluation unit 33 transmits a prompt to the natural language processing device 50 for evaluating whether the dialogue is in accordance with the business manual for each piece of business manual data prepared in advance. The template of the prompt may be, for example, as follows:

[0050] "#Instructions Please determine whether the following dialogue is in accordance with the business manual. If the dialogue is in accordance with the business manual, return 1; if the dialogue is not in accordance with the business manual, return -1; if the dialogue cannot be evaluated in the business manual, return 0 as the evaluation value. #Format <Evaluation value from -1 to 1> <Reason> #Dialogue {Dialogue text data} #Business manual {Business manual data}"

[0051] The response evaluation unit 33 performs scoring based on the evaluation value obtained for each piece of business manual data. For example, the response evaluation unit 33 may perform scoring based on the ratio of the number of evaluation results with an evaluation value of 1 to the number of evaluation results with an evaluation value of -1. The response evaluation unit 33 may also cause the natural language processing device 50 to perform scoring by sending a prompt based on the evaluation result to the natural language processing device 50. Note that the response evaluation unit 33 according to another embodiment may perform a single evaluation based on the business manuals narrowed down by RAG, rather than evaluating each of a plurality of business manuals.

[0052] Fourth Embodiment An agent may become mentally exhausted by handling calls. An evaluation system 1 according to a fourth embodiment not only evaluates the content of the agent's handling but also determines the agent's mental state.

[0053] The individual evaluation unit 35 of the management device 30 according to the fourth embodiment evaluates the mental state of an agent. One method for evaluating the mental state of an agent is to determine whether the agent is currently unable to do what he or she was able to do in the past. For example, the individual evaluation unit 35 according to the fourth embodiment compares the scores for each evaluation item related to responses during the evaluation period with the scores for each evaluation item related to responses prior to the evaluation period, and evaluates the agent's mental state as poor if the scores during the evaluation period are lower than the past scores by a difference equal to or greater than a predetermined threshold. The individual evaluation unit 35 may include response evaluation result data during the evaluation period and response evaluation result data prior to the evaluation period, and may cause the natural language processing device 50 to evaluate the agent's mental state by a prompt that prompts the agent to evaluate the agent's mental state.

[0054] The output unit 36 ​​according to the fourth embodiment displays the evaluation result of the mental state of the operator and the reason for the evaluation on the evaluation screen for the manager, thereby enabling the manager to follow up on the operator in consideration of the mental state.

[0055] (Summary) According to at least one embodiment described above, a presentation device such as the management device 30 includes a dialogue acquisition unit 31 and a response evaluation unit 33. The dialogue acquisition unit 31 acquires dialogue data representing the content of a dialogue between the person being evaluated and the interlocutor. The response evaluation unit 33 outputs a response evaluation instruction to the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data, and acquires a response evaluation result in accordance with the response evaluation instruction from the natural language processing device. The response evaluation instruction includes an instruction statement to score each of multiple evaluation items related to the response of the person being evaluated, and a criterion presentation statement presenting multiple evaluation criteria for each of the multiple evaluation items. This allows the presentation device to evaluate the quality of the dialogue content using a score based on certain criteria.

[0056] The evaluation system 1 according to the above-described embodiment evaluates the way operators at a call center interact with customers, but is not limited to this. For example, the evaluation system 1 according to other embodiments may evaluate other types of interactions, such as the way a sales representative approaches customers or the way a company's reception staff handles customers.

[0057] Furthermore, although the natural language processing apparatus 50 according to the above-described embodiment performs natural language processing based on a large-scale text-to-text language model, the natural language processing apparatus 50 is not limited to this. The natural language processing apparatus 50 may be any apparatus that receives an input of a sentence in a natural language and generates a sentence in a natural language corresponding to the input sentence.

[0058] 9 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, a main memory 92, a storage 93, and an interface 94. The management device 30 described above is implemented in the computer 90. The operations of each of the processing units described above are stored in the storage 93 in the form of a program. The processor 91 reads the program from the storage 93, loads it into the main memory 92, and executes the above-described processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to each of the storage units described above in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0059] The program may be for implementing some of the functions to be performed by the computer 90. For example, the program may be implemented in combination with other programs already stored in storage or in combination with other programs implemented in other devices. In another embodiment, the computer 90 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

[0060] Examples of storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0061] The program may also be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in storage 93.

[0062] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents.

[0063] According to one embodiment, the above-described presentation device can be applied to a service for evaluating the responses of operators at a call center.

[0064] 1...Evaluation system 10...Operator terminal 11...Input / output device 20...Administrator terminal 30...Management device 31...Dialogue acquisition unit 32...Text conversion unit 33...Response evaluation unit 34...Extraction unit 35...Personal evaluation unit 36...Output unit 37...Storage unit 38...Difficulty evaluation unit 50...Natural language processing device 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface

Claims

1. A presentation device comprising: a dialogue acquisition unit that acquires dialogue data representing the content of the dialogue between the person being evaluated and the interlocutor; and a response evaluation unit that outputs response evaluation instructions to a natural language processing device to cause the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data, and acquires response evaluation results in accordance with the response evaluation instructions from the natural language processing device, wherein the response evaluation instructions include: an instruction statement that causes scoring to be carried out for each of a plurality of evaluation items related to the response of the person being evaluated; and a criterion presentation statement that presents a plurality of evaluation criteria for each of the plurality of evaluation items.

2. The presentation device according to claim 1, wherein the response evaluation instructions include example sentences that exemplify, for each of the plurality of evaluation criteria, a dialogue that satisfies the evaluation criteria or a dialogue that does not satisfy the evaluation criteria.

3. The presentation device according to claim 1, wherein the response evaluation instructions include a format suggestion statement that suggests a format for the response evaluation results, and the presentation device is equipped with an extraction unit that extracts the scoring results for the multiple evaluation items from the response evaluation results in accordance with the format.

4. A presentation device as described in claim 1, comprising: a difficulty evaluation unit that outputs difficulty evaluation instructions to a natural language processing device to cause the natural language processing device to evaluate the difficulty of the response by the person being evaluated for the dialogue data, and obtains a difficulty evaluation result in accordance with the difficulty evaluation instructions by the natural language processing device, wherein the standard presentation sentence included in the response evaluation instructions varies depending on the difficulty evaluation result.

5. The presentation device according to claim 1, wherein the instruction sentence of the response evaluation instruction includes an instruction to present the basis for the scoring from the content of the conversation.

6. The presentation device according to claim 1, further comprising: a personal evaluation unit that generates personal evaluation data indicating an evaluation of the person being evaluated from a plurality of response evaluation results corresponding to a plurality of dialogue data by the person being evaluated.

7. The presentation device described in claim 1, wherein the dialogue data is recorded data of a dialogue between the person being evaluated and the interlocutor, the presentation device is equipped with a speaker separation unit that generates dialogue text data from the dialogue data, associating a character string representing the content of an utterance with the speaker of the utterance content, and the instruction sentence of the response evaluation instruction includes an instruction to perform the scoring on the dialogue text data.

8. The presentation device according to claim 7, wherein the dialogue data is stereo audio data in which the speech of the person being evaluated and the speech of the interlocutor are recorded separately on the left and right.

9. The presentation device according to any one of claims 1 to 8, wherein the natural language processing device has a large-scale language model, and the instruction to evaluate the response is a prompt to be input to the natural language processing device.

10. A presentation method comprising the steps of: a computer acquiring dialogue data representing the content of a dialogue between a person being evaluated and an interlocutor; outputting a response evaluation instruction to a natural language processing device to cause the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data; and acquiring a response evaluation result in accordance with the response evaluation instruction by the natural language processing device, wherein the response evaluation instruction includes an instruction statement to score each of a plurality of evaluation items related to the response of the person being evaluated; and a criterion presentation statement presenting a plurality of evaluation criteria for each of the plurality of evaluation items.

11. A program that causes a computer to perform the following steps: acquiring dialogue data representing the content of a dialogue between a person being evaluated and an interlocutor; outputting response evaluation instructions to a natural language processing device to cause the natural language processing device to evaluate the response of the person being evaluated based on the dialogue data; and acquiring response evaluation results in accordance with the response evaluation instructions by the natural language processing device, wherein the response evaluation instructions include an instruction statement that causes scoring to be performed for each of a plurality of evaluation items related to the response of the person being evaluated, and a criterion presentation statement that presents a plurality of evaluation criteria for each of the plurality of evaluation items.

Images