Information processing device, information processing method, and program

Abstract

To appropriately assess customer emotions in a call center. [Solution] An information processing device comprising: an acquisition unit that acquires text data indicating the content of a conversation between a customer and an operator in a call center; an evaluation unit that inputs evaluation data based on the text data and a prompt including a request to select from multiple candidates that categorize the customer's emotions into a large language model (LLM) and acquires the output of the large language model to evaluate the customer's emotions; and an output unit that outputs the processing result of the evaluation unit.

Description

Technical Field

[0004] ,

[0006] , , , , ,

[0005] , , , , , , ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, inventions of apparatuses for evaluating an operator's response in a call center and diagnosing response skills have been disclosed (Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

【Patent Document

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] The above conventional technology rather evaluates the operator's response itself and does not attempt to evaluate the customer's feelings. Also, as a result of setting rules in detail based on rules, the obtained merits may not match the processing cost. Thus, in the conventional technology, there were cases where the customer's feelings in the call center could not be suitably evaluated.

[0005] The present invention has been made in consideration of such circumstances, and one of the objectives is to provide an information processing apparatus, an information processing method, and a program capable of suitably evaluating the customer's feelings in a call center.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing device comprising: an acquisition unit that acquires text data indicating the content of a conversation between a customer and an operator in a call center; an evaluation unit that inputs evaluation data based on the text data and a prompt including a request to select from a plurality of candidates that categorize the customer's emotions into a large language model (LLM), and evaluates the customer's emotions by acquiring the output of the large language model; and an output unit that outputs the processing result of the evaluation unit. [Effects of the Invention]

[0007] According to one aspect of the present invention, customer emotions in a call center can be appropriately evaluated. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the configuration and operating environment of the information processing device 100. [Figure 2] This figure shows an example of a validation routine to improve prompts. [Figure 3] This figure shows an example of the response status of conventional surveys. [Figure 4] This figure compares the results obtained from the verification experiment of this embodiment with those of the conventional method. [Modes for carrying out the invention]

[0009] [overview] The following describes embodiments of the information processing device, information processing method, and program according to the present invention with reference to the drawings. The information processing device evaluates and outputs customer emotions from conversations between customers and operators in a call center provided in conjunction with various services. The conversation is not limited to voice calls; it may also be a text-based chat. The service is, for example, a network service such as an electronic payment service.

[0010] Figure 1 shows an example of the configuration and operating environment of the information processing device 100. The information processing device 100 acquires conversations between the customer 10 and the operator 60 of the call center 50 in the form of voice data or transcribed text data (if the conversation is a chat, the chat content becomes the text data as is). In the following, the information processing device 100 will be assumed to acquire conversations from the call center 50 in the form of voice data via the network NW, but the voice data or text data may also be delivered to the information processing device 100 in a state where it is stored on a portable storage medium or the like.

[0011] A terminal device 180 is connected to the information processing device 100. The terminal device 180 is, for example, a general-purpose personal computer, a tablet terminal, or a smartphone. The terminal device 180 functions as an input / output device for the information processing device 100. The terminal device 180 may communicate with the information processing device 100 via a network NW, but here it is assumed to be connected via a local communication line.

[0012] Furthermore, the information processing device 100 communicates with the LLM server 200 via the network. The LLM server 200 provides Large Language Models (LLM) services via the network. LLMs are built on collective intelligence such as crawled data and are a type of generative AI. By learning from large amounts of text data, LLMs understand patterns in human language and perform natural language generation (NLG) tasks. For example, an LLM divides input information into tokens (morphemes), vectorizes them, and outputs a response that corresponds to contextual understanding by inputting the vectors into a trained model such as a Deep Neural Network (DNN). It is preferable that the LLM can be fine-tuned by the information processing device 100. The LLM may not be provided by the LLM server 200, which is an external device of the information processing device 100, but may be an internal function of the information processing device 100 (local LLM). In this case, the information processing device 100 holds a program equivalent to the LLM in its memory and uses the LLM's functions by executing it. When using a local LLM, it becomes possible to actively handle customer personal information, etc.

[0013] The information processing device 100 may be a function of a service server (for example, a payment server in an electronic payment service) that provides services corresponding to the call center 50, i.e., a virtual machine, but here it will be described as an independent device.

[0014] [composition] The information processing device 100 includes, for example, an acquisition unit 110, an evaluation unit 120, and an output unit 130. The acquisition unit 110 includes a text conversion processing unit 112, and the evaluation unit 120 includes a pre-processing unit 122, a prompt generation unit 124, and a request unit 126. These components are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components are LSIs (Large Scale Integrations), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Graphite Arrays). The program may be implemented by hardware (including circuitry) such as a Gate Array or a GPU (Graphics Processing Unit), or by the collaboration of software and hardware. The program may be stored in advance on a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored on a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed on the storage device when the storage medium is inserted into a drive device.

[0015] Furthermore, the information processing device 100 includes a storage unit 150. The storage unit 150 can be an HDD, flash memory, RAM (Random Access Memory), etc. The storage unit 150 may also be a NAS (Network Attached Storage) device that the information processing device 100 can access via a network. Information such as a prompt format 152 is stored in the storage unit 150.

[0016] The acquisition unit 110 acquires text data representing the content of conversations between customers 10 and operators 60 in the call center 50. For example, the acquisition unit 110 acquires text data by having the text processing unit 112 perform text processing on the audio data. The text processing includes, for example, speech recognition processing and morphological analysis processing.

[0017] The evaluation unit 120 evaluates the sentiment of customer 10 by inputting a prompt including a request to select from the evaluated data based on text data and a plurality of candidates classifying the sentiment of customer 10 into the LLM and obtaining the output of the LLM. The sentiment of the customer is classified into, for example, two types: "positive (favorable)" and "negative (negative, backward)", but may also be classified into other sentiments such as "neutral", "indifferent", "angry", etc.

[0018] Among the evaluation unit 120, the preprocessing unit 122 generates evaluated data by deleting unnecessary information from the text data. Unnecessary information includes unnecessary words and information that does not require determination (for example, mere confirmation items, time information at which the utterance was made, elapsed time information indicating how much time has elapsed since the start of the conversation).

[0019] The prompt generation unit 124 generates a prompt for input to the LLM. For example, the prompt generation unit 124 obtains the text constituting a part of the prompt from the prompt format 152 and generates a prompt. The prompt format 152 describes the full text or parts of the prompt in, for example, several patterns, and the prompt generation unit 124 appropriately obtains the text from these to generate a prompt. As an example, in the [prompt example] (paragraph number

[0024] ) described later, for evaluating the emotions of "positive" and "negative" customers, parts of the prompt such as "# output format", "# example 1", and "# example 2" that include descriptions regarding positive / negative judgments are selected. According to the present embodiment, for example, the prompt generation unit 124 may read out parts of the prompt stored in the storage unit 150 in advance according to the type of emotion to be evaluated (for example, in addition to "positive" and "negative", types such as "neutral", "indifferent", and "anger" are also assumed), and generate a prompt for input to the LLM. In this case, regarding which customer's emotion is to be the evaluation target, the designation of the user may be received via the acquisition unit 110. Note that the processing of the prompt generation unit 124 may be partially performed with human manual work such as the operation of selecting parts. Also, the prompt includes the data to be evaluated, and the prompt generation unit 124 fits the data to be evaluated into a format suitable for input to the LLM and makes it a part of the prompt.

[0020] The request unit 126 inputs the prompt generated by the prompt generation unit 124 to the LLM using, for example, the API (Application Programming Interface) provided by the LLM server 200, and obtains the output of the LLM. Alternatively, the request unit 126 may input the prompt to the LLM by inputting the prompt to the website provided by the LLM server 200.

[0021] The output unit 130 outputs the processing results of the evaluation unit 120 to, for example, a terminal device 180 for display. The output unit 130 has, for example, the functionality of a web server, and outputs the processing results of the evaluation unit 120 to the terminal device 180 in the format of a web page, and displays the image in the browser running on the terminal device 180.

[0022] The prompts generated by the information processing device 100 are improved through the verification routine shown in Figure 2. Figure 2 shows an example of a verification routine for improving prompts. First, when an initial prompt is input, the request unit inputs it into the LLM and obtains the output of the LLM. Next, correctness determination and correctness analysis are performed using any method. At this time, the analyst (person) inputs an arbitrary prompt into the LLM and engages in question-and-answer sessions, and modifies the prompt based on the results. This process is repeated until the correctness determination results are satisfactory.

[0023] The following describes specific examples of prompts and evaluations. Below is an example of wording generated as a prompt. The parts of the text annotated with ...(2), etc., are feature parts that have been found through iterative verification to yield a valid evaluation when such notation is included. These will be explained after the prompt example.

[0024] [Example prompt] #### Instructions You are an excellent psychological counselor. …(2) Summarize the content of the following call text and determine whether the customer had a positive or negative feeling towards the operator at the end of the call. Please determine the customer's emotions solely from their utterances. …(3) The call is likely to be a complaint, and since Japanese customers rarely express positive emotions, it is important to carefully look for positive elements. …(4) Pay attention to the emotions expressed towards the end of the conversation. (5) Create a summary of at least the customer's statements from the conversation, and then make a judgment based on that summary. The output results will be used for analysis, so please ensure the output format conforms to {output format}. In particular, please consider the following characteristics of Japanese people when judging emotions: …(6) 1. Interpreting Ambiguous Expressions: Determine from the context whether the customer's ambiguous expressions, such as "I understand" or "I'll think about it," convey a positive or negative emotion. 2. Consider the culture of harmony: When customers show a superficially positive reaction to avoid conflict, try to discern the true feelings behind it. 3. Compensate for the lack of facial expressions and body language: Since facial expressions and body language cannot be seen over the phone, pay attention to the choice of words and tone of voice to judge emotions. 4. Interpretation of indirect expressions: Determine from the context whether the customer's indirect expressions such as "I'll think about it" or "I'll consider it" convey a positive or negative emotion. 5. Consider emotional suppression: If the customer is suppressing their emotions, try to determine the true emotions behind it. 6. Consider the spirit of hospitality: When a customer is behaving respectfully and courteously, try to discern the true feelings behind it. ####Output format • Determining whether something is positive or negative: • Reasons for the decision (a detailed explanation based on the points above) • Keywords in the speech that were emphasized in the judgment: …(8) ·Improvement measures:” #### Example 1 …(7) (Past evaluation data corresponding to Example 1) • Judgment: Positive Reason: The customer initially appeared confused and somewhat emotional, but ultimately said, "That was very helpful. Thank you," and seemed satisfied with the operator's response. Although the problem was not resolved, the operator's sincere explanation built trust. Considering the ambiguous expressions and emotional restraints typical of Japanese people, we determined that the final words of gratitude reflected their genuine feelings. Keywords: "That was very helpful," "I think I can do it," "I'll give it a try." • Improvement plan: In the future, we will increase the number of concrete suggestions for problem solving and strengthen training to meet customer expectations. #### Example 2 (Past evaluation data corresponding to Example 2) • Judgment: Negative Reason: At the end of the conversation, the customer said, "So I have to wait a little longer. Okay," indicating dissatisfaction that the problem had not been resolved. Furthermore, the customer's "I understand" seemed more like reluctant acceptance than genuine understanding. The customer was still troubled by their inability to use the electronic payment service and felt that the operator's guidance was not providing the solution they expected. Considering the indirect expressions and emotional restraint characteristic of Japanese people, we determined that their superficial acceptance indicated underlying dissatisfaction. Keywords: "That's not worth discussing," "Do you really understand?", "It's all your business." • Improvement suggestion: Operators should provide more specific information and next steps to resolve customer issues. For example, providing more detailed information such as how long the investigation will take or if there are any additional procedures the customer needs to take can alleviate customer anxiety. Rather than being an issue with individual operators, it seems that a more organizational operational environment is needed that promises follow-up towards problem resolution and prompt responses. #### Data Description "speaker": "CU" Customer's utterance "speaker": "OP" Operator's utterance ####Call assumption A call was received by the customer service call center of a business that offers a code image payment service, from a Japanese person. Because customers call with problems that couldn't be resolved through the FAQ site, the initial stages of their inquiries tend to be more negative. #### Call Text (Evaluation data (for this request))

[0025] As indicated in (2) above, the prompt should include information instructing the LLM to respond as a psychological counselor. This clarifies the LLM's role and is expected to yield more accurate judgments than mere syntactic analysis.

[0026] As indicated in (3) above, the prompt should include information instructing the LLM to make decisions based solely on the customer's utterances in the data being evaluated. This is expected to result in decisions that are more focused on the customer's emotions.

[0027] As indicated in (4) above, the prompt should include information for the LLM indicating that the customer's utterances in the data being evaluated are likely to be negative. This reduces the possibility of judgments being made based solely on the superficial wording.

[0028] As indicated in (5) above, the prompt should include information for the LLM indicating that the customer's utterances in the data being evaluated are likely to be negative. This reduces the possibility of judgments being made based solely on the superficial wording.

[0029] As indicated in (6) above, the prompt may include a note regarding customer attributes. Customer attributes include, for example, nationality. Nationality may be considered to be the same country as the destination of the application program that mediates the network service. In other words, a customer using an application program distributed in Japan may be considered Japanese. Alternatively, some method may be used to verify nationality. Nationality may also be interpreted as race. This is expected to lead to a more accurate judgment, because the information on which the LLM is primarily learned is not necessarily the country where the call center is located. For example, if an LLM primarily learned in the United States attempts to judge the emotions of a Japanese person, there is a concern that it may judge based on the emotional expressions of a typical American. In this embodiment, including this note can mitigate such concerns.

[0030] As indicated in (7) above, it is advisable to include past decision examples in the prompt. This will further improve the accuracy of LLM's decisions.

[0031] As indicated in (8) above, the prompt should include information instructing the system to output the keywords that were key to the decision. This allows for accurate verification of accuracy and other related tasks.

[0032] [Examples] The following shows an example of the prompt created by the processing of this embodiment and the judgment result by LLM. Note that the part of the prompt corresponding to the program rules has been omitted for simplification.

[0033] (Example 1) --------------------Prompt 1-------------------- “speaker”: “OP”, “text”: “Thank you for calling. This is AA from the ○○ Customer Support Center.” “speaker”: “CU”, “text”: “Good morning.” “speaker”: “OP”, “text”: “Good morning.” “speaker”: “CU”, “text”: “Um, well, before that,” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, "text": "Um, my account is locked and I can't use it right now, so I contacted you about it." “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “And then they said it was due to some trouble with XX (an affiliate of the electronic payment service).” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Well, I did contact XX and they did take care of it.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “So, um, another week then.” “speaker”: “CU”, “text”: “Ah, about that much.” “speaker”: “CU”, “text”: “I think your account will be deactivated, but you will receive another email. It will say something like, 'Please log in again.'” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “I’ve been waiting. It’s been about two weeks now.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “I can’t use XX yet.” “speaker”: “OP”, “text”: “I see. Since you have contacted us before, would it be alright if we could check your history and other relevant information?” “speaker”: “CU”, “text”: “Yes. Ah.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes. So, regarding the phone number you registered, is it correct that it is [phone number ****-****] as you entered earlier? Yes. Thank you. Now, could you please tell me your full name?” “speaker”: “CU”, “text”: “Yes. My name is BB.” “speaker”: “OP”, “text”: “This is BB-sama, correct?” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, "text": "Understood. I will check this information for you, so could you please wait for about 2 or 3 minutes? Yes. Please wait a moment." “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, "text": "BB-sama, I apologize for keeping you waiting for so long. Yes, I have just checked and it appears you previously made an inquiry." “speaker”: “CU”, “text”: “Yes, yes.” “speaker”: “OP”, "text": "Well, we are currently coordinating with the relevant department, and it seems that the investigation is taking some time, so we are very sorry, but yes. Please wait until we get back to you." “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Haa haa.” “speaker”: “CU”, “text”: “Oh, I understand.” “speaker”: “OP”, “text”: “Yes, I am very sorry, but thank you for your understanding.” “speaker”: “CU”, “text”: “So, please wait a little longer. Yes.” “speaker”: “OP”, “text”: “Yes. AA has guided you.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, "text": "Yes. Thank you for contacting us." ――――――――――――Judgment result 1―――――――――――― • Judgment: Negative The reason is that the customer ultimately said, "So, I have to wait a little longer. Okay," indicating dissatisfaction with being kept waiting without the problem being resolved. Their response of "I understand" also suggested reluctance rather than genuine acceptance. The customer is still troubled by their inability to use [product / service name] and feels that the operator's guidance has not provided the solution they expected. Keywords: "So, please wait a little longer. Yes." "Understood." "Unusable." • Improvement suggestion: Operators should provide more specific information and next steps to resolve customer issues. For example, providing more detailed information such as how long the investigation will take or if there are any additional procedures the customer needs to take can alleviate customer anxiety. Rather than being an issue with individual operators, it seems that what is needed is a more organizational operational environment that can promise follow-up towards problem resolution and prompt responses.

[0034] (Example 2) --------------------Prompt 2-------------------- “speaker”: “OP”, “text”: “Thank you for calling.” This is CC from the ○○ Customer Support desk. “speaker”: “CU”, “text”: “Oh, excuse me.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, I changed my phone number.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “And,” “speaker”: “CU”, “text”: “Um, when I tried to use XX, “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um.” “speaker”: “CU”, “text”: “Huh? What? Huh?” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Something about the password.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, well, I can't log in at all even when I enter my password.” “speaker”: “OP”, "text": "Yes, understood. By the way, has your phone number changed or is it the same?" “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “No.” “speaker”: “CU”, “text”: “It's the same.” “speaker”: “OP”, “text”: “Understood. I will now guide you through the operation process, so could you please open the app now?” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, yes. It's open now.” “speaker”: “OP”, “text”: “Thank you. So, is this the new registration and login screen?” “speaker”: “CU”, “text”: “Well, I'm doing a lot of different things right now.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “This is a screen showing that another device is logged in on my device.” “speaker”: “OP”, “text”: “Oh.” “speaker”: “OP”, “text”: “That’s good to hear. In that case, would you mind choosing the one you don’t have on hand?” “speaker”: “CU”, “text”: “Oh, I see. Yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Then you should receive an SMS message, so please open the URL in that SMS message.” “speaker”: “CU”, “text”: “Ah.” “speaker”: “CU”, “text”: “Here it is. Yes.” “speaker”: “OP”, “text”: “Once you open it, I think you can just put it in as is.” “speaker”: “CU”, “text”: “Ah.” “speaker”: “CU”, “text”: “Please wait a moment.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, oh my.” “speaker”: “CU”, “text”: “Ah, it just disappeared.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “URL is” “speaker”: “CU”, “text”: “So I just need to send it again then.” “speaker”: “OP”, “text”: “Yes, please.” “speaker”: “CU”, “text”: “Press,” “speaker”: “CU”, “text”: “Here it comes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “So, I just need to enter here. Log in.” “speaker”: “OP”, “text”: “Yes, please.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, it went in. It came out.” “speaker”: “OP”, “text”: “That’s good. Yes. That’s good.” “speaker”: “CU”, “text”: “Ah.” “speaker”: “CU”, “text”: “So, this is OK now, right?” “speaker”: “OP”, “text”: “Yes. It’s OK, no problem.” “speaker”: “CU”, “text”: “Oh, excuse me. Thank you.” “speaker”: “OP”, “text”: “No, not at all, thank you.” “speaker”: “CU”, “text”: “I keep pressing the opposite side, but no matter how many times I try, it doesn't work.” “speaker”: “OP”, “text”: “That’s right.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “That’s good to hear. You’re welcome. That’s great, thank you.” “speaker”: “CU”, “text”: “Thank you very much. Yes.” “speaker”: “OP”, “text”: “Yes. Excuse me.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Excuse me, could you please turn it off?” “speaker”: “OP”, “text”: “Yes. Understood. I will take my leave now.” “speaker”: “CU”, “text”: “Yes. I'm sorry. Yes. I apologize.” ――――――――――――Judgment result 2―――――――――――― • Judgment: Positive Reason: The customer initially appeared confused by the problem, but was able to resolve it by following the operator's instructions. Ultimately, they repeatedly said "thank you" and "I'm sorry," indicating their gratitude for the operator's assistance. They also seemed relieved that the problem had been resolved. Keywords: "Thank you," "Excuse me," "Oh, it's in," "So this is OK, right?" • Improvements: While the operator's response was excellent, there were many instances where the customer hesitated when initially explaining the problem. Asking questions that encourage the customer to elaborate would facilitate smoother communication. Additionally, explaining the operating procedures more clearly and specifically could reduce customer anxiety.

[0035] (Example 3) ----------------------------Prompt 3---------------------------- “speaker”: “OP”, “text”: “Thank you for calling. This is EE from the customer support desk of XX (company name of electronic payment service).” “speaker”: “CU”, “text”: “Oh, excuse me, um, well, let me check with the person in question.” “speaker”: “CU”, “text”: “Regarding the matter, um...” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “I sent my driver's license with a photo of my face several times.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, well, how should I put it, it wasn't accepted, or rather, it wasn't confirmed.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, "text": "It happened about three times." “speaker”: “OP”, “text”: “So, is it a situation where the review process won't be successful?” “speaker”: “CU”, “text”: “Um, about the screening process, well, I have a gold card.” “speaker”: “CU”, “text”: “Um…” “speaker”: “OP”, “text”: “It seems that the identity verification process is not progressing smoothly.” “speaker”: “CU”, “text”: “Ah.” “speaker”: “CU”, “text”: “Oh yes, yes yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “So, first, I would like to identify the account.” “speaker”: “OP”, "text": "Um, regarding the phone number you use for XX, is it correct that it is ****-****?" “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Well, that’s definitely true.” “speaker”: “OP”, “text”: “Yes. Thank you.” “speaker”: “OP”, “text”: “Excuse me, but are you the customer on the phone who is actually the user of [service name]?” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes. Thank you.” “speaker”: “OP”, “text”: “Please provide your full name.” “speaker”: “CU”, “text”: “Um, FF.” “speaker”: “OP”, “text”: “It's FF-sama, isn't it?” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes, thank you. Would it be possible for you to open the app now?” “speaker”: “CU”, “text”: “Um, is it okay to open this? Yes. Yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Um, once you open it, you will see an Account button in the lower right corner of the home screen, so could you please press that?” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, please wait a moment.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um…” “speaker”: “CU”, “text”: “Yes. It opened.” “speaker”: “OP”, “text”: “Thank you.” “speaker”: “OP”, “text”: “Yes. If you open your account, scroll all the way down.” “speaker”: “OP”, “text”: “A user ID is listed.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes. Um, could you please tell me the first five digits from the left?” “speaker”: “CU”, “text”: “Please wait a moment.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, the first five digits from the left are um, 47.” “speaker”: “OP”, “text”: “Ah.” “speaker”: “OP”, “text”: “47. Um, please give me the first five digits from the left.” “speaker”: “CU”, “text”: “Hello?” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, the leftmost one?” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Oh, I'm sorry, please wait a moment.” “speaker”: “CU”, “text”: “Hello.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, 08.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “C” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “4 kana” “speaker”: “OP”, “text”: “0 Yes.” “speaker”: “CU”, “text”: “C04” “speaker”: “OP”, “text”: “Yes. It's alright, yes, we've confirmed it. Thank you for your cooperation. We have now looked into the account you reported, and we have found that “speaker”: “CU”, “text”: “No.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “The address you entered when applying for identity verification,” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, "text": "We were unable to verify your identity because the address on your identification document was incorrect." “speaker”: “OP”, “text”: “Well, a common issue is when you enter your address. For example.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “When there was an address like 1-chome 2-ban 3-go,” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Is it written as 1-2-3 with a hyphen, or as 1-chome 2-3?” “speaker”: “OP”, “text”: “Does it say 1-1-2-3?” “speaker”: “CU”, “text”: “Yes.” speaker: “OP”, “text”: “You need to do it exactly as written in the document.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes. And then,” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Um, the building name, and the room number, etc.” “speaker”: “OP”, “text”: “For example, if the room number is something like XX-XX, is it written as XX-XX or XX-XX, and are the documents finished?” “speaker”: “CU”, “text”: “Yes.” “OP”, “text”: “Please enter the building name exactly as it appears on the document, as indicated on your driver's license, since you have submitted this application. Please enter it exactly as it appears on your driver's license.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “CU”, “text”: “Ah, just as it says on my driver's license,” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Um, you see...” “speaker”: “CU”, “text”: “Um, I received a gold card from XX Card, and it was sent to that address.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “CU”, “text”: “Well, in the description, I've included 'Flat A,' but that doesn't appear on the driver's license itself, so it's not included.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Please enter the information exactly as it appears on your driver's license. This is strictly for use within this app.” “speaker”: “CU”, “text”: “Oh, really?” “speaker”: “CU”, “text”: “Oh. Oh, I see. I understand. I guess that’s what I got wrong.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “That is highly likely, so I would like to ask you to please go through the procedure again, even though it is a great inconvenience.” “speaker”: “CU”, “text”: “I understand.” “speaker”: “OP”, “text”: “Yes. So, that concludes today’s presentation. This has been EE.” “speaker”: “CU”, “text”: “Yes, I’m sorry.” “speaker”: “CU”, “text”: “Yes. Thank you.” “speaker”: “OP”, “text”: “Yes.” “speaker”: “OP”, “text”: “Thank you for calling.” “speaker”: “CU”, “text”: “Yes.” “speaker”: “OP”, “text”: “It appears the call has not been disconnected, customer.” “speaker”: “OP”, “text”: “It appears the call hasn't been disconnected.” “speaker”: “OP”, “text”: “Since we haven't received a reply, we will leave this call. We apologize for the inconvenience. Goodbye.” ――――――――――――Judgment result 3―――――――――――― • Judgment: Positive Reason: The customer was initially confused by the issue of their identity verification documents not being accepted, but after the operator's clear explanation, they understood the cause of the problem and seemed satisfied, saying "I understand." Ultimately, they expressed their gratitude with "Yes, thank you," indicating a positive feeling towards the operator's service. Key words in the speech that were considered important for the decision: "I understand," "Thank you." • Improvement Measures: While the response this time was generally good, the customer's comment at the end, "It's a hassle to have to take photos again," suggests that dissatisfaction with the cumbersome procedure may remain. In the future, simplifying the procedure and providing clearer guidance could reduce the burden on customers. Additionally, strengthening the procedure to confirm that the customer hangs up the phone at the end of the call should also be considered.

[0036] The following describes the effects and usefulness of this embodiment. Conventionally, when attempting to conduct a survey on the response of this type of call center, messages were mainly sent via SMS (Short Message Service) using the phone number, or via email if the email address was known. However, the response rate relative to the total number of calls was very low, making it difficult to verify whether the operator's response was good or not. Figure 3 shows an example of the conventional survey response situation. Due to the low ratio relative to the total number of calls, the number of responses per type (consultation content) and the number of responses per operator were also very low, making sufficient verification increasingly difficult.

[0037] In contrast, the information processing device of this embodiment can evaluate customer emotions solely based on the content of conversations. Theoretically, the number of conversations becomes the same as the number of replies, dramatically increasing the amount of data available for verification. Figure 4 shows a comparison of the results obtained from a verification experiment of this embodiment with the conventional method. According to this embodiment, it was found that correct evaluation results can be obtained with an accuracy rate of over 80%. Here, "correct" refers to, for example, a response that has been verified and labeled by a human. As shown in the figure, although the reliability is lower compared to customer emotions directly indicated by the survey results, the sample size is dramatically increased, making it possible to provide hints for business improvement.

[0038] According to the embodiments described above, customer emotions in a call center can be suitably evaluated.

[0039] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0040] 10 customers 50 Call Centers 60 Operators 100 Information Processing Devices 110 Acquisition Department 120 Evaluation Department 130 Output section 150 Storage section

Claims

1. An acquisition unit that acquires text data showing the content of conversations between customers and operators in a call center, An evaluation unit evaluates the customer's emotions by inputting evaluation data based on the aforementioned text data and a prompt requesting the customer to select from multiple candidates that categorize the customer's emotions into a Large Language Model (LLM), and obtaining the output of the Large Language Model. An output unit that outputs the processing result of the evaluation unit, An information processing device equipped with the following features.

2. The aforementioned multiple candidates include at least two: positive and negative. The information processing apparatus according to claim 1.

3. The system further includes a preprocessing unit that removes unnecessary information from the text data to generate the data to be evaluated. The information processing apparatus according to claim 1 or 2.

4. The prompt includes information instructing the large-scale language model to respond as a psychological counselor. The information processing apparatus according to claim 1 or 2.

5. The prompt includes requesting the user to select from a list of candidates that categorize the customer's emotions, focusing solely on the customer's utterances in the data being evaluated. The information processing apparatus according to claim 1 or 2.

6. The prompt includes information indicating that the customer's utterance in the data being evaluated is likely to be negative. The information processing apparatus according to claim 2.

7. The prompt includes information instructing the system to summarize the data to be evaluated and then make a decision based on the summary. The information processing apparatus according to claim 1 or 2.

8. The prompt includes a note regarding the customer's attributes, The information processing apparatus according to claim 1 or 2.

9. The aforementioned customer attribute is nationality. The information processing apparatus according to claim 8.

10. The destination of the application program for mediating network services related to the aforementioned call center, The information processing apparatus according to claim 8.

11. The aforementioned prompt includes past decision cases, The information processing apparatus according to claim 1 or 2.

12. The prompt includes information instructing the system to output keywords that were key to the decision. The information processing apparatus according to claim 1 or 2.

13. Information processing device, A process to obtain text data showing the content of conversations between customers and operators in a call center, The process involves inputting evaluation data based on the aforementioned text data and a prompt requesting the customer to select from multiple candidates that categorize the customer's emotions into a Large Language Model (LLM), and obtaining the output of the Large Language Model to evaluate the customer's emotions. A process to output the results of the evaluation process, An information processing method that performs the following.

14. On the computer, A process to obtain text data showing the content of conversations between customers and operators in a call center, The process involves inputting evaluation data based on the aforementioned text data and a prompt requesting the customer to select from multiple candidates that categorize the customer's emotions into a Large Language Model (LLM), and obtaining the output of the Large Language Model to evaluate the customer's emotions. A process to output the results of the evaluation process, A program to execute.

Images