Reception support device, reception support method, and program

Abstract

To provide a technique enabling more appropriate reception in consideration of the emotion of a client serving as a reception partner, the transition of the emotion, and an utterance status or utterance contents of a responder during reception for an utterance of the client.SOLUTION: When an information processing device supports the reception of a responder for a client while the responder is conversing with the client, a reception support device acquires information representing emotion series of the client during a reception period and determines the transition of the emotion of the client on the basis of the acquired emotion series. The reception support device also acquires each of utterance voice data uttered by the client and reception voice data uttered by the responder, recognizes a reception status or reception contents of the responder to the utterance of the client on the basis of each of the acquired voice data, and determines the acceptability. In addition, the reception support device generates reception support information for the responder and presents the information to the responder on the basis of determination results of the transition of the emotion of the client and determination results of the acceptability of an utterance status and the reception contents.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] One aspect of the present invention relates to a response support device, method, and program used to support an operator's response to a customer in, for example, a call center. [Background technology]

[0002] In recent years, the number of ways for companies to communicate with their customers has increased, and customer satisfaction and corporate image are becoming more and more important. In order to improve customer satisfaction, it is necessary to improve the quality of customer service at call centers and storefronts where customers' requests and opinions can be heard directly.

[0003] The likeability rating of call center operators and store clerks is one of the important factors for improving customer satisfaction, etc. If the likeability rating can be estimated, it can be used to evaluate the service quality of operators and store clerks, search for good examples, and educate operators and store clerks, and can be applied to improve the quality of service at call centers and store clerks.

[0004] Therefore, a technology has been proposed in which features are generated from a speech voice, and the likeability rating of the speaker of the speech voice is estimated from the features based on the correlation between the features of the generated speech voice and the speaker's likeability rating (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6859283 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 is useful as a means of advance speech training because it can estimate the likeability of a speaker who has uttered an utterance by taking into account not only the acoustic features of the utterance but also features such as the way in which interjections are used, the use of honorific language, and the clarity of sentence endings.

[0007] However, in actual customer service work at call centers, etc., the emotions of the customers they are dealing with vary widely, and their emotions often change during the service. For this reason, it is expected that operators will not be able to respond appropriately to unexpected customer emotions or changes in customer emotions during service if they are only given speech training in advance based on the way customers generally speak.

[0008] Also, in order to respond smoothly in actual customer service work, it has been pointed out that it is necessary to consider not only the customer's feelings but also the speech situation when the operator responds to the customer's utterance, for example, whether the speed of speech is too fast, whether the operator is talking one-sidedly, whether the operator is talking while the customer is speaking, etc. Furthermore, it is necessary to consider the content of the operator's response to the content of the customer's utterance. However, at present, measures to address these issues have not yet been adequately taken.

[0009] This invention has been made in light of the above-mentioned circumstances, and its main objective is to provide technology that enables a more appropriate response by taking into consideration the emotions and changes in the emotions of the person being spoken to, as well as the speech situation or content of the speech at the time the person being spoken to responds to the speech of the person being spoken to. [Means for solving the problem]

[0010] In order to solve the above problems, a first aspect of the response support device or response support method according to the present invention is to support the service of the service recipient by an information processing device when the service recipient is talking to the service recipient, by acquiring information representing an emotion sequence of the service recipient during the service period, and judging a transition of the emotion of the service recipient based on the acquired emotion sequence. In addition, speech voice data uttered by the service recipient and response voice data uttered by the service recipient are acquired, and based on the acquired speech voice data, the speech situation of the service recipient and the response situation of the service recipient are recognized, and the appropriateness of the service recipient's response situation with respect to the speech situation of the service recipient is judged. Then, based on the judgment result of the transition of the service recipient's emotion and the judgment result of the appropriateness, response support information for the service recipient is generated, and the generated response support information is presented to the service recipient.

[0011] According to the first aspect of the present invention, during a period when a person serves a person, the emotion sequence of the person served and the relevance between the speech situation of the person served and the person served are comprehensively considered to generate and present the person served with support information. Therefore, for example, when the person served with respect to the speech situation of the person served is inappropriate and the emotion sequence of the person served shifts to, for example, a negative emotion, appropriate support can be provided.

[0012] In order to solve the above problem, a second aspect of the response support device or response support method according to the present invention is to support the service of the service recipient by an information processing device during the service period, obtain information representing the emotional sequence of the service recipient during the service period, and determine the transition of the emotion of the service recipient based on the obtained emotional sequence. In addition, the service recipient's speech voice data and the response voice data uttered by the service recipient following the service recipient's speech are obtained, respectively, and the speech content of the service recipient and the response content of the service recipient are recognized from the obtained speech voice data and the response voice data, respectively, and the appropriateness of the response content of the service recipient to the speech content of the service recipient is determined. Then, response support information for the service recipient is generated based on the judgment result of the transition of the service recipient's emotion and the judgment result of the appropriateness, and the generated response support information is presented to the service recipient.

[0013] According to the second aspect of the present invention, while the person in charge is responding to the person being served, the emotion sequence of the person being served and the appropriateness of the person's response to the content of the person's speech are taken into consideration comprehensively to generate response support information and present it to the person in charge. Therefore, for example, when the person in charge does not respond appropriately to the content of the person's speech and the emotion sequence of the person being served shifts to, for example, a negative emotion as a result, appropriate response support can be provided.

[0014] Further, in order to solve the above problems, a third aspect of the response support device or response support method according to the present invention, when using an information processing device to support a response of a person to a person when the person is talking to the person, acquires information representing an emotion sequence of the person during a response period, and judges a transition of the emotion of the person based on the acquired emotion sequence. Also, during the response period, speech voice data uttered by the person and response voice data uttered by the person are acquired, and based on the acquired speech voice data, the speech situation of the person to be served and the response situation of the person to be served are recognized, and the appropriateness of the response situation of the person to the speech situation of the person to be served is judged. Furthermore, during the service period, speech voice data of the person served and response voice data uttered by the person served following the speech of the person served are respectively acquired, the speech content of the person served and the response content of the person served are respectively recognized from the acquired speech voice data and response voice data, and the appropriateness of the response content of the person served with respect to the recognized speech content of the person served is judged. Then, response support information for the person served is generated based on the judgment result of the transition of the person served's emotions, the judgment result of the appropriateness, and the judgment result of the appropriateness of the response content, and the generated response support information is presented to the person served.

[0015] According to the third aspect of the present invention, during a period when a person is serving a person, the emotion sequence of the person served, the relevance between the speech situation of the person served and the service situation of the person served, and the appropriateness of the service content of the person served to the content of the speech of the person served are comprehensively considered to generate service support information and present it to the person served. Therefore, for example, when the service situation of the person served to the speech situation of the person served is inappropriate and the emotion sequence of the person served shifts to a negative emotion due to the influence, or when the person served does not respond appropriately to the content of the speech of the person served and the emotion sequence of the person served shifts to a negative emotion due to the influence, it is possible to provide appropriate service support. It is possible to provide appropriate service support. Effect of the Invention

[0016] In other words, according to each aspect of the present invention, it is possible to provide technology that enables a more appropriate response by taking into consideration the emotions and changes in the emotions of the person being spoken to, as well as the speech situation or content of the speech when the person responds to the speech of the person being spoken to. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of a system including a response support device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a response support device according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing an example of a software configuration of the response support device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing an example of a processing procedure and processing contents of the utterance situation determination processing executed by the control unit of the response support device shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing an example of a processing procedure and processing contents of an utterance content determination process executed by the control unit of the response support device shown in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of a processing procedure and processing contents of the compound determination processing executed by the control unit of the response support device shown in FIG. [Figure 7] FIG. 7 is a sequence diagram showing an operation procedure of the system shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a display of a composite advice message. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] [One embodiment] (Configuration example) (1) System FIG. 1 is a diagram showing an example of the configuration of a system including a response support device according to an embodiment of the present invention.

[0020] The system according to one embodiment of the present invention is equipped with a response support device SV1, which monitors calls made via a network NW between customer terminals UT1 to UTn used by customers and operator terminals MT1 to MTk used by call center operators, and supports the operators in responding to customers based on the results of the calls.

[0021] The network NW includes a wide area network with the Internet at its core, and an access network for accessing the wide area network. The access network may be, for example, a public data communication network using wired or wireless communication, a local area network (LAN) using wired or wireless communication, or a cable television (CATV) network.

[0022] The system according to one embodiment also includes an emotion analysis server SV2. The emotion analysis server SV analyzes the emotion of the customer based on the customer's speech data while the operator is talking to the customer, and provides the obtained analysis data of the customer's emotion to the response support device SV1 for the response support process.

[0023] The customer terminals UT1 to UTn are, for example, smartphones, and have a function of performing voice communication with other terminals including the operator terminals MT1 to MTk using a communication protocol defined in the network NW. Note that the customer terminals UT1 to UTn are not limited to smartphones, and any terminals having a similar voice communication function can be used.

[0024] The operator terminals MT1 to MTk are, for example, personal computers to which headsets HS1 to HSk are connected. By using these headsets HS1 to HSk, the operators can carry out voice communication with the customers using the customer terminals UT1 to UTn.

[0025] (2) Response Support Device SV1 The response support device SV1 is configured, for example, by a server computer located on the web or in the cloud.

[0026] 2 and 3 are block diagrams showing an example of a hardware configuration and a software configuration, respectively, of the response support device SV1.

[0027] The response support device SV1 includes a control unit 1 using a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and a communication interface (hereinafter, the interface will be abbreviated as I / F) unit 4 are connected to the control unit 1 via a bus 5.

[0028] The communication I / F unit 4 operates in accordance with a communication protocol defined by the network NW under the control of the control unit 1. The communication I / F unit 4 acquires voice communication data transmitted and received between the customer terminals UT1-UTn and the operator terminals MT1-MTk, and transmits information to the operator terminals MT1-MTk to support the operators in responding to calls.

[0029] The program storage unit 2 is configured by combining a non-volatile memory such as a solid state drive (SSD) that can be written and read at any time as a storage medium, and a non-volatile memory such as a read only memory (ROM), and stores application programs required for executing various control processes according to an embodiment, in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

[0030] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium, and a volatile memory such as a RAM (Random Access Memory), and its memory area includes a voice data storage unit 31, a text data storage unit 32, an emotion support information storage unit 33, an utterance situation support information storage unit 34, and an utterance content support information storage unit 35.

[0031] The voice data storage unit 31 temporarily stores the speech voice data transmitted from the customer terminals UT1 to UTn and the operator terminals MT1 to MTk for voice mining processing.

[0032] The text data storage unit 32 stores the text data converted from the spoken voice data.

[0033] The emotion support information storage unit 33 stores an emotion advice message according to the determination result of the emotion of the customer.

[0034] The speech situation support information storage unit 34 stores a speech situation advice message according to the result of the judgment of the appropriateness of the speech situation of the agent with respect to the customer.

[0035] The utterance content assistance information storage unit 35 stores an utterance content advice message corresponding to the result of the judgment on the suitability of the utterance content of the operator with respect to the utterance content of the customer.

[0036] The control unit 1 includes a voice mining processing block 11, a customer service support back-end processing block 12, and a customer service support front-end processing block 13.

[0037] The voice mining processing block 11 has a function of processing telephone conversation voice, and includes a voice data acquisition processing unit 111 and a text data generation processing unit 112 .

[0038] The voice data acquisition processing unit 111 acquires speech data of the customer and the operator via the communication I / F unit 4 each time a call is made between the customer terminals UT1 to UTn and the operator terminals MT1 to MTk, and stores the acquired speech data of the customer and the operator in the voice data storage unit 31 in chronological order.

[0039] The text data generation processing unit 112 reads each of the customer and operator's spoken voice data from the voice data storage unit 31, converts it into text data, and stores each of the converted text data in the text data storage unit 32 with a timestamp indicating the time of speech.

[0040] The response support back-end processing block 12 includes an emotion analysis data acquisition processing unit 121 , an emotion determination processing unit 122 , an utterance situation determination processing unit 123 , an utterance content determination processing unit 124 , and a compound determination processing unit 125 .

[0041] The emotion analysis data acquisition processing unit 121 reads out the customer's speech voice data from the voice data storage unit 31 every time the call is made, and transfers it to the emotion analysis server SV2 from the communication I / F unit 4. Then, the emotion analysis data acquisition processing unit 121 acquires the emotion analysis data of the customer returned from the emotion analysis server SV2 via the communication I / F unit 4.

[0042] The emotion determination processing unit 122 receives the analysis data of the customer's emotion from the emotion analysis data acquisition processing unit 121, and determines whether the customer's emotion is a negative emotion or not based on the received analysis data. If the emotion determination processing unit 122 determines that the customer's emotion is a negative emotion, it selectively reads out an emotion advice message according to the type of the negative emotion from the emotion support information storage unit 33, and sends the read emotion advice message together with the emotion determination result to the compound determination processing unit 125.

[0043] The speech situation determination processing unit 123 reads text data obtained by converting the speech voices of the customer and the operator from the text data storage unit 32 each time a call is made between the operator and the customer. Then, based on the text data read, it determines whether the speech situation of the operator when responding to the customer's speech situation is appropriate or not. Examples of the speech situation include "speech speed," "speech ratio," and "number of overlaps" of speech.

[0044] Furthermore, the utterance situation determination processing unit 123 reads out an utterance situation advice message corresponding to the type of the utterance situation from the utterance situation support information storage unit 34 according to the judgment result of the suitability. Then, the utterance situation advice message thus read out is sent to the compound judgment processing unit 125 together with the judgment result of the utterance situation and the identification information of the operator to be supported. An example of the utterance situation judgment processing will be described in the operation example.

[0045] The speech content determination processing unit 124 reads text data obtained by converting the speech voices of the customer and the operator from the text data storage unit 32 each time a call is made between the operator and the customer, and recognizes the speech content of the customer and the response content of the operator based on the text data read.

[0046] Furthermore, the utterance content determination processing unit 124 determines whether the operator's response to the customer's utterance is appropriate, reads out an utterance content advice message according to the result of the suitability determination from the utterance content support information storage unit 35, and sends this utterance content advice message together with the result of the suitability determination of the above-mentioned response to the compound determination processing unit 125. Note that an example of the process of determining the suitability of the above-mentioned response will be described in the operation example.

[0047] The compound determination processing unit 125 receives the respective determination results and advice messages from the emotion determination processing unit 122, the utterance situation determination processing unit 123, and the utterance content determination processing unit 124. Then, based on the received determination results and advice messages, the compound determination processing unit 125 generates response support information by combining these, and outputs the generated response support information to the support information transmission processing unit 131 of the response support front-end processing block 13 together with identification information of the operator or operator terminal to be supported.

[0048] The support information transmission processing unit 131 transmits the response support information from the communication I / F unit 4 to the operator terminals MT1 to MTk used by the operators to be supported.

[0049] Each of the processing units 111, 112, 121 to 125, and 131 of the control unit 1 is realized by causing a hardware processor of the control unit 1 to execute an application program stored in the program storage unit 2. A part or all of the processing units 111, 112, 121 to 125, and 131 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0050] (Example of operation) Next, an example of the operation of the response support device SV1 configured as above will be described.

[0051] FIG. 7 is a diagram showing an example of a transmission / reception sequence of information data between the response support device SV1, the emotion analysis server SV2, the customer terminals UT1 to UTn, and the operator terminals MT1 to MTk.

[0052] (1) Voice mining processing For example, assume that a call has just started between the customer terminal UT1 and the operator terminal MT1. At this time, under the control of the voice data acquisition processing unit 111 of the voice mining processing block 11, the control unit 1 of the response support device SV1 acquires the customer's speech voice data transmitted from the customer terminal UT1 and the operator's response voice data transmitted from the operator terminal MT1 via the communication I / F unit 4. Then, the acquired speech voice data and response voice data are temporarily stored in the voice data storage unit 31.

[0053] When the customer's speech data and the operator's response voice data are acquired, the voice mining processing block 11 reads the customer's speech data and the operator's response voice data from the voice data storage unit 31 in predetermined units, and converts the read customer's speech data and the operator's response voice data into text data, respectively, using the text data generation processing unit 112. The converted text data of the voices of the customer and the operator are then stored in the text data storage unit 32 with a timestamp indicating the identification information of the customer and the operator and the time of speech, respectively.

[0054] (2) Acquiring customer sentiment analysis data When the call is started, the voice mining processing block 11 notifies the customer service support back-end processing block 12 of a call start event. When the customer service support back-end processing block 12 receives the call start event, under the control of the emotion analysis data acquisition processing unit 121, the customer service support back-end processing block 12 first sends a voice data / text data acquisition request to the voice mining processing block 11. In response to the voice data / text data acquisition request, the voice mining processing block 11 reads out the customer's spoken voice data from the voice data storage unit 31 and returns it to the emotion analysis data acquisition processing unit 121 of the customer service support back-end processing block 12.

[0055] The emotion analysis data acquisition processing unit 121 transmits the received speech voice data of the customer together with an emotion analysis request from the communication I / F unit 4 to the emotion analysis server SV2. Upon receiving the emotion analysis request, the emotion analysis server SV2 analyzes the emotion sequence of the customer at that time from the speech voice data transmitted together with the analysis request.

[0056] The following method, for example, is used as a method for analyzing emotion sequences. That is, the emotion analysis server SV2 first calculates a voice feature vector as a feature of the emotion sequence of the customer from the customer's speech voice data. The voice feature vector is calculated, for example, by cepstrum analysis, which is often used as a voice analysis method. More specifically, the voice feature vector is calculated as a vector of about 10 to 100 dimensions that combines MFCC (Mel Frequency Cepstral Coefficient), ΔMFCC, ΔΔMFCC, logarithmic power, Δlogarithmic power, etc.

[0057] The emotion analysis server SV2 then compares the calculated voice feature vector with the emotion data set, selects the emotion data showing the highest likelihood as the emotion of the customer, and returns the selected emotion data of the customer to the customer support device SV1 as emotion analysis data.

[0058] The calculation of the likelihood is described in detail in, for example, the following document. References: Kiyohiro Shikano, Katsunori Ito, Tatsuya Kawahara, Kazuya Takeda, Mikio Yamamoto, "IT Text Speech Recognition System", pp.1-51, 2001, Ohmsha.

[0059] The response support device SV1 receives the customer's emotion analysis data returned from the emotion analysis server SV2 via the communication I / F unit 4 under the control of the emotion analysis data acquisition processing unit 121, and notifies the emotion determination processing unit 122 of the received customer's emotion analysis data.

[0060] (3) Judging the customer’s emotions When the emotion determination processing unit 122 receives the emotion analysis data of the customer from the emotion analysis data acquisition processing unit 121, the emotion determination processing unit 122 determines whether the emotion of the customer is a negative emotion based on the emotion analysis data. Examples of negative emotions include "anxiety" and "anger."

[0061] When the emotion determination processing unit 122 determines that the emotion of the customer is a negative emotion, it then reads out an emotion advice message corresponding to the type of the negative emotion from the emotion support information storage unit 33. Then, it notifies the composite determination processing unit 125 of the read emotion advice message together with the emotion determination result.

[0062] (4) Determining the operator's response to the customer's speech The response support back-end processing block 12 of the response support device SV1, under the control of the speech situation determination processing unit 123, executes a process of determining the speech situation of the operator when responding to the speech situation of the customer as follows.

[0063] FIG. 4 is a flowchart showing an example of the procedure and content of the utterance situation determination process executed by the utterance situation determination processor 123. As shown in FIG.

[0064] (3-1) Determining speech rate That is, in steps S10 and S11, the speech situation determination processing unit 123 first reads text data of the customer's speech and text data of the operator's speech when interacting from the voice data storage unit 31, respectively. Next, in step S12, the speech situation determination processing unit 123 calculates the speech rates (speech speed) of the customer and the operator based on the above-mentioned text data that has been read. Then, in step S13, the speech situation determination processing unit 123 compares the speech rate of the operator with that of the customer and determines whether or not the speech rate of the operator is faster than that of the customer by a preset threshold value or more, in other words, whether or not the operator speaks "fast."

[0065] When the speech situation determination processing unit 123 determines that the speech rate of the operator is "fast", in step S14, it reads out an utterance situation advice message corresponding to "fast" from the utterance situation support information storage unit 34. Then, it notifies the composite determination processing unit 125 of the read out utterance situation advice message together with the speech situation determination result and the identification information of the operator or operator terminal to be supported.

[0066] (3-2) Determining speech ratio Next, in step S15, the speech situation determination processing unit 123 calculates the speech times of the customer and the operator based on the text data corresponding to the speech voice data of each of the customer and the operator. Then, the speech situation determination processing unit 123 calculates the ratio between the calculated speech time of the customer and the speech time of the operator. Then, in step S16, the speech ratio of the operator is compared with a preset threshold value, and if the speech ratio of the operator is equal to or greater than the threshold value, the operator at that time is determined to be "talking too much."

[0067] When the utterance situation determination processing unit 123 determines that the operator is "talking too much" in the above determination, in step S17, it reads out an utterance situation advice message corresponding to "talking too much" from the utterance situation support information storage unit 34. Then, it notifies the composite determination processing unit 125 of the read out utterance situation advice message together with the utterance situation determination result and the identification information of the operator or operator terminal to be supported.

[0068] (3-3) Determining the number of overlapping utterances Next, in step S18, the speech situation determination processing unit 123 counts the number of times that the timing of the speech of the operator when answering the call overlaps with the speech of the customer based on the time stamp indicating the speech time added to the text data of each of the speech voice data of the customer and the operator. Then, in step S19, it is determined whether the counted number of speech overlaps is greater than a preset threshold value. If the number of speech overlaps is greater than the threshold value, the speech situation determination processing unit 123 determines that the speech of the operator when answering the call "overlaps" with the speech of the customer too much.

[0069] When the utterance situation determination processing unit 123 determines that the operator is "overlapping" in the above determination, in step S20, it reads out an utterance situation advice message corresponding to "overlapping" from the utterance situation support information storage unit 34. Then, it notifies the composite determination processing unit 125 of the read out utterance situation advice message together with the utterance situation determination result and the identification information of the operator or operator terminal to be supported.

[0070] (4) Judging the appropriateness of the content of the speech The response support back-end processing block 12 of the response support device SV1 then, under the control of the utterance content determination processing unit 124, determines whether the utterance content of the operator in response to the utterance content of the customer is appropriate or not as follows.

[0071] FIG. 5 is a flowchart showing an example of the processing procedure and processing contents of the utterance content determination processing executed by the utterance content determination processing unit 124.

[0072] (4-1) When a customer's statement contains a specific term First, in step S30, the speech content determination processing unit 124 reads the text data of the customer's speech from the text data storage unit 32. Then, in step S31, it determines whether or not the read text data of the customer's speech contains a "specific term." For example, if the call is when the customer is explaining an accident, it determines whether or not the text data contains a term that describes the situation at the time, such as "driving" or "truck."

[0073] As a result of the above determination, it is assumed that the text data contains the "specific term." In this case, in step S32, the utterance content determination processing unit 124 reads out message template data from the utterance content support information storage unit 35. Then, using the read out template data, it generates an utterance content advice message for notifying the operator that the customer's utterance contains the "specific term."

[0074] For example, a speech content advice message may be generated such as "Your utterance contains the terms 'driving' and 'truck'."

[0075] Then, the utterance content determination processing unit 124 notifies the compound determination processing unit 125 of the generated utterance content advice message together with the identification information of the agent to be assisted.

[0076] (4-2) When it is assumed that the operator did not pay attention to important terms contained in the customer's remarks The speech content determination processing unit 124 determines whether or not the text data of the customer's speech contains a preset "important term." For example, it determines whether or not the speech data when the customer is explaining the accident contains important terms that may affect the customer's physical condition, such as "rear-end collision" or "crash," or terms that express the customer's mental state, such as "terrible," "I don't understand," or "what should I do."

[0077] As a result of the above-mentioned determination, it is assumed that the text data of the customer's speech contains the above-mentioned "important term." In this case, in step S34, the speech content determination processing unit 124 reads text data of the speech of the operator responding to the customer's comment from the text data storage unit 32. Then, in step S35, it determines whether or not the text data of the speech of the operator responding contains a "response term" corresponding to the above-mentioned "important term."

[0078] As a result of the above determination, it is assumed that the text data of the operator's speech does not contain a "response term" corresponding to the above "important term." In this case, the speech content determination processing unit 124 presumes that the operator may not have noticed that the customer's speech contains an "important term," and in step S36, reads out template data for an advice message from the speech content support information storage unit 35. Then, using the read-out template data, a speech content advice message is generated to notify the operator that the customer's speech contains the above "important term."

[0079] For example, generate a message like, "Your comment contains the term 'rear-end'. Take care of yourself!"

[0080] Then, the utterance content determination processing unit 124 notifies the compound determination processing unit 125 of the generated utterance content advice message together with the identification information of the operator or operator terminal to be assisted.

[0081] (5) Complex Judgments The response support backend processing block 12 of the response support device SV1, under the control of the composite judgment processing unit 125, executes a process of generating composite response support information by combining the judgment results of the emotion judgment processing unit 122, the speech situation judgment processing unit 123 and the speech content judgment processing unit 124 as follows.

[0082] FIG. 6 is a flowchart showing an example of the procedure and content of the compound judgment process executed by compound judgment processing unit 125.

[0083] (5-1) Combination of negative emotions and speech situations First, in steps S40, S41 and S42, compound judgment processor 125 acquires the emotion judgment result, the utterance situation judgment result and the utterance content judgment result from emotion judgment processor 122, the utterance situation judgment processor 123 and the utterance content judgment processor 124, respectively.

[0084] Next, in step S43, the compound determination processing unit 125 determines whether or not the acquired emotion determination result is a negative emotion. If it is determined to be a negative emotion, then in step S50, the compound determination processing unit 125 determines whether or not the speech situation determination result is inappropriate, that is, whether or not it indicates "talking too fast," "talking too much," or "over-talking." If it is determined to be inappropriate, then in step S51, the compound determination processing unit 125 generates a compound advice message that combines both the negative emotion and the speech situation determination result.

[0085] For example, if the operator's speech overlaps with the customer's speech three or more times during a call, and the customer's emotion is "anger," "The operator's speech overlaps too much with what the customer is saying, and we are detecting anger in the customer." This generates a composite advice message.

[0086] Then, the compound judgment processing unit 125 outputs the generated compound advice message to the support information transmission processing unit 131. The support information transmission processing unit 131 transmits the compound advice message from the communication I / F unit 4 to the operator terminal MT1 corresponding to the operator to be supported.

[0087] (5-2) Combination of negative emotions and speech content If the emotion determination result is a negative emotion and the utterance content determination result is inappropriate, in this case, the compound determination processing unit 125 determines the combination of the determination results in step S52, and then generates a compound advice message in step S53 by combining both the negative emotion and the utterance content determination result.

[0088] For example, if the operator says "It's tough" during a call and the customer is feeling "anxious," "The customer's comment contained the word 'difficult', which is making him / her feel 'anxious'." This generates a composite advice message.

[0089] Then, the compound judgment processing unit 125 outputs the generated compound advice message to the support information transmission processing unit 131. The support information transmission processing unit 131 transmits the compound advice message from the communication I / F unit 4 to the operator terminal MT1 corresponding to the operator to be supported.

[0090] (5-3) Negative emotions only It is assumed that the customer's emotion is determined to be negative, but the judgment results of the speech situation and the speech content are both satisfactory. In this case, when the compound judgment processing unit 125 judges the above judgment result in step S54, it generates an advice message for the above negative emotion in step S55.

[0091] The compound judgment processing unit 125 selects, for example, an emotion advice message generated in the emotion judgment processing unit 122 for conveying the customer's negative emotion to the operator. Then, the selected emotion advice message is output to the support information transmission processing unit 131. The support information transmission processing unit 131 transmits the emotion advice message from the communication I / F unit 4 to the operator terminal MT1 corresponding to the operator to be supported.

[0092] (5-4) When there is a problem only with the speech situation When it is determined in step S43 that the customer's emotion is not a negative emotion, the compound judgment processing unit 125 proceeds to step S44. Then, in step S44, the compound judgment processing unit 125 determines whether or not there is a problem with the utterance situation, and if there is a problem with the utterance situation, in step S45, generates an utterance situation advice message for notifying the operator of the problem with the utterance situation.

[0093] For example, the compound judgment processing unit 125 selects the speech situation advice message generated by the speech situation judgment processing unit 123 for informing the operator of the speech situation, such as "talking too fast," "talking too much," or "overlapping" in remarks.

[0094] Then, the compound judgment processing unit 125 outputs the selected utterance situation advice message to the support information transmission processing unit 131. The support information transmission processing unit 131 transmits the utterance situation advice message from the communication I / F unit 4 to the operator terminal MT1 corresponding to the operator to be supported.

[0095] (5-5) When there is a problem only with the content of the speech If the compound judgment processor 125 judges in step S43 that the emotion of the customer is not negative and judges in step S44 that there is no problem with the speech situation, the process proceeds to step S46. Then, in step S46, the compound judgment processor 125 judges whether there is a problem with the speech content, and if it judges that there is a problem with the speech content, in step S47, it generates an utterance content advice message for notifying the operator that the utterance content is inappropriate.

[0096] For example, the compound judgment processing unit 125 directly selects the speech content advice message generated by the speech content judgment processing unit 124 for notifying the operator that the customer's speech content contains, for example, a "specific term" or an "important term."

[0097] Then, the compound judgment processing unit 125 outputs the selected utterance content advice message to the support information transmission processing unit 131. The support information transmission processing unit 131 transmits the utterance content advice message from the communication I / F unit 4 to the operator terminal MT1 corresponding to the operator to be supported.

[0098] (5-6) Emotion + Speech Situation + Speech Content In the above explanation, we have described a case in which the priority of the operator's assistance with the customer's emotions is set to the highest, and if it is determined that there is a problem with the speech situation or speech content, a composite advice message of "emotion + speech situation" or a composite advice message of "emotion + speech content" is generated and sent.

[0099] However, other than that, for example, the priority of the speech situation and the speech content may be the same, and when it is determined that there is a problem with both, a composite advice message of "emotion + speech situation + speech content" may be generated and transmitted. In that case, the determination results of the emotion, speech situation, and speech content may be included in the composite advice message and transmitted, regardless of the presence or absence of a problem.

[0100] FIG. 8 shows an example of the composite advice message displayed on the operator terminal MT1 in this case. When such a composite advice message is displayed, the operator can check the result of the judgment of his / her own speech content against the customer's speech content together with the customer's emotions while talking to the customer, and can also check his / her own speech speed, speech ratio, and number of overlaps at the same time. Then, by referring to this advice message, the operator can correct his / her own way of handling the customer in real time while talking.

[0101] (effect) As described above, in one embodiment, in a call center, during a period when an operator is talking to a customer, speech data of the customer and the operator is acquired, and among them, the speech data of the customer is sent to the emotion analysis server SV2 to acquire analysis data of the customer's emotion, and it is judged whether the customer's emotion is negative or not. Also, based on the text data converted from each of the voice data of the customer and the operator, the speech speed, speech ratio, and number of overlapping speeches of the operator to the customer are calculated, and based on the calculation results, it is judged whether the speech situation of the operator is appropriate or not, and the appropriateness of the speech content of the operator's response to the content of the customer's speech is judged. Then, based on the judgment results of the emotion of the customer, the speech situation of the operator to the customer, and the speech content, a composite advice message for the operator is generated and transmitted to the operator terminal.

[0102] Therefore, for example, if the customer's emotion changes to a negative emotion while the operator is talking to the customer, an appropriate response support message that takes the customer's emotion change into consideration can be presented to the operator. As a result, if the customer's emotion changes to a negative emotion during a call, the operator can grasp the customer's emotion change in real time and respond appropriately.

[0103] At the same time, it is judged whether the speech situation and speech content of the operator in response to the customer's speech are appropriate or not, and if it is judged to be inappropriate, a composite advice message is generated by combining the advice message for the negative emotion with an advice message according to the judgment result of the speech situation or speech content, and is presented to the operator. Therefore, if the operator's own speech situation or speech content in response to the customer is inappropriate, the operator can confirm this fact together with the customer's negative emotion at that time, and this makes it possible to modify the operator's own speech situation or speech content in consideration of the customer's emotion. As a result, it becomes possible to respond more smoothly.

[0104] [Other embodiments] (1) In one embodiment, the speech situation determination process is described using an example in which the determination process is performed in the order of “speech rate,” “speech ratio,” and “number of overlaps.” However, the order of the above determination process is not limited to the above example and can be set arbitrarily.

[0105] In addition, in determining the "speech rate," it is possible to determine not only "too fast" but also "too slow," and further, in determining the "speech ratio," it is possible to determine not only "too much talking" but also "not enough speaking."

[0106] (2) In one embodiment, the process of determining the content of an utterance is described as an example in which both a message output process is executed when a “specific term” is detected and a response term is spoken or not when a “key term” is detected. However, only one of the two processes may be executed, and the types and number of “specific terms” and “key terms” to be detected can be arbitrarily set depending on the application, etc.

[0107] (3) In one embodiment, in the composite judgment process, the priority of the operator's support for the customer's emotion is set to the highest, and if it is judged that there is a problem with the speech situation or the speech content, a composite advice message of "emotion + speech situation", a composite advice message of "emotion + speech content", or a composite advice message of "emotion + speech situation + speech content" is generated and transmitted. However, the priority of the emotion, speech situation, and speech content can be arbitrarily set according to the purpose of use, etc.

[0108] (4) The type of customer emotion used in the composite judgment process is not limited to "negative emotion" and may be "positive emotion", and emotions including the state of mind may be judged and the result may be reflected in the composite judgment. For example, the numerical value of the emotion of joy and the numerical value of the emotion of distrust may be compared with thresholds to judge both emotions, and advice messages according to the judgment results may be generated and output.

[0109] (5) Depending on the application, only one of the three elements, emotion, speech situation, and speech content, may be determined, and an advice message corresponding to "emotion," an advice message corresponding to "speech situation," and an advice message corresponding to "speech content" may be generated and transmitted separately based on the determination result. Furthermore, any two of the above three elements may be determined, and a composite advice message may be generated and transmitted based on the determination result.

[0110] (6) In one embodiment, an example is given of the case where the response support device SV1 is installed on a server computer on the Web or in the cloud, but the functions of the response support device SV1 may also be provided in a terminal such as an operator terminal MT1 to MTk or an administrator terminal used by a system administrator.

[0111] (7) In one embodiment, analysis data on customer emotions is obtained from an emotion analysis server SV2 that is provided independently of the customer service support device SV1. However, the emotion analysis server function may be provided as one of the functions of the customer service support device SV1.

[0112] (8) The present invention can be used not only in call centers, but also in question and answer sessions between lecturers and participants in various seminars, and exchanges of opinions in meetings.

[0113] (9) The emotion analysis processing function, emotion determination processing function, speech situation determination processing function, and speech content determination processing function described in the embodiment may be realized using a machine learning model such as a support vector machine (SVM), a random forest, a neural network, etc. In addition, the functional configuration of the response support device, its processing procedure, processing content, etc. may be modified in various ways without departing from the gist of the present invention.

[0114] Although the embodiment of the present invention has been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiment may be appropriately adopted.

[0115] In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0116] SV1…Response support device SV2…Sentiment analysis server UT1~UTn…Customer terminal MT1~MTk…Operator terminal HS1~HSk…Headset NW…Network 1...Control unit 2...Program memory section 3. Data storage section 4. Communication I / F section 5. Bus 11…Audio mining processing block 111...Audio data acquisition processing unit 112...Text data generation processing unit 12…Support back-end processing block 121...emotion analysis data acquisition processing unit 122...Emotion determination processing unit 123... Speech situation determination processing unit 124... Speech content determination processing unit 125...Composite judgment processing unit 13…Support front-end processing block 131...Support information transmission processing unit 31...Audio data storage unit 32...Text data storage unit 33…Emotional support information storage unit 34... Speech situation support information storage unit 35... Speech content support information storage unit

Claims

1. A communication support device that supports a person who is being served with a conversation with the person who is being served, comprising: a first processing unit that acquires information representing an emotion sequence of the person being served during a service period, and determines a transition of the emotion of the person being served based on the acquired emotion sequence; a second processing unit that acquires, during the service period, speech voice data uttered by the person being served and response voice data uttered by the person serving, and recognizes the speech situation of the person being served based on the acquired speech voice data, and recognizes the speech situation of the person serving; a third processing unit that determines whether an utterance situation of the responder at the time of responding is appropriate for the recognized utterance situation of the respondee; a fourth processing unit that generates response support information for the person who is being served based on a result of determining a transition in the emotion of the person who is being served and a result of determining whether the utterance situation is appropriate, and presents the generated response support information to the person who is being served; A response support device comprising:

2. The second processing unit recognizes at least one of a speech rate, a speech time, and a speech overlap degree as the speech situation of the person being served and the speech situation of the person serving, and The third processing unit determines whether the speech situation is appropriate based on at least one of a difference in speech speed between the handled person and the responder, a difference in speech time between the handled person and the responder, and a speech overlap degree of the responder with respect to the handled person; the fourth processing unit generates the response support information when the emotion sequence includes a preset negative emotion or a preset positive emotion, and at least one of the speech rate difference, the speech time difference, and the speech overlap degree does not satisfy a preset condition. The response support device according to claim 1 .

3. A communication support device that supports a person who is being served with a conversation with the person who is being served, comprising: a first processing unit that acquires information representing an emotion sequence of the person being served during a service period, and determines a transition of the emotion of the person being served based on the acquired emotion sequence; a second processing unit that acquires, during the service period, speech voice data of the person served and response voice data uttered by the person served following the speech of the person served, and recognizes the speech content of the person served and the response content of the person served from the acquired speech voice data and response voice data, respectively; A third processing unit that determines whether the contents of the response by the respondent to the contents of the utterance by the respondent are appropriate or not; a fourth processing unit that generates response support information for the person serving based on a result of determining a transition in the emotion of the person served and a result of determining whether the content of the response is appropriate, and presents the generated response support information to the person serving; A response support device comprising:

4. The second processing unit recognizes a key term that is preset as the content of the utterance, The third processing unit determines whether an operator has noticed the important term based on the response voice data, The fourth processing unit generates the response support information when it is determined that the emotion sequence includes a preset negative emotion and the person in charge is not aware of the important term. The response support device according to claim 3.

5. A communication support device that supports a person who is being served with a conversation with the person who is being served, comprising: a first processing unit that acquires information representing an emotion sequence of the person being served during a service period, and determines a transition of the emotion of the person being served based on the acquired emotion sequence; a second processing unit that acquires, during the service period, speech voice data uttered by the person being served and response voice data uttered by the person serving, and recognizes the speech situation of the person being served based on the acquired speech voice data, and recognizes the speech situation of the person serving; a third processing unit that determines whether an utterance situation of the responder at the time of responding is appropriate for the recognized utterance situation of the respondee; a fourth processing unit that acquires, during the service period, the speech voice data of the person being served and response voice data uttered by the person serving subsequent to the speech of the person being served, and recognizes the speech content of the person being served and the response content of the person serving from the acquired speech voice data and response voice data, respectively; A fifth processing unit that determines whether the contents of the response by the respondent to the contents of the utterance by the respondent are appropriate or not; a sixth processing unit that generates response support information for the person who is responding based on a result of determining a transition in the emotion of the person who is being responded to, a result of determining whether the speech situation is appropriate, and a result of determining whether the content of the response is appropriate, and presents the generated response support information to the person who is responding; A response support device comprising:

6. A method for supporting a caller in a conversation with a callee by using an information processing device, comprising: acquiring information representing an emotion sequence of the person being served during a service period, and determining a transition of the emotion of the person being served based on the acquired emotion sequence; a step of acquiring speech voice data uttered by the person being served and response voice data uttered by the person serving during the service period, respectively, and recognizing the speech state of the person being served based on the acquired speech voice data, and recognizing the response state of the person serving; A step of determining whether the response state of the respondent is appropriate for the speech state of the responded person; generating response support information for the person serving based on the judgment result of the transition of the person's emotion and the judgment result of the suitability, and presenting the generated response support information to the person serving; A response support method comprising the steps:

7. A method for supporting a caller in a conversation with a callee by using an information processing device, comprising: acquiring information representing an emotion sequence of the person being served during a service period, and determining a transition of the emotion of the person being served based on the acquired emotion sequence; a step of acquiring speech voice data of the person being served and response voice data uttered by the person serving following the speech of the person being served during the service period, and recognizing the speech content of the person being served and the response content of the person serving from the acquired speech voice data and response voice data, respectively; A step of determining whether the contents of the response by the respondent are appropriate to the recognized contents of the utterance by the respondent; generating response support information for the person serving based on the judgment result of the transition of the person's emotions and the judgment result of the appropriateness of the content of the response, and presenting the generated response support information to the person serving; A response support method comprising the steps:

8. A method for supporting a caller in a conversation with a callee by using an information processing device, comprising: acquiring information representing an emotion sequence of the person being served during a service period, and determining a transition of the emotion of the person being served based on the acquired emotion sequence; a step of acquiring speech voice data uttered by the person being served and response voice data uttered by the person serving during the service period, respectively, and recognizing the speech state of the person being served based on the acquired speech voice data, and recognizing the response state of the person serving; A step of determining whether the response state of the respondent is appropriate for the speech state of the responded person; a step of acquiring, during the service period, the speech voice data of the person being served and the response voice data uttered by the person serving subsequent to the speech of the person being served, and recognizing the speech content of the person being served and the response content of the person serving from the acquired speech voice data and response voice data, respectively; A step of determining whether the contents of the response by the respondent are appropriate to the recognized contents of the utterance by the respondent; generating response support information for the person serving based on the judgment result of the transition of the person's emotion, the judgment result of the appropriateness, and the judgment result of the appropriateness of the content of the response, and presenting the generated response support information to the person serving; A response support method comprising the steps:

9. 3. A program for causing a processor included in the response assistance device to execute at least one of the processes performed by first to fourth processing units included in the response assistance device according to claim 1 or 2.

10. 5. A program for causing a processor included in the response assistance device to execute at least one of the processes performed by first to fourth processing units included in the response assistance device according to claim 3 or 4.

11. 6. A program for causing a processor included in the response assistance device to execute at least one of the processes performed by first to sixth processing units included in the response assistance device according to claim 5.

Images