Online customer service system, control program, and control method

Abstract

We provide an online customer service system that can appropriately assist in the interaction between operators and customers. [Solution] The online customer service system 10 comprises a dialogue assistance device 12, an operator terminal 16, and a customer terminal 18. The dialogue assistance device comprises a dialogue status acquisition means for acquiring the dialogue status between the operator and the customer, a dialogue status determination means for determining whether the dialogue between the customer and the operator is progressing smoothly, an advice mode determination means for determining the content of advice and to which advice target to give advice when it is determined that the dialogue is not progressing smoothly, and an accompanying agent control means for controlling an accompanying agent to give advice of the determined content to the determined advice target.

Description

Technical Field

[0001] This invention relates to an online customer service system, a control program, and a control method. In particular, for example, it relates to an online customer service system, a control program, and a control method that assist in the interaction between an operator who responds to customers online and the customers.

Background Art

[0002] An example of a conventional system of this type is disclosed in Patent Document 1. The technology of Patent Document 1 is a system for assisting an operator in handling business with customers over the phone in a call center where a plurality of telephones connected to an outside line through an in-house switch are installed. It includes an operator terminal composed of a computer used by the operator, a supporter terminal used by a supporter who assists the operator, terminal management means for associating and managing the telephones and computers with their users, line switching means for controlling the in-house switch when there is a predetermined information input and appropriately switching the line of the operator's telephone during a call to another appropriate telephone, voice data acquisition means for sampling the transmitted voice of the operator during a phone call and converting it into digital voice data, perplexity evaluation means for stepwise evaluating the degree of perplexity of the operator by analyzing the converted digital voice data based on a predetermined algorithm and outputting the evaluation result, perplexity display means for appropriately displaying and outputting the evaluation result output by the perplexity evaluation means on the supporter terminal, call monitoring means for making the call on the operator's telephone available for listening by the supporter, supporter method presentation means for enabling selection of a plurality of support methods and displaying and outputting them on the operator terminal when there is a predetermined user input on the supporter terminal, and support processing means for executing a predetermined support process based on the selected method when there is a user input for selecting one of the plurality of support methods on the operator terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

[0004] In the technology described in Patent Document 1, when an operator (user) interacting with a customer is confused, a support person provides advice to the operator or interacts with the customer on the operator's behalf. However, in the technology described in Patent Document 1, the support person cannot always assist the operator, for example, when they are assisting another customer. Furthermore, since the technology described in Patent Document 1 only assists the operator, it is insufficient from the perspective of assisting the dialogue between the operator and the customer in business negotiations. Therefore, a technology that can appropriately assist the dialogue between the operator and the customer is desired.

[0005] Therefore, the primary objective of this invention is to provide a novel online customer service system, control program, and control method system.

[0006] Another object of this invention is to provide an online customer service system, control program, and control method that can appropriately assist the interaction between the operator and the customer. [Means for solving the problem]

[0007] The first invention is an online customer service system comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the dialogue between the operator and the customer, wherein the dialogue assistance device comprises a dialogue status acquisition means for acquiring the dialogue status between the operator and the customer, a dialogue status determination means for determining whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means, an advice mode determination means for determining the content of advice based on the dialogue status acquired by the dialogue status acquisition means when the dialogue status determination means determines that the dialogue is not progressing smoothly, and for determining whether to give advice to the operator only, the customer only, or both the operator and the customer, and an accompanying agent control means for controlling an accompanying agent to give advice of the content determined by the advice mode determination means to the advice targets determined by the advice mode determination means.

[0008] According to the first invention, when the dialogue between the operator and the customer is not progressing smoothly, an appropriate target for advice is selected and the accompanying agent provides advice, thereby appropriately assisting the dialogue between the operator and the customer. Therefore, the dialogue between the operator and the customer can be made to proceed appropriately.

[0009] The second invention is dependent on the first invention, and the accompanying agent includes a virtual character displayed on the display device of the customer terminal and the display device of the operator terminal, respectively.

[0010] The third invention is dependent on the first or second invention, and the dialogue situation acquired by the dialogue situation acquisition means includes at least one of audio data or text data relating to the dialogue between the operator and the customer.

[0011] The fourth invention is dependent on the first or second invention, and the dialogue situation acquired by the dialogue situation acquisition means includes image data of the operator and the customer taken while the operator and the customer are having a dialogue.

[0012] The fifth invention is an online customer service system comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for assisting the dialogue between the operator and the customer, wherein the dialogue assistance device comprises: a dialogue status acquisition means for acquiring the status of the dialogue between the operator and the customer; a dialogue status determination means for determining whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means; an advice mode determination means for determining the content of advice based on the dialogue status acquired by the dialogue status acquisition means when the dialogue status determination means determines that the dialogue is not progressing smoothly, and for determining whether to give advice to the operator only, the customer only, or both the operator and the customer; and a terminal control means for controlling the customer terminal and the operator terminal to give advice with the content determined by the advice mode determination means to the advice targets determined by the advice mode determination means.

[0013] According to the fifth invention, when the dialogue between the operator and the customer is not progressing smoothly, the system selects an appropriate target for advice and provides advice, thereby effectively assisting the dialogue between the operator and the customer. Therefore, the dialogue between the operator and the customer can be made to proceed smoothly.

[0014] The sixth invention is a control program executed in a dialogue assistance device of an online customer service system, which comprises a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the dialogue between the operator and the customer. The control program causes the processor of the dialogue assistance device to execute: a dialogue status acquisition means for acquiring the dialogue status between the operator and the customer; a dialogue status determination means for determining whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means; an advice mode determination means for determining the content of advice based on the dialogue status acquired by the dialogue status acquisition means when the dialogue status determination means determines that the dialogue is not progressing smoothly, and for determining whether to give advice to the operator only, the customer only, or both the operator and the customer; and an accompanying agent control means for controlling an accompanying agent to give advice with the content determined by the advice mode determination means to the advice targets determined by the advice mode determination means.

[0015] The seventh invention is a control method for a dialogue assistance device in an online customer service system, comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the dialogue between the operator and the customer, wherein the processor of the dialogue assistance device acquires the dialogue status between the operator and the customer, determines whether the dialogue between the customer and the operator is progressing smoothly based on the acquired dialogue status, and when it is determined that the dialogue is not progressing smoothly, determines the content of advice based on the acquired dialogue status, and also determines whether to give advice to the operator only, the customer only, or both the operator and the customer, and controls the accompanying agent to give the determined advice to the determined target.

[0016] According to the sixth or seventh invention, similar to the first invention, the interaction between the operator and the customer can be appropriately assisted. Therefore, the interaction between the operator and the customer can be facilitated appropriately. [Effects of the Invention]

[0017] According to this invention, it is possible to appropriately assist the interaction between the operator and the customer and to appropriately proceed with the interaction.

[0018] The above object, other objects, features, and advantages of this invention will become more apparent from the detailed description of the embodiments described below with reference to the drawings.

Brief Description of the Drawings

[0019] [Figure 1] It is a diagram showing an online customer service system according to an embodiment of this invention. [Figure 2] It is a block diagram showing an example of the electrical configuration of the dialogue assistance device. [Figure 3] It is a block diagram showing an example of the electrical configuration of the operator terminal. [Figure 4] It is a block diagram showing an example of the electrical configuration of the customer terminal. [Figure 5] It is a diagram showing an example of the operator-side dialogue screen displayed on the display device of the operator terminal. [Figure 6] It is a diagram showing an example of the customer-side dialogue screen displayed on the display device of the customer terminal. [Figure 7] It is a diagram showing an example of the state when the co-seat agent interrupts the dialogue on the operator-side dialogue screen. [Figure 8] It is a diagram showing an example of the state when the co-seat agent interrupts the dialogue on the customer-side dialogue screen. [Figure 9] It is a diagram showing a part of an example of the prompt input to the dialogue situation determination model. [Figure 10] It is a diagram showing another part of the example of the prompt in FIG. 9. [Figure 11] It is a diagram showing an example of the input and output of the dialogue situation determination model. [Figure 12] It is a diagram showing a part of an example of the prompt input to the advice mode determination model. [Figure 13] It is a diagram showing another part of the example of the prompt in FIG. 12. [Figure 14]This figure shows an example of the input and output of the advice mode determination model. [Figure 15] This figure shows an example of a RAM memory map for a dialogue assistance device. [Figure 16] This figure shows an example of the RAM memory map of the operator terminal. [Figure 17] This figure shows an example of a memory map of the RAM in a customer terminal. [Figure 18] This flowchart shows an example of the dialogue assistance processing performed by the CPU of a dialogue assistance device. [Modes for carrying out the invention]

[0020] Referring to Figure 1, an online customer service system 10 (hereinafter simply referred to as "System 10"), which is one embodiment of the present invention, includes a dialogue assistance device 12. The dialogue assistance device 12 is communicated with an operator terminal 16 and a customer terminal 18 via a network 14. The operator terminal 16 and the customer terminal 18 are also communicated with each other via the network 14.

[0021] Furthermore, the dialogue assistance device 12 is connected to the dialogue situation determination model 12a and the advice mode determination model 12b in a communicative manner. However, some or all of the dialogue situation determination model 12a and the advice mode determination model 12b may be included in the dialogue assistance device 12, or they may be connected to the dialogue assistance device 12 in a communicative manner via the network 14.

[0022] System 10 is used, as an example, in online, remote insurance sales negotiations. As will be explained in detail later, in System 10, an AI-powered, autonomous conversational agent is present during the sales negotiation between the sales representative (operator) and the customer. When the conversation between the sales representative and the customer is not progressing smoothly, the agent intervenes and provides advice to the sales representative, the customer, or both, to help the conversation proceed appropriately.

[0023] The dialogue assistance device 12 is an information processing device such as a personal computer and a server, and it comprehensively controls the entire system 10. This dialogue assistance device 12 uses AI to assist in conversations between sales representatives and customers. In short, the dialogue assistance device 12 controls the behavior of an accompanying agent that acts as the sales representative's supervisor, correcting mistakes in the conversation and supplementing the content of the conversation.

[0024] Network 14 consists of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network can include public telephone networks, mobile phone networks, wired LANs, wireless LANs, CATV (Cable Television), etc.

[0025] The operator terminal 16 is an information processing device such as a personal computer, and is used by a human operator who is a sales representative for a business negotiation. In this embodiment, a dialogue screen 100 (see Figure 5) including a photograph of the customer the operator is dealing with is displayed on the display device 50 of the operator terminal 16, and the operator interacts with the customer online through this dialogue screen 100. In addition, an image of an accompanying agent (agent image 106) is also displayed on the display device 50 of the operator terminal 16, and based on instructions from the dialogue assistance device 12, the operator terminal 16 outputs advice from the accompanying agent in the form of voice and text display, and conveys this advice to the operator.

[0026] This embodiment describes a case where a business negotiation dialogue is conducted using voice, but it is also possible to conduct a business negotiation dialogue using text. Furthermore, the operator or customer may be given the option to choose between a voice-based or text-based business negotiation dialogue.

[0027] The customer terminal 18 is an information processing device such as a PC, smartphone, or tablet, used by the customer (user) who is the business partner. The customer terminal 18 may be a terminal owned by the customer, or it may be a terminal installed in a store or other location where the system 10 is applied.

[0028] In this embodiment, a dialogue screen 110 (see Figure 6) including an image of the operator's avatar (operator avatar) that interacts with the customer is displayed on the display device 70 of the customer terminal 18, and the customer interacts with the operator online through this dialogue screen 110. An image of an accompanying agent (agent image 114) is also displayed on the display device 70 of the customer terminal 18, and the operator terminal 16 outputs advice from the accompanying agent via voice and text display, etc., based on instructions from the dialogue assistance device 12, and conveys this advice to the customer.

[0029] The dialogue status determination model 12a is a language model for determining whether the dialogue between the operator and the customer is progressing smoothly. As an example, a large-scale language model (hereinafter referred to as "LLM") such as ChatGPT can be used.

[0030] The advice mode determination model 12b is a language model for determining what kind of advice to give to which recipient (operator only, customer only, or both operator and customer) when the interaction between the operator and the customer is not progressing smoothly. As an example, an LLM such as ChatGPT can be used.

[0031] Figure 2 is a block diagram showing an example of the electrical configuration of the dialogue assistance device 12. As shown in Figure 2, the dialogue assistance device 12 includes a CPU 20. The CPU 20 is connected via an internal bus to a RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26.

[0032] The CPU 20 is a processor that controls the dialogue assistance device 12 and, by extension, the entire system 10. The CPU 20 functions as a dialogue status acquisition means, a dialogue status determination means, an advice mode determination means, and an accompanying agent control means according to this invention. However, instead of the CPU 20, a System-on-a-Chip (SoC) including multiple functions such as CPU function and GPU (Graphics Processing Unit) function may be provided.

[0033] RAM22 is the main memory of the interactive assistance device 12 and is used as the work area or buffer area of ​​the CPU 20. Although not shown in the diagram, the interactive assistance device 12 is provided with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the operating system and various application software are stored in this auxiliary storage device.

[0034] Communication I / F24 is a wired interface that, under the control of the CPU 20, transmits and receives control signals and data to and from external computers such as the operator terminal 16 and customer terminal 18 via the network 14. However, a wireless interface for connecting to a wireless LAN can also be used as the communication I / F24.

[0035] Input / output interface 26 is connected to input devices 28 such as a keyboard and computer mouse, and display devices 30 such as a liquid crystal display. The input / output interface 26 outputs operation data (or operation information) input from the input devices 28 to the CPU 20. It also outputs image data generated by the CPU 20 to the display device 30, and displays a screen or image corresponding to the image data on the display device 30. Note that the configuration of the interactive assistance device 12 shown in Figure 2 is just an example and is not limited to this configuration.

[0036] Figure 3 is a block diagram showing an example of the electrical configuration of the operator terminal 16. As shown in Figure 3, the operator terminal 16 includes a CPU 40. The CPU 40 is connected to the RAM 42, communication I / F 44, and input / output I / F 46 via an internal bus.

[0037] The CPU 40 is a processor that controls the overall operation of the operator terminal 16. However, instead of the CPU 40, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0038] RAM 42 is the main memory of the operator terminal 16 and is used as the work area or buffer area of ​​the CPU 40. Although not shown in the diagram, the operator terminal 16 is also equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software, such as the operating system and various application software, is stored in this auxiliary storage device.

[0039] The communication interface 44 is a wired interface for sending and receiving control signals and data between the interaction support device 12 and external computers such as customer terminals 18 via the network 14 under the control of the CPU 40. However, a wireless interface for connecting to a wireless LAN can also be used as the communication interface 44.

[0040] The input / output interface 46 is connected to an input device 48, a display device 50, a microphone 52, a speaker 54, and a camera 56, among others. The input device 48 is a keyboard, computer mouse, and touch panel, etc. The display device 50 is, for example, a liquid crystal display. The dialogue screen 100, which will be described later, is displayed on this display device 50. The microphone 52 is an operator voice detection means for detecting the voice emitted by the operator using the operator terminal 16, and is provided in a manner that can detect the operator's voice. The speaker 54 is a customer voice output means for outputting the voice of a customer, and is provided in a manner that the operator can hear the output sound from the speaker 54. The speaker 54 is also an agent voice output means for outputting the voice (synthesized voice) of an accompanying agent. Note that a headset in which the microphone 52 and speaker 54 are integrated may be used. The camera 56 is an operator shooting means for shooting the operator using the operator terminal 16, and is provided in a manner that can shoot the operator.

[0041] The input / output interface 46 outputs operation data (or operation information) received from the input device 48 to the CPU 40. The input / output interface 46 also outputs image data generated by the CPU 40 to the display device 50, causing the display device 50 to display the screen corresponding to the image data. However, image data received from external computers such as the dialogue assistance device 12 and the customer terminal 18 may also be output to the display device 50 by the CPU 40.

[0042] Furthermore, the input / output interface 46 converts the operator's voice detected by the microphone 52 into digital audio data and outputs it to the CPU 40, and converts the audio data output by the CPU 40 into an analog audio signal and outputs it from the speaker 54. In addition, the input / output interface 46 outputs image data (operator image) including the operator captured (detected) by the camera 56 to the CPU 40. The audio data output from the CPU 40 includes customer voice data received from the customer terminal 18 and agent voice data received from the dialogue assistance device 12. The image of the operator captured by the camera 56 may be a still image or a moving image. Note that the configuration of the operator terminal 16 shown in Figure 3 is just an example and is not limited to this.

[0043] Figure 4 is a block diagram showing an example of the electrical configuration of a customer terminal 18. As shown in Figure 4, the customer terminal 18 includes a CPU 60. The CPU 60 is connected to RAM 62, a communication interface 64, and an input / output interface 66 via an internal bus.

[0044] The CPU 60 is a processor that controls the overall operation of the customer terminal 18. However, instead of the CPU 60, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0045] RAM62 is the main memory of the customer terminal 18 and is used as the work area or buffer area of ​​the CPU 60. Although not shown in the diagram, the customer terminal 18 is also equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software, such as the operating system and various application software, is stored in this auxiliary storage device.

[0046] The communication interface 64 is a wired interface for sending and receiving control signals and data between the interaction support device 12 and an external computer such as the operator terminal 16, via the network 14 under the control of the CPU 60. However, a wireless interface for connecting to a wireless LAN can also be used as the communication interface 64.

[0047] The input / output interface 66 is connected to an input device 68, a display device 70, a microphone 72, a speaker 74, and a camera 76, among others. The input device 68 is a keyboard, computer mouse, and touch panel, etc. The display device 70 is, for example, a liquid crystal display. The interactive screen 110, which will be described later, is displayed on this display device 70. The microphone 72 is a customer voice detection means for detecting the voice emitted by a customer using the customer terminal 18, and is provided in a manner that can detect customer voices. The speaker 74 is an operator voice output means for outputting the operator's voice, and is provided in a manner that allows the user to hear the output sound from the speaker 74. The speaker 74 is also an agent voice output means for outputting the voice (synthesized voice) of an accompanying agent. The camera 76 is a customer shooting means for shooting the customer using the customer terminal 18, and is provided in a manner that can shoot the customer.

[0048] The input / output interface 66 outputs operation data (or operation information) received from the input device 68 to the CPU 60. The input / output interface 66 also outputs image data generated by the CPU 60 to the display device 70, causing the display device 70 to display a screen corresponding to the image data. However, image data received from external computers such as the dialogue assistance device 12 and the operator terminal 16 may also be output to the display device 70 by the CPU 60.

[0049] Furthermore, the input / output interface 66 converts the user's voice detected by the microphone 72 into digital audio data and outputs it to the CPU 60, and converts the audio data output by the CPU 60 into an analog audio signal and outputs it from the speaker 74. In addition, the input / output interface 66 outputs data of the customer's image (customer image) captured by the camera 76 to the CPU 60. The audio data output from the CPU 60 includes operator voice data received from the operator terminal 16 and agent voice data received from the dialogue assistance device 12. The customer's image captured by the camera 76 may be a still image or a moving image. Note that the configuration of the customer terminal 18 shown in Figure 4 is just an example and is not limited to this.

[0050] The operation of System 10 in this embodiment will now be described. In System 10, an operator (sales representative) using the operator terminal 16 can conduct online business negotiations with a customer using the customer terminal 18. For example, in System 10, a customer accesses a website providing a predetermined service, such as a general company site or an electronic trading site, on the customer terminal 18 and requests a business negotiation (such as an explanation of insurance details) with a sales representative. The website's matching server then matches the customer with a sales representative, and a connection is established between this sales representative's operator terminal 16 and the customer terminal 18. A connection is also established between the operator terminal 16 and the customer terminal 18 and the dialogue assistance device 12, and the online business negotiation begins.

[0051] When conducting an online business negotiation, the display device 50 of the operator terminal 16 displays a dialogue screen 100 as shown in Figure 5. Meanwhile, the display device 70 of the customer terminal 18 displays a dialogue screen 110 as shown in Figure 6.

[0052] As shown in Figure 5, the operator's dialogue screen 100 has a display area 102 located approximately in the center of the screen, and a button image 104 located in the lower right corner of the screen. The display area 102 is the area for outputting customer image data, and displays the image of the customer the operator is interacting with. The button image 104 is an icon for ending the business negotiation dialogue with the customer. When the button image 104 is turned on using the computer mouse, a notification of the end of the business negotiation dialogue is sent to the customer terminal 18 and the dialogue assistance device 12.

[0053] As shown in Figure 6, the customer-facing dialogue screen 110 displays an avatar image (hereinafter referred to as "avatar image") 112, which is a representation of the operator. In this embodiment, the avatar image 112 is a computer-generated image of a virtual character (CG character) modeled after a human woman. However, the avatar image 112 is not limited to this and can also be an image of a virtual character such as an animal or robot, an anime character, or a game character.

[0054] The avatar image 112 displayed on the customer's dialogue screen 110 performs actions corresponding to the operator image captured by the camera 56 of the operator terminal 16. To this end, the operator terminal 16 performs a motion detection process on the operator image and generates avatar image data for displaying the avatar image 112 that performs actions corresponding to the operator image. This avatar image data is transmitted to the customer terminal 18, and upon receiving the avatar image data, the customer terminal 18 displays the avatar image 112 on the dialogue screen 110 based on the avatar image data.

[0055] However, the dialogue screen 110 may display an image of the operator themselves instead of the avatar image 112. Alternatively, the operator terminal 16 may generate action data corresponding to the operator's image and send it to the customer terminal 18, where the customer terminal 18 generates the avatar image data.

[0056] Furthermore, the operator terminal 16 generates voice data, i.e., operator voice data, for controlling the avatar's speech (voice output) based on the operator's voice. The operator voice data may be the operator's voice data, or it may be data obtained by converting the operator's voice into the avatar's voice.

[0057] The operator terminal 16 transmits the generated motion data and operator voice data to the customer terminal 18. The customer terminal 18 generates and displays (or updates) the avatar image 112 according to the received motion data and makes the avatar move. The customer terminal 18 also outputs the received operator voice data to the speaker 74. At this time, the avatar image 112 is lip-synced.

[0058] Meanwhile, when the customer terminal 18 detects the customer's voice through the microphone 72, it transmits voice data corresponding to the detected voice, i.e., customer voice data, to the operator terminal 16. The operator terminal 16 outputs the received customer voice data to the speaker 54. Therefore, the operator can hear the customer's voice.

[0059] Furthermore, the customer terminal 18 transmits image data corresponding to the image (in this embodiment, a moving image) of the customer captured by the camera 76, i.e., customer image data, to the operator terminal 16. The operator terminal 16 outputs the received customer image data to the display device 50. Therefore, the operator can view the customer's image.

[0060] Thus, while an operator using the operator terminal 16 and a customer using the customer terminal 18 can conduct online business negotiations, the conversation between the operator and the customer does not always proceed smoothly. For example, the operator may forget to convey information to the customer or convey information incorrectly. Also, the customer may not be able to effectively communicate what they want to know to the operator, or they may not know what questions to ask.

[0061] Therefore, in this embodiment, an AI-powered autonomous dialogue-capable companion agent is present during the dialogue between the operator and the customer. When the dialogue between the operator and the customer is not progressing smoothly, the companion agent intervenes and provides advice, thereby ensuring that the dialogue between the operator and the customer proceeds appropriately. Furthermore, depending on the content of the advice, it may be better to inform both the operator and the customer, or it may be better not to inform either the operator or the customer, or it may not be necessary to inform either of them. Therefore, the system is designed to select the appropriate recipient of the advice. In this embodiment, the dialogue assistance device 12 controls the behavior of the companion agent, and voice data (agent voice data), text data, action data (agent action data), and commands for the companion agent to execute advice are transmitted from the dialogue assistance device 12 to the operator terminal 16 and the customer terminal 18. A detailed explanation follows below.

[0062] As shown in Figure 5, the operator's dialogue screen 100 displays an image of an accompanying agent (hereinafter referred to as "agent image") 106 in the upper right corner of the screen. In this embodiment, the agent image 106 is a computer-generated image of a virtual character (CG character) that resembles the operator's superior. However, the agent image 106 is not limited to this and can also be an image of a virtual character such as an animal or robot, an anime character, or a game character. Also, as shown in Figure 6, the customer's dialogue screen 110 displays an agent image 114 similar to that on the operator's dialogue screen 100 in the upper right corner of the screen.

[0063] In this way, by displaying agent images 106 and 114 on the operator's and customer's dialogue screens 100 and 110, respectively, the operator and customer can recognize that there is an observer present in the dialogue. By recognizing the presence of the observer, the operator and customer can have the expectation that they will receive advice when they encounter difficulties, and can proceed with the dialogue with confidence.

[0064] In system 10, when the conversation between the operator and the customer is progressing smoothly, that is, when there are no problems with the conversation, the accompanying agent listens to the conversation with a smile, etc. Specifically, based on instructions from the conversation assistance device 12, smiling agent images 106 and 114 are displayed on the conversation screens 100 and 110, respectively, and the accompanying agent's voice and spoken text are not output.

[0065] On the other hand, when the dialogue between the operator and the customer is not progressing smoothly, that is, when there is a problem with the dialogue, the accompanying agent intervenes in the dialogue and provides advice to the selected recipient of advice (either the operator only, the customer only, or both the operator and the customer). Specifically, the dialogue assistance device 12 transmits predetermined control signals and data to the terminal operated by the selected recipient of advice (either the operator terminal 16 only, the customer terminal 18 only, or both the operator terminal 16 and the customer terminal 18), causing at least one of the accompanying agent's voice and spoken text regarding the selected advice to be output on the terminal, and also causing the agent images 106 and 114 to lip-sync.

[0066] Here, whether the interaction between the operator and the customer is progressing smoothly is determined based on the interaction status between the operator and the customer acquired by the interaction assistance device 12. This interaction status includes at least one of the audio data or text data regarding the interaction between the operator and the customer (i.e., the history of the interaction between the operator and the customer), image data of the operator and the customer taken while they were interacting, and the communication status between the operator terminal 16 and the customer terminal 18.

[0067] Therefore, operator voice data detected by the microphone 52 of the operator terminal 16, and operator image data captured by the camera 56, are transmitted from the operator terminal 16 to the dialogue assistance device 12. Similarly, customer voice data detected by the microphone 72 of the customer terminal 18, and customer image data captured by the camera 76, are transmitted from the customer terminal 18 to the dialogue assistance device 12. The dialogue assistance device 12 transcribes the operator voice data and customer voice data into text, and the text is recorded chronologically, thereby creating a dialogue history between the operator and the customer. However, if the operator and the customer conduct a business negotiation dialogue using text input, that text data is transmitted from the operator terminal 16 and the customer terminal 18 to the dialogue assistance device 12, and the dialogue history between the operator and the customer is created using that text data.

[0068] Furthermore, situations in which the conversation between the operator and the customer is not progressing smoothly include situations where at least one of the operator or the customer is having difficulty, situations where the pace of the conversation is poor, situations where the operator gives incorrect answers, fails to fill in predetermined hearing items, fails to proceed with the conversation according to predetermined content, and situations where the conversation cannot continue due to unforeseen problems such as communication problems. When the conversation assistance device 12 detects that any of these situations are occurring based on the acquired conversation status between the operator and the customer, it determines that the conversation between the operator and the customer is not progressing smoothly.

[0069] For example, the dialogue assistance device 12 performs image analysis based on a neural network or rules on operator image data and customer image data, and determines that the dialogue is not progressing smoothly when it detects a troubled expression or gesture from the operator or customer. It also performs acoustic analysis based on a neural network or rules on operator voice data and determines that the dialogue is not progressing smoothly when it detects that the operator has not been able to answer the customer's questions within a certain time or that the period of silence between the two exceeds a predetermined time. Furthermore, by analyzing the dialogue history between the operator and customer using LLM, or by comparing the dialogue history with hearing item memos and dialogue content memos, it determines that the dialogue is not progressing smoothly when it detects that the operator has given incorrect answers, failed to fill in hearing items, or failed to proceed with the dialogue as intended. In addition, when the communication status detection function of system 10 detects that a communication problem has occurred between the operator terminal 16 and the customer terminal 18, it determines that the dialogue is not progressing smoothly. Furthermore, the operator terminal 16 and the customer terminal 18 may each be provided with an advice request button to request advice from an accompanying agent, and when this advice request button is pressed by the operator or customer, it may be determined that the conversation is not progressing smoothly.

[0070] If the dialogue assistance device 12 determines that the dialogue is not progressing smoothly, the accompanying agent intervenes to offer advice. Depending on the content of the advice, it may be better to inform both the operator and the customer, or it may be better not to inform one of them, or it may not be necessary to inform the other. Therefore, the appropriate recipient of the advice is selected from the operator only, the customer only, or both the operator and the customer. For example, if the operator's explanation is incorrect, carelessly pointing this out to both the operator and the customer may make the operator feel uncomfortable, and the customer may feel uneasy about whether this person (the operator) is reliable. For this reason, if the operator's explanation is incorrect, for example, the agent may discreetly inform only the operator that the explanation is wrong, and then provide supplementary explanations to both the operator and the customer, guiding the operator to the correct explanation. Also, for example, when the operator asks, "Do you have any questions?", the customer may not be able to think of an appropriate question and answer, "No, I don't have any," which may prevent the dialogue from becoming lively. In such cases, for example, an accompanying agent can take the customer's perspective and ask the operator follow-up questions or common questions in a way that is understandable to the customer. This can lead to a more engaging conversation and a deeper understanding of the product or service by the customer.

[0071] The content of the advice should be determined by, for example, analyzing the previous conversation history between the operator and the customer using LLM (Liquid Language Management) to detect the cause of the conversation not progressing smoothly, and then deriving a solution based on knowledge and information related to the business negotiation. Furthermore, the target audience for the advice should be determined, for example, according to pre-configured settings, or by using predetermined judgment rules or LLM based on the content of the conversation.

[0072] Figures 7 and 8 show examples of the operator's dialogue screen 100 and the customer's dialogue screen 110, respectively, when an accompanying agent intervenes in the conversation between the operator and the customer. Figures 7 and 8 show specific examples of how the accompanying agent provides advice when the operator gives the wrong amount to the customer.

[0073] As shown in Figures 7 and 8, when the system detects that the operator has given the wrong amount to the customer, a text message 108a saying "The amount is wrong" is displayed on the operator's dialogue screen 100, and a synthesized voice from the accompanying agent saying "The amount is wrong" is output from the speaker 54 of the operator's terminal 16. In conjunction with this, the agent image 106 on the dialogue screen 100 is lip-synced. On the other hand, there is no change in the accompanying agent's behavior on the customer's terminal 18. In other words, the accompanying agent only informs the operator that the amount is wrong.

[0074] Subsequently, a text message 108b appears on the operator's dialogue screen 100 stating, "Excuse me, the price varies depending on age, gender, and plan. How much would it be for you?", and a synthesized voice of the accompanying agent with the same content is output from the speaker 54 of the operator's terminal 16. In conjunction with this, the agent image 106 on the dialogue screen 100 lip-syncs. Meanwhile, a text message 116 appears on the customer's dialogue screen 110 stating, "Excuse me, the price varies depending on age, gender, and plan. How much would it be for you?", and a synthesized voice of the accompanying agent with the same content is output from the speaker 74 of the customer's terminal 18. In conjunction with this, the agent image 114 on the dialogue screen 110 lip-syncs. In other words, the accompanying agent provides supplementary explanations to both the operator and the customer, prompting the operator to answer with the correct amount. This allows the operator to smoothly correct the amount without causing distrust in the customer.

[0075] Figures 9 and 10 show examples of prompts used to determine whether the interaction between the operator and the customer is progressing smoothly, that is, to detect when the interaction is not progressing smoothly. Prompts like those shown in Figures 9 and 10 are input to the interaction status determination model 12a, which then determines whether the interaction between the operator and the customer is progressing smoothly. In other words, the interaction status determination model 12a functions as an AI for determining whether the interaction between the operator and the customer is progressing smoothly.

[0076] As shown in Figures 9 and 10, prompts used to determine whether the interaction between the operator and the customer is progressing smoothly include instructions (i.e., commands) such as, "You are an insurance sales supervisor. Based on the conversation between the sales representative and the customer, please identify any problems in the following cases in about 10 characters. If there are no problems with the concise verbal response, please answer 'No problems'," along with examples, data that serves as background knowledge for the judgment, and example outputs.

[0077] When conducting online business negotiations, each time the operator and the customer speak, the spoken content is recognized by speech recognition and transcribed into text, and the content of that text is sequentially input into the dialogue status determination model 12a. Then, the determination result of whether the dialogue between the operator and the customer is progressing smoothly is sequentially output. As shown in Figure 11, for example, if the customer's utterance (question) is "What specific coverage does this insurance product include?" and the operator's utterance (answer) is "Yes, this insurance includes coverage for medical expenses and hospitalization costs," the determination result "Incomplete coverage" is output, indicating that there is a problem with the dialogue. Also, for example, if the customer's next utterance is "Specifically, which illnesses does this insurance product include?" and the operator's utterance is "Yes, this insurance includes coverage for medical expenses and hospitalization costs," the determination result "No problem" is output, indicating that there is no problem with the dialogue. Similarly, for each interaction (i.e., exchange) between the operator and the customer, a result indicating whether the interaction between the operator and the customer is progressing smoothly will be output.

[0078] Figures 12 and 13 show examples of prompts used by the accompanying agent to determine the subject and content of the advice it will provide. Prompts like those shown in Figures 12 and 13 are input to the advice mode determination model 12b, which then determines the subject and content of the advice. In other words, the advice mode determination model 12b functions as an AI for determining the subject and content of the advice.

[0079] As shown in Figures 12 and 13, prompts for determining who to advise and what to advise include instructions (i.e., commands) such as, "You are an insurance sales supervisor. Based on the conversation between the salesperson and the customer, please respond according to the following rules," along with the rules, data that serves as background knowledge for the decision, and example outputs.

[0080] When conducting online business negotiations, similar to the case of the dialogue situation determination model 12a, the transcribed utterances are sequentially input into the advice mode determination model 12b, and the advice target and advice content are sequentially output. As shown in Figure 14, for example, if the customer's utterance is "What specific coverages are included in this insurance product?" and the operator's utterance is "Yes, this insurance includes coverage for medical expenses and hospitalization costs," then the determination result is output indicating that advice should be sent to the sales representative that there is an error in the coverage. Also, for example, if the customer's next utterance is "What specific illnesses or accidents are covered?" and the operator's utterance is "Most major illnesses and accidents are covered. For example, cancer and heart disease," then the determination result is output indicating that there is no advice target. Similarly, the advice target and advice content are output for each conversation between the operator and the customer.

[0081] Figure 15 shows an example of the memory map 200 of the RAM 22 built into the dialogue assistance device 12. As shown in Figure 15, the RAM 22 includes a program storage area 202 and a data storage area 204. The program storage area 202 stores a dialogue assistance program, which is an example of an information processing program executed by the dialogue assistance device 12 in this embodiment. This dialogue assistance program executed by the dialogue assistance device 12 includes a main processing program 202a, a communication program 202b, a speech recognition program 202c, a dialogue status determination program 202d, an advice mode determination program 202e, and an agent control program 202f, among others.

[0082] The main processing program 202a is a program for executing the main routine of the dialogue assistance processing of the dialogue assistance device 12 in this embodiment. The communication program 202b is a program for communicating (sending and receiving data, etc.) with external devices such as the operator terminal 16 and the customer terminal 18.

[0083] The speech recognition program 202c is designed to recognize the voice data of operators (sales representatives) and customers and transcribe it into text. The transcription of the operator's and customer's voice data into text, and the recording of this text in chronological order, creates a dialogue history between the operator and the customer.

[0084] The dialogue situation determination program 202d is a program for determining whether the dialogue between the operator and the customer is progressing smoothly, based on the dialogue situation between the operator and the customer. For example, the dialogue situation determination program 202d includes a program for analyzing facial expressions using image data of the operator and the customer to detect if one or both of the operator and the customer are having trouble (i.e., the dialogue is not going smoothly). The dialogue situation determination program 202d also includes a program for inputting the content of the operator's and customer's utterances into the dialogue situation determination model 12a, which has been inputted with a prompt to detect that the dialogue between the customer and the operator is not going smoothly, in order to obtain a determination result on whether the dialogue between the customer and the operator is going smoothly.

[0085] The advice mode determination program 202e is a program that determines the content of advice based on the interaction situation between the operator and the customer, and also determines whether to provide advice to the operator or the customer, or to both the operator and the customer. For example, the advice mode determination program 202e includes a program that inputs the utterances of the operator and the customer into the advice mode determination model 12b, which has been inputted prompts for determining the advice content and the target of the advice, and obtains the determination result for the advice content and the target of the advice.

[0086] The agent control program 202f is a program for controlling accompanying agents to provide advice of the determined content to the determined target. For example, the agent control program 202f includes a program for creating synthesized speech data (agent speech data 204j) from the text data of the advice content, and for sending this agent speech data 204j to the terminal operated by the target, and for instructing that terminal to output it from its speaker. The agent control program 202f also includes a program for sending the text data of the advice content to the terminal operated by the target, and for instructing that the text be displayed on the dialogue screen of the terminal's display device. Furthermore, the agent control program 202f includes a program for generating agent operation data 204k to operate the agent, and for sending this agent operation data 204k to the terminal operated by the target, and for instructing that the agent image be displayed on the dialogue screen of the terminal's display device.

[0087] Although not shown in the diagram, the program storage area 202 also stores other programs for executing the dialogue assistance program of the dialogue assistance device 12.

[0088] Furthermore, the data storage area 204 stores prompt data 204a, customer voice data 204b, operator voice data 204c, dialogue history data 204d, customer image data 204e, operator image data 204f, dialogue status data 204g, advice content data 204h, advice target data 204i, agent voice data 204j, and agent operation data 204k.

[0089] The prompt data 204a is data for prompts to be input to the dialogue situation determination model 12a and the advice mode determination model 12b, respectively.

[0090] Customer voice data 204b is the customer's voice data received from the customer terminal 18. Operator voice data 204c is the operator's voice data received from the operator terminal 16. Dialogue history data 204d is data recorded by transcribing the operator's and customer's voice data into text and recording it in chronological order.

[0091] Customer image data 204e is customer image data received from customer terminal 18. Operator image data 204f is operator image data received from operator terminal 16.

[0092] Dialogue status data 204g is data on the determination result of whether the dialogue between the customer and the operator is progressing smoothly, obtained from the dialogue status determination model 12a according to the dialogue status determination program 202d.

[0093] The advice content data 204h is text data about the advice content obtained from the advice method determination model 12b according to the advice method determination program 202e. The advice target data 204i is data about the advice target obtained from the advice method determination model 12b according to the advice method determination program 202e.

[0094] Agent voice data 204j is voice data of an accompanying agent that assists in the dialogue between the operator and the customer, and is data obtained by converting the text of the advice content into synthesized speech. Agent action data 204k is action data of the accompanying agent generated in accordance with the advice content, and includes data such as facial expressions and mouth movements.

[0095] Although not shown in the diagram, the data storage area 204 stores other data necessary for the dialogue assistance device 12 to perform dialogue assistance processing, and also contains timers (counters) and flags necessary for performing dialogue assistance processing.

[0096] Figure 16 shows an example of the memory map 300 of the RAM 42 built into the operator terminal 16. As shown in Figure 16, the RAM 42 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores an interactive control program, which is an example of an information processing program executed in the operator terminal 16 of this embodiment. This interactive control program for the operator terminal 16 includes a main processing program 302a, an operation detection program 302b, a communication program 302c, an image generation program 302d, an image output program 302e, a shooting program 302f, an operation data generation program 302g, a sound detection program 302h, a sound output program 302i, and an agent control program 302j, among others.

[0097] The main processing program 302a is a program for executing the main routine of the dialogue control processing of the operator terminal 16 in this embodiment. The operation detection program 302b is a program for detecting operation data 304a input from the input device 48 according to the operator's operation and storing it in the data storage area 304. The communication program 302c is a program for communicating with external devices such as the dialogue assistance device 12 and the customer terminal 18.

[0098] The image generation program 302d is a program for generating image data corresponding to all or part of a screen or image to be displayed on the display device 50, using image generation data 304b. However, when generating image data for the operator's interaction screen 100, customer image data 304c and agent operation data 304i are also used. The image output program 302e is a program for outputting the image data generated according to the image generation program 302d to the display device 50. Therefore, the screen or image corresponding to the image data is displayed on the display device 50.

[0099] The shooting program 302f is a program that causes the camera 56 to perform shooting processing and stores the operator image data 304e input from the camera 56 in the data storage area 304. The motion data generation program 302g is a program that recognizes the operator's movements from the operator image data 304e and generates avatar motion data 304g to move the avatar.

[0100] The sound detection program 302h is a program that detects the operator's voice input from the microphone 52 and temporarily stores the operator's voice data 304f corresponding to the detected voice in the data storage area 304. The sound output program 302i is a program that outputs the customer voice data 304d received from the customer terminal 18 and the agent voice data 304h received from the dialogue assistance device 12 to the speaker 54. Therefore, the customer's voice corresponding to the customer voice data 304d and the voice of the accompanying agent (synthesized voice) corresponding to the agent voice data 304h are output from the speaker 54.

[0101] The agent control program 302j is a program for controlling the speech and actions of the dialogue agent based on instructions from the dialogue assistance device 12. Specifically, the agent control program 302j is a program for displaying text messages on the dialogue screen 100, outputting the voice of the accompanying agent from the speaker 54, and manipulating the agent image 106 on the dialogue screen 100. In this embodiment, the operator is notified of the advice content by outputting agent voice data 304h and displaying text, but only one of these may be adopted. When displaying text, the dialogue assistance device 12 sends the text of the advice content to the operator terminal 16.

[0102] Although not shown in the diagram, the program storage area 302 also stores other programs for executing the dialogue control program of the operator terminal 16.

[0103] The data storage area 304 stores operation data 304a, image generation data 304b, customer image data 304c, customer voice data 304d, operator image data 304e, operator voice data 304f, avatar movement data 304g, agent voice data 304h, and agent movement data 304i, among others.

[0104] Operation data 304a is data about the operator's operations detected according to the operation detection program 302b. Image generation data 304b is data for generating a screen or image to be displayed on the display device 50 of the operator terminal 16.

[0105] Customer image data 304c is customer image data received from customer terminal 18. Customer voice data 304d is customer voice data received from customer terminal 18.

[0106] Operator image data 304e is data of the operator image captured by camera 56. Operator voice data 304f is data of the operator's voice detected by microphone 52 or the voice converted into the avatar's voice. Avatar movement data 304g is avatar movement data generated based on the operator image captured by camera 56, and includes data such as facial expressions and mouth movements.

[0107] Agent voice data 304h is voice data of the accompanying agent received from the dialogue assistance device 12. Agent operation data 304i is operation data of the accompanying agent received from the dialogue assistance device 12.

[0108] Although not shown in the diagram, the data storage area 304 stores other data necessary for the operator terminal 16 to perform dialogue control processing, and also contains timers (counters) and flags necessary for performing dialogue control processing.

[0109] Figure 17 shows an example of a memory map 400 of RAM 62 built into the customer terminal 18. As shown in Figure 16, RAM 62 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores an interaction control program, which is an example of an information processing program executed in the customer terminal 18 of this embodiment. The interaction control program executed in the customer terminal 18 of this embodiment includes a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, a shooting program 402f, a sound detection program 402g, a sound output program 402h, an avatar control program 402i, and an agent control program 402j.

[0110] The main processing program 402a is a program for executing the main routine of the dialogue control processing of the customer terminal 18 in this embodiment. The operation detection program 402b is a program for detecting operation data 404a input from the input device 68 according to the customer's operation and storing it in the data storage area 404. The communication program 402c is a program for communicating with external devices such as the dialogue assistance device 12 and the operator terminal 16.

[0111] The image generation program 402d is a program for generating image data corresponding to all or part of a screen or image to be displayed on the display device 70, using image generation data 404b. However, when generating image data corresponding to the customer interaction screen 110, avatar operation data 404f and agent operation data 404h are also used. The image output program 402e is a program for outputting the image data generated according to the image generation program 402d to the display device 70. Therefore, the screen or image corresponding to the image data is displayed on the display device 70. The shooting program 402f is a program for causing the camera 76 to perform a shooting process and for storing the customer image data 404c input from the camera 76 in the data storage area 404.

[0112] The sound detection program 402g is a program that detects the voice of a customer input from the microphone 72 and temporarily stores the customer voice data 404d corresponding to the detected voice in the data storage area 404. The sound output program 402h is a program that outputs the operator voice data 404e received from the operator terminal 16 and the agent voice data 404g received from the dialogue assistance device 12 to the speaker 74. Therefore, the voice of the operator or avatar corresponding to the operator voice data 404e and the voice of the accompanying agent corresponding to the agent voice data 404g are output from the speaker 74.

[0113] The avatar control program 402i is a program that controls the operator's avatar according to the avatar motion data 404f.

[0114] The agent control program 402j is a program for controlling the speech and actions of the dialogue agent based on instructions from the dialogue assistance device 12. Specifically, the agent control program 402j is a program for displaying text messages on the dialogue screen 110, outputting the voice of the accompanying agent from the speaker 74, and manipulating the agent image 114 on the dialogue screen 110. In this embodiment, the advice content is notified to the customer by outputting agent voice data 404g and displaying text, but only one of these may be adopted. When displaying text, the dialogue assistance device 12 sends the text of the advice content to the customer terminal 18.

[0115] Although not shown in the diagram, the program storage area 402 also stores other programs for executing the dialogue control program of the customer terminal 18.

[0116] The data storage area 404 stores operation data 404a, image generation data 404b, customer image data 404c, customer voice data 404d, operator voice data 404e, avatar movement data 404f, agent voice data 404g, and agent movement data 404h.

[0117] Operation data 404a is data about customer operations detected according to the operation detection program 402b. Image generation data 404b is data for generating a screen or image to be displayed on the display device 70 of the customer terminal 18.

[0118] Customer image data 304c is data of customer images captured by camera 76. Customer audio data 304d is data about customer voices detected by microphone 72.

[0119] Operator voice data 404e is data about the operator's voice received from the operator terminal 16, or the voice obtained by converting the operator's voice into the avatar's voice. Avatar movement data 404f is avatar movement data received from the operator terminal 16.

[0120] Agent voice data 404g is voice data of the accompanying agent received from the dialogue assistance device 12. Agent operation data 404h is operation data of the accompanying agent received from the dialogue assistance device 12.

[0121] Although not shown in the diagram, the data storage area 404 also stores other data necessary for the customer terminal 18 to perform dialogue control processing, and is equipped with timers (counters) and flags necessary for performing dialogue control processing.

[0122] Figure 18 is a flowchart showing an example of the dialogue assistance processing performed by the CPU 20 of the dialogue assistance device 12. As shown in Figure 18, when the CPU 20 starts the dialogue assistance processing, it first acquires the dialogue status between the operator and the customer in step S1. For example, it acquires operator voice data and operator image data from the operator terminal 16, and customer voice data and customer image data from the customer terminal 18. It also creates a dialogue history between the operator and the customer from the acquired operator voice data and customer voice data.

[0123] In the next step, S3, it is determined whether the interaction between the operator and the customer is going smoothly. For example, this is done by performing facial expression analysis on the image data of the operator and the customer, or by inputting the content of the operator's and customer's utterances into the interaction situation determination model 12a and obtaining the determination result. If the answer in step S3 is "YES," that is, if the interaction between the operator and the customer is going smoothly, the process proceeds to step S9. On the other hand, if the answer in step S3 is "NO," that is, if the interaction between the operator and the customer is not going smoothly, the process proceeds to step S5.

[0124] In step S5, the target and content of the advice are determined based on the interaction between the operator and the customer. For example, the results of determining the advice content and target are obtained by inputting the utterances of the operator and the customer into the advice mode determination model 12b.

[0125] In the subsequent step S7, the accompanying agent is controlled to provide the determined advice to the chosen recipient (i.e., the operator only, the customer only, or both the operator and the customer). Specifically, the accompanying agent sends voice data (agent voice data), text data, action data (agent action data), and commands to the terminal operated by the recipient. The actions of the accompanying agent on the terminal operated by the recipient are controlled according to instructions from the CPU 20.

[0126] In the following step S9, it is determined whether or not there is a termination instruction. For example, if a signal (termination notification) indicating that the button image 104 (termination icon) has been pressed is received from the operator terminal 16, it is determined that there is a termination instruction. If the answer in step S9 is "NO", that is, if there is no termination instruction, the process returns to step S1. On the other hand, if the answer in step S9 is "YES", that is, if there is a termination instruction, this dialogue assistance process is terminated.

[0127] As described above, according to this embodiment, an AI-powered autonomous dialogue-capable companion agent is present during the conversation between the operator and the customer. When the conversation between the operator and the customer is not progressing smoothly, the companion agent selects an appropriate target for advice and provides guidance, thereby appropriately assisting the conversation between the operator and the customer. Consequently, the conversation between the operator and the customer can be facilitated appropriately.

[0128] In the above-described embodiment, System 10 was shown being applied to insurance sales negotiations, but System 10 can also be applied to negotiations other than insurance sales (such as automobile sales). Furthermore, although System 10 is primarily used for sales negotiations, it can also be applied to online customer service in places other than sales negotiations, such as hotels, train stations, bus terminals, airports, hospitals, government offices, and shopping malls.

[0129] Furthermore, although the above-described embodiment shows one dialogue assistance device 12, one operator terminal 16, and one customer terminal 18, the system 10 may include one or more dialogue assistance devices 12, one or more operator terminals 16, and one or more customer terminals 18. Also, although the above-described embodiment shows the dialogue assistance device 12 and the operator terminal 16 provided separately, the operator terminal 16 can also perform the functions of the dialogue assistance device 12. In other words, the operator terminal 16 can also function as the dialogue assistance device 12.

[0130] Furthermore, while the above-described embodiment uses a virtual character displayed on a display device as an accompanying agent, the accompanying agent in the online customer service system according to this invention may be a robot existing in real space and positioned near the operator terminal 16 and the customer terminal 18. Alternatively, one of the accompanying agents on the operator side or the customer side may be a CG character, and the other accompanying agent may be a robot. In this invention, an accompanying agent refers to a surrogate entity that can autonomously converse using AI instead of a human, and includes characters existing in virtual space and robots existing in real space.

[0131] Furthermore, in the above-described embodiment, the image of the accompanying agent (agent image) is displayed on the display device from the beginning of the interaction between the operator and the customer. However, the agent image may be displayed on the display device of the terminal receiving advice only when the accompanying agent sends advice.

[0132] Furthermore, in other embodiments of the online customer service system according to this invention, it is not necessarily required to display virtual characters and robots or other accompanying agents on the display device or to place them near the terminal. For example, by changing the tone of voice when giving advice, or by displaying the advice text in a way that indicates whose advice it is, it is possible to provide advice using at least one of voice and text display, so that it is clearly understood that a different personality from the operator and the customer is speaking. In this case, the CPU 20 of the dialogue assistance device 12 functions as a terminal control means that controls the customer terminal and the operator terminal to provide advice of the determined content to the determined advice target.

[0133] Please note that the various screens, prompts, inputs / outputs, and specific numerical values ​​shown in this specification and drawings are merely examples and can be modified as needed. [Explanation of Symbols]

[0134] 10… Online customer service system 12 ... Dialogue assistance device 12a ... Dialogue situation determination model 12b ... Advisory style determination model 14…Network 16 ... Operator terminal 18 ... Customer terminal 20, 40, 60 ...CPU 22,42,62 …RAM 24, 44, 64 ... Communication I / F 26, 46, 66 ... Input / Output Interfaces 28, 48, 68 ... Input devices 30,50,70…display device 52,72…Mike 54,74 ...speaker 56,76 ... Camera 100,110 ...Dialogue screen 106,114 ... Images of accompanying agents (virtual characters)

Claims

1. An online customer service system comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the dialogue between the operator and the customer, The aforementioned dialogue assistance device is A dialogue status acquisition means for acquiring the dialogue status between the operator and the customer, A dialogue status determination means that determines whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means. When the dialogue status determination means determines that the dialogue is not progressing smoothly, the advice mode determination means determines the content of the advice based on the dialogue status acquired by the dialogue status acquisition means, and also determines whether to give advice to the operator only, the customer only, or both the operator and the customer, and An online customer service system comprising an accompanying agent control means for controlling the accompanying agent to provide advice to the advice target determined by the advice method determination means, in accordance with the content determined by the advice method determination means.

2. The online customer service system according to claim 1, wherein the accompanying agent includes a virtual character displayed on the display device of the customer terminal and the display device of the operator terminal, respectively.

3. The online customer service system according to claim 1 or 2, wherein the dialogue status acquired by the dialogue status acquisition means includes at least one of audio data or text data relating to the dialogue between the operator and the customer.

4. The online customer service system according to claim 1 or 2, wherein the dialogue status acquired by the dialogue status acquisition means includes image data of the operator and the customer taken while the operator and the customer are conversing.

5. An online customer service system comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device that assists the dialogue between the operator and the customer, The aforementioned dialogue assistance device is A dialogue status acquisition means for acquiring the dialogue status between the operator and the customer, A dialogue status determination means that determines whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means. When the dialogue status determination means determines that the dialogue is not progressing smoothly, the advice mode determination means determines the content of the advice based on the dialogue status acquired by the dialogue status acquisition means, and also determines whether to give advice to the operator only, the customer only, or both the operator and the customer, and An online customer service system comprising terminal control means for controlling the customer terminal and the operator terminal to provide advice to the advice target determined by the advice method determination means, with the content determined by the advice method determination means.

6. A control program executed in the dialogue assistance device of an online customer service system, comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the conversation between the operator and the customer, The processor of the aforementioned dialogue assistance device: A dialogue status acquisition means for acquiring the dialogue status between the operator and the customer, A dialogue status determination means that determines whether the dialogue between the customer and the operator is progressing smoothly based on the dialogue status acquired by the dialogue status acquisition means. When the dialogue status determination means determines that the dialogue is not progressing smoothly, the advice mode determination means determines the content of the advice based on the dialogue status acquired by the dialogue status acquisition means, and also determines whether to give advice to the operator only, the customer only, or both the operator and the customer, and A control program that causes the accompanying agent control means to execute the accompanying agent control means to provide advice to the advice target determined by the advice method determination means, in the manner determined by the advice method determination means.

7. A control method for a dialogue assistance device in an online customer service system, comprising a customer terminal used by a customer, an operator terminal used by an operator who interacts with the customer online, and a dialogue assistance device for controlling an accompanying agent to assist in the dialogue between the operator and the customer, The processor of the aforementioned dialogue assistance device is The conversation status between the operator and the customer is obtained, Based on the acquired dialogue status, it is determined whether the dialogue between the customer and the operator is progressing smoothly. When it is determined that the aforementioned dialogue is not progressing smoothly, the content of the advice is determined based on the obtained dialogue status, and it is also determined whether the advice should be given to the operator only, the customer only, or both the operator and the customer, and A control method for controlling the accompanying agent to provide advice of the determined content to the determined target of advice.

Images