Information processing system, information processing device, information processing program, and information processing method

Abstract

It can provide appropriate support to operators who interact with customers. [Solution] The information processing system (10) includes a support device (12), which is connected to an operator terminal (16) via a network (14) for communication. When a customer terminal (18) requests assistance, the support device establishes a connection with the operator terminal (S1), and the operator terminal establishes a connection with the customer terminal (S113), and the operator responds to the customer online. Prior to responding to the customer, the operator sets an assistance level, and the support device determines the assistance content based on the assistance level, the progress of the customer service, and the dialogue history (S15), and transmits it to the operator terminal (S17). The operator terminal notifies the operator of the assistance content (S139), and the operator, while responding to the customer, speaks either the notified assistance content as is, or content based on the assistance content.

Description

Technical Field

[0001] This invention relates to an information processing system, an information processing apparatus, an information processing program, and an information processing method, and particularly, for example, to an information processing system, an information processing apparatus, an information processing program, and an information processing method in which an operator responds to customers online.

Background Art

[0002] An example of this type of background art is disclosed in Patent Document 1. The store management server disclosed in Patent Document 1 determines whether an instruction to a customer service staff in a store is necessary or the type of instruction required by the customer service staff based on a store image that is an image of a store where customers consume goods, and obtains an instruction corresponding to the determined instruction type from the stored instructions and outputs it to the customer service staff.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the above Patent Document 1, the types of instructions required by the customer service staff determined based on the store image are serving and clearing dishes, replenishing drinking water, supporting the customer service staff, guiding customers, accounting, work outside the building such as a parking lot, and work for sudden or exceptional events such as a fire, and no support is provided for notifying the content of what the customer service staff says during a festival when serving customers.

[0005] Therefore, the main object of this invention is to provide a novel information processing system, information processing apparatus, information processing program, and information processing method.

[0006] Another object of this invention is to provide an information processing system, information processing device, information processing program, and information processing method that can appropriately support the content of speech spoken by customer service staff. [Means for solving the problem]

[0007] The first invention is an information processing system comprising a customer terminal used by a customer, an operator terminal used by an operator who responds to a customer and is communicably connected to the customer terminal, and a support device that is communicably connected to the operator terminal and assists the operator, wherein the support device comprises a progress determination means for determining the progress of customer service by the operator based on the dialogue history between the operator and the customer when the operator is responding to a customer online, a determination means for determining the content of support for the operator according to the support level that assists the operator in responding to the customer and the progress determined by the progress determination means, and a support content transmission means for transmitting the support content determined by the determination means to the operator terminal, wherein the operator terminal comprises a receiving means for receiving the support content transmitted by the support content transmission means, and a notification means for notifying the operator of the support content received by the receiving means.

[0008] The second invention is subordinate to the first invention, and the support device further comprises setting means for setting a support level according to the operator's operation, customer service level determination means for determining the operator's customer service level based on the dialogue history, and adjustment means for adjusting the support level according to the customer service level determined by the customer service level determination means.

[0009] The third invention is dependent on the first or second invention, and the support level is set in two or more stages. The determination means determines, when the support level is the lowest, if there is a problem with the operator's utterance, to be support content that expresses matters related to the problem in a short sentence or keywords as what the operator should utter. When the support level is the highest, the entire content that the operator should utter is determined as support content according to the progress. If the support level is set in three or more stages, and the support level is other than the highest or lowest, the support content that expresses what the operator should utter is determined to be support content that expresses in a short sentence or keywords according to the progress.

[0010] The fourth invention is an information processing device comprising: progress determination means for determining the progress of customer service by an operator based on the dialogue history between the operator and the customer when the operator is interacting with a customer online; a support level to support the operator's interaction with the customer; a determination means for determining the content of support for the operator according to the progress determined by the progress determination means; and a support content transmission means for transmitting the support content determined by the determination means to the operator's terminal.

[0011] The fifth invention is an information processing program executed by an information processing device, which causes the processor of the information processing device to execute a progress determination step that determines the progress of customer service by the operator based on the dialogue history between the operator and the customer when the operator is serving a customer online, a decision step that determines the level of support to assist the operator in serving the customer and the content of support for the operator according to the progress determined in the progress determination step, and a support content transmission step that transmits the support content determined in the decision step to the operator's terminal.

[0012] The sixth invention is an information processing method for an information processing device, wherein the processor of the information processing device, when the operator is interacting with a customer online, determines the progress of the operator's customer service based on the dialogue history between the operator and the customer, determines the level of support to assist the operator in interacting with the customer and the content of the support provided to the operator according to the determined progress, and transmits the determined support content to the operator's terminal. [Effects of the Invention]

[0013] This invention makes it possible to appropriately support the content of what a customer service employee says.

[0014] The above-mentioned objectives, other objectives, features, and advantages of this invention will become even clearer from the following detailed description of embodiments with reference to the drawings. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 shows an information processing system according to an embodiment of this invention. [Figure 2] Figure 2 is a block diagram showing an example of the electrical configuration of the support device shown in Figure 1. [Figure 3] Figure 3 is a block diagram showing an example of the electrical configuration of the operator terminal shown in Figure 1. [Figure 4] Figure 4 is a block diagram showing an example of the electrical configuration of the customer terminal shown in Figure 1. [Figure 5] Figure 5 shows an example of the operator's dialogue screen displayed on the operator's terminal's display device. [Figure 6] Figure 6 shows an example of a customer-side dialogue screen displayed on the customer terminal's display device. [Figure 7] Figure 7 shows an example of a customer-operator dialogue history arranged chronologically. [Figure 8] Figure 8(A) shows an example of prompts input to the progress determination model, and Figure 8(B) shows an example of input and output to the progress determination model. [Figure 9]FIG. 9(A) is a diagram showing an example of a prompt input to the customer service content determination model, and FIG. 9(B) is a diagram showing an example of the input / output of the customer service content determination model. [Figure 10] FIG. 10(A) is a diagram showing an example of a prompt input to the customer service level determination model, and FIG. 10(B) is a diagram showing an example of the output of the customer service level determination model. [Figure 11] FIG. 11(A) is a diagram showing an example of a prompt input to the customer service evaluation model, and FIG. 11(B) is a diagram showing an example of the output of the customer service evaluation model. [Figure 12] FIG. 12 is a diagram showing an example of the memory map of the RAM of the support device shown in FIG. 2. [Figure 13] FIG. 13 is a diagram showing an example of the specific content of the data storage area of the memory map shown in FIG. 12. [Figure 14] FIG. 14 is a diagram showing an example of the memory map of the RAM of the operator terminal shown in FIG. 3. [Figure 15] FIG. 15 is a diagram showing an example of the memory map of the RAM of the customer terminal shown in FIG. 4. [Figure 16] FIG. 16 is a flowchart showing a part of an example of the information processing of the CPU of the support device shown in FIG. 2. [[ID=!22]] [Figure 17] FIG. 17 is another part of an example of the information processing of the CPU of the support device shown in FIG. 2, and is a flowchart following FIG. 16. [Figure 18] FIG. 18 is a flowchart showing a first part of an example of the dialogue control process of the CPU of the operator terminal shown in FIG. 3. [Figure 19] FIG. 19 is a second part of an example of the dialogue control process of the CPU of the operator terminal shown in FIG. 3, and is a flowchart following FIG. 18. [Figure 20] [[ID=3!3]]FIG. 20 is a third part of an example of the dialogue control process of the CPU of the operator terminal shown in FIG. 3, and is a flowchart following FIG. 19. [Figure 21] FIG. 21 is a flowchart showing an example of the dialogue control process of the CPU of the customer terminal shown in FIG. 4.

BEST MODE FOR CARRYING OUT THE INVENTION

[0016] Referring to Figure 1, the information processing system 10 in this embodiment includes a support device 12, which is communicatively connected to an operator terminal 16 and a customer terminal 18 via a network 14. The support device 12 is also communicatively connected to a progress determination model 12a, a customer service level determination model 12c, a customer service content determination model 12b, and a customer service evaluation model 12d. However, some or all of the progress determination model 12a, customer service level determination model 12c, customer service content determination model 12b, and customer service evaluation model 12d may be included in the support device 12, or they may be communicatively connected via the network 14.

[0017] In this embodiment, one support device 12, one operator terminal 16, and one customer terminal 18 are shown, but in reality, multiple support devices 12, multiple operator terminals 16, and multiple customer terminals 18 may be connected to the network 14. Therefore, a connection is established between one customer terminal 18 used by a particular customer among the multiple customer terminals 18 and one operator terminal 16 used by the operator who is assisting this customer, and further, a connection is established between one support device 12 that assists this operator and the operator terminal 16.

[0018] The support device 12 is an information processing device, and is, for example, a PC or a server. In this embodiment, the support device 12 uses artificial intelligence (hereinafter referred to as "AI") to support the operator who is dealing with a customer. Briefly, when the operator is dealing with a customer, the support device 12 notifies the operator of what the operator should say to the customer or matters related to that content.

[0019] Network 14 consists of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network can include public telephone networks, mobile phone networks, wired LANs, wireless LANs, CATV (Cable Television), etc.

[0020] The operator terminal 16 is another information processing device, such as a PC or a server. The operator terminal 16 is used by a service provider or operator (hereinafter simply referred to as "operator") who provides voice-based customer service and interacts with customers.

[0021] In this embodiment, a dialogue screen (see Figure 5) including a captured image of the customer being served or interacted with by the operator is displayed on the display device 60 of the operator terminal 16, and the operator interacts with the customer via the dialogue screen. The operator terminal 16 also outputs instructions or advice from the support device 12 and conveys these instructions or advice to the operator.

[0022] In this embodiment, we will describe the case where a voice response service is provided, but a text response service can also be provided. Furthermore, the operator or customer may be allowed to choose between a voice response service or a text response service.

[0023] The customer terminal 18 is another information processing device, and may include, for example, a PC, smartphone, or tablet device. The customer terminal 18 is owned by and used by the customer. However, the customer terminal 18 may also be a terminal installed in the store where the information processing system 10 is applied and used by the customer.

[0024] In this embodiment, a dialogue screen (see Figure 5) including an image corresponding to the avatar of the operator who interacts with the customer (hereinafter referred to as "avatar image") is displayed on the display device 90 of the customer terminal 18, and the customer interacts with the operator through the dialogue screen. In other words, the customer uses the interaction service (or dialogue service) provided by the operator terminal 16.

[0025] The progress determination model 12a is a language model for determining the progress of an interaction when an operator is interacting with a customer. For example, a large-scale language model such as ChatGPT can be used.

[0026] The customer service content determination model 12b is a language model used to determine whether there are any problems with the interaction with a customer or whether any interaction has been forgotten. For example, a large-scale language model such as ChatGPT can be used.

[0027] The customer service level determination model 12c is a language model for determining the level of customer service provided by an operator in their interactions with customers. For example, a large-scale language model such as ChatGPT can be used.

[0028] The customer service evaluation model 12d is a language model that evaluates how an operator interacts with customers, and as an example, it can use large-scale language models such as ChatGPT.

[0029] Figure 2 is a block diagram showing an example of the electrical configuration of the support device 12 shown in Figure 1. As shown in Figure 2, the support device 12 includes a CPU 20, which is connected to a RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26 via an internal bus.

[0030] The CPU 20 is responsible for the overall control of the support device 12. However, instead of the CPU 20, a System-on-a-chip (SoC) that includes multiple functions such as CPU functionality and GPU (Graphics Processing Unit) functionality may be provided.

[0031] RAM22 is the main memory and is used as the work area or buffer area for the CPU20. Although not shown in the diagram, the support device 12 is provided with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the operating system, middleware, and various application programs are stored in this auxiliary storage device.

[0032] Communication I / F24 is a wired interface for sending and receiving control signals and data to and from an external computer, such as an operator terminal 16, via the network 14 under the control of the CPU 20. However, a wireless interface for connecting to a wireless LAN can also be used as Communication I / F24.

[0033] Input / output interface 26 is connected to an input device 28 and a display device 30. The input device 28 is a keyboard and a computer mouse. A touch panel may also be provided. The display device 30 is, for example, a liquid crystal display.

[0034] Furthermore, the input / output interface 26 outputs operation data (or operation information) received from the input device 28 to the CPU 20, and also outputs image data generated by the CPU 20 to the display device 30, causing the display device 30 to display a screen or image corresponding to the image data.

[0035] Note that the electrical configuration of the support device 12 shown in Figure 2 is just one example and is not limited to that configuration.

[0036] Figure 3 is a block diagram showing an example of the electrical configuration of the operator terminal 16 shown in Figure 1. As shown in Figure 3, the operator terminal 16 includes a CPU 50, which is connected to the RAM 52, communication I / F 54, and input / output I / F 56 via an internal bus.

[0037] The CPU 50 is responsible for the overall control of the operator terminal 16. However, instead of the CPU 50, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0038] RAM 52 is the main memory and is used as the work area or buffer area for the CPU 50. Although not shown in the diagram, the operator terminal 16 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the operating system, middleware, and various application programs are stored in this auxiliary storage device.

[0039] Communication I / F 54 is a wired interface for sending and receiving control signals and data between the support device 12 and external computers such as customer terminals 18 via the network 14 under the control of the CPU 50. However, a wireless interface for connecting to a wireless LAN can also be used as the communication I / F 54.

[0040] The input / output interface 56 is connected to an input device 58, a display device 60, a microphone 62, a speaker 64, and a camera 66. However, in this embodiment, a headset equipped with a microphone 62 and a speaker 64 is used to convey the support content (hereinafter referred to as "support content") to the operator by voice.

[0041] The input device 58 is a keyboard and a computer mouse. A touch panel may also be provided. The display device 60 is, for example, a liquid crystal display.

[0042] Furthermore, the input / output interface 56 outputs operation data (or operation information) input from the input device 58 to the CPU 50, and also outputs image data generated by the CPU 50 to the display device 60, causing the display device 60 to display a screen or image corresponding to the image data. However, image data received from an external computer (for example, the customer terminal 18) may also be output by the CPU 50. As an example, the image data received from the customer terminal 18 is captured image data including the customer (hereinafter referred to as "customer image data").

[0043] Furthermore, the input / output interface 56 converts the operator's voice detected by the microphone 62 into digital audio data (hereinafter referred to as "operator audio data") and outputs it to the CPU 50, converts the audio data output by the CPU 50 into an analog audio signal and outputs it from the speaker 64, and outputs image data including the operator captured (detected) by the camera 66 (hereinafter referred to as "operator image data") to the CPU 50. However, in this embodiment, the audio data output by the CPU 50 is, for example, the customer's audio data received from the customer terminal 18 (hereinafter referred to as "customer audio data"). Also, the image including the operator can be a still image or a moving image.

[0044] Note that the electrical configuration of the operator terminal 16 shown in Figure 3 is just one example and is not limited to this configuration.

[0045] Figure 4 is a block diagram showing an example of the electrical configuration of the customer terminal 18 shown in Figure 1. As shown in Figure 4, the customer terminal 18 includes a CPU 80, which is connected to RAM 82, communication I / F 84, and input / output I / F 86 via an internal bus.

[0046] The CPU 80 is responsible for the overall control of the customer terminal 18. However, instead of the CPU 80, an SoC (System on a Chip) that includes multiple functions such as CPU and GPU functions may be provided.

[0047] RAM 82 is the main memory and is used as the work area or buffer area for the CPU 80. Although not shown in the diagram, the customer terminal 18 is equipped with an HDD and ROM as auxiliary storage devices. However, non-volatile memory such as an SSD may be used instead of the HDD, or in addition to the HDD. Various software such as the operating system, middleware, and various application programs are stored in this auxiliary storage device.

[0048] The communication interface 84 is a wired interface for sending and receiving control signals and data to and from an external computer, such as an operator terminal 16, via the network 14 under the control of the CPU 80. However, a wireless interface for connecting to a wireless LAN can also be used as the communication interface 84.

[0049] The input / output interface 86 is connected to an input device 88, a display device 90, a microphone 92, a speaker 94, and a camera 96.

[0050] The input device 88 is a keyboard and a computer mouse. A touch panel may also be provided. However, if a smartphone or tablet is used as the customer terminal 18, the input device 88 is a touch panel and hardware buttons. The display device 90 is, for example, a liquid crystal display.

[0051] Furthermore, the input / output interface 86 outputs operation data (or operation information) received from the input device 88 to the CPU 80, and also outputs image data generated by the CPU 80 to the display device 90, causing the display device 90 to display a screen corresponding to the image data. However, image data received from an external computer (for example, the operator terminal 16) may also be output by the CPU 80.

[0052] Furthermore, the input / output interface 86 converts the operator's voice detected by the microphone 92 into digital audio data and outputs it to the CPU 80, converts the audio data output by the CPU 80 into an analog audio signal and outputs it from the speaker 94, and outputs image data including the customer captured (detected) by the camera 96, i.e., customer image data, to the CPU 80. However, in this embodiment, the audio data output from the CPU 80 is customer audio data received from the customer terminal 18. Also, the image including the customer can be a still image or a moving image.

[0053] Note that the electrical configuration of the customer terminal 18 shown in Figure 4 is just one example and is not limited to this configuration.

[0054] Furthermore, if the customer terminal 18 is a smartphone, it is equipped with a call circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this embodiment, such calls are not made, so they are omitted from the illustration.

[0055] The operation of the information processing system 10 in this embodiment will be described below. In this information processing system 10, when a customer accesses a website that provides a predetermined service, such as a general company site or an electronic trading site, using the customer terminal 18, or when a customer requests assistance from an operator using a customer terminal 18 installed in a store, the customer terminal 18 performs (or starts) dialogue control processing. When the customer terminal 18 starts dialogue control processing, an operator is matched by a matching server, such as a server operating the website or a server that can communicate with the customer terminal 18 installed in a store. The operator's operator terminal 16 then starts dialogue control processing and establishes a connection with the customer terminal 18. However, prior to assisting the customer, the operator establishes a connection with the support device 12 and requests the support device 12 to start providing assistance.

[0056] Figure 5 shows an example of the operator-side dialogue screen 100 displayed on the display device 60 of the operator terminal 16. As shown in Figure 5, the operator-side dialogue screen 100 has a display area 102 in the center of the screen, and button images 104 are provided below the display area 102.

[0057] Display area 102 is the area for outputting customer image data. The image of the customer being assisted by the operator is displayed. Button image 104 is an icon for ending the interaction with the customer. When button image 104 is turned on, a notification that the interaction has ended is sent to the customer terminal 18.

[0058] Figure 6 shows an example of a customer-side dialogue screen 200 displayed on the display device 90 of the customer terminal 18 when a customer and an operator interact. The customer-side dialogue screen 200 may be displayed overlaid on a screen of a predetermined service, included as part of a screen of a predetermined service, or displayed in place of a screen of a predetermined service. However, when displayed as part of a screen of a predetermined service, only the avatar image 202 may be displayed.

[0059] The customer-facing dialogue screen 200 includes an avatar image (hereinafter referred to as "avatar image") 202 representing the operator. In the example shown in Figure 6, the avatar image 202 is an image of a character modeled after a human woman. Also, in the example shown in Figure 6, the avatar image 202 on the customer-facing dialogue screen 200 is an image of the upper body of the character. For example, the background is filled with a predetermined color.

[0060] However, avatar image 202 can also be an image of a character resembling an animal or robot, an anime character, a game character, etc. Alternatively, avatar image 202 may be an image of the entire character. In this embodiment, since the avatar may be made to move in accordance with the operator's movements, the avatar needs to have a skeletal structure similar to that of a human.

[0061] The avatar image 202 or the customer's dialogue screen 200 is displayed on the display device 90 when the dialogue control process starts. However, at the beginning of the dialogue control process, the avatar image data has not yet been received from the operator terminal 16, so the avatar image 202 is not displayed.

[0062] When an operator is interacting with a customer and the customer and operator are conversing, avatar image data corresponding to the avatar image 202 is generated in the operator terminal 16. The avatar then performs actions corresponding to the operator image captured by the camera 66 of the operator terminal 16. To this end, the operator terminal 16 performs a motion detection process on the operator image and generates avatar image data for displaying the avatar image 202 that performs actions corresponding to the operator image. This avatar image data is transmitted to the customer terminal 18, and upon receiving the avatar image data, the customer terminal 18 generates image data for displaying the customer's dialogue screen 200 based on the avatar image data, and the customer's dialogue screen 200 is displayed on the display device 90 based on this image data.

[0063] However, the dialogue screen 200 may display an image of the operator themselves instead of the avatar image 202. Alternatively, the operator terminal 16 may generate action data corresponding to the operator's image and send it to the customer terminal 18, where the customer terminal 18 may generate the avatar image data.

[0064] Furthermore, the operator terminal 16 generates voice data, i.e., operator voice data, for controlling the avatar's speech (voice output) based on the operator's voice. The operator voice data may be the operator's voice data, or it may be data obtained by converting the operator's voice into the avatar's voice.

[0065] The operator terminal 16 transmits the generated operation data and operator voice data to the customer terminal 18. The operator terminal 16 also transmits the generated operator voice data to the support device 12. The customer terminal 18 operates the avatar according to the received operation data and generates and displays (or updates) the avatar image 202. Thus, the avatar is operated. In other words, the customer-side dialogue screen 200 shown in Figure 6 is displayed (or updated). The customer terminal 18 also outputs the received operator voice data to the speaker 34. At this time, the avatar image 202 is lip-synced.

[0066] When an operator is assisting a customer and the customer and operator are in conversation, the customer terminal 18 detects the customer's voice through the microphone 92 and transmits voice data corresponding to the detected voice, i.e., customer voice data, to the operator terminal 16. The operator terminal 16 outputs the received customer voice data to the speaker 64. Therefore, the operator can hear the customer's voice. The operator terminal 16 also transmits the received customer voice data to the support device 12.

[0067] Furthermore, when a customer and an operator are interacting, the camera 96 ​​on the customer terminal 18 captures a photograph of the customer, and the image data corresponding to the captured image of the customer (a moving image in this embodiment), i.e., the customer image data, is transmitted to the operator terminal 16. The operator terminal 16 outputs the received customer image data to the display device 60. In other words, the operator's dialogue screen 100, as shown in Figure 5, is displayed (or updated). Therefore, the operator can see the customer's image. However, the customer terminal 18 can also prevent the transmission of customer image data to the operator terminal 16 by turning off the camera 96 ​​function.

[0068] Thus, while operators interact with customers online, novice operators who have never interacted with customers before, or those unfamiliar with customer interaction, may undergo training using AI-powered avatars. In this case, novice operators or those unfamiliar with customer interaction will only interact with customers after training. In other words, time is required for training. Furthermore, even operators who are accustomed to interacting with customers may forget to convey important information or convey incorrect information.

[0069] Therefore, in this embodiment, when an operator is interacting with a customer, the operator is supported by instructions or advice on what to say to the customer, as needed.

[0070] Instructions or advice to the operator are predetermined according to the level of support. The level of support is set in multiple stages. For example, the support level can be set in three stages: high, medium (normal), and low. In this example, a higher support level indicates a greater degree of support.

[0071] The support level is specified or set by the operator themselves when they begin assisting a customer. Furthermore, if the operator allows automatic adjustment of the support level, the system will determine the operator's customer service level (i.e., their proficiency in customer service) during the interaction and automatically adjust (raise or lower) the support level according to the determined customer service level. In this embodiment, a higher customer service level indicates greater proficiency in customer service.

[0072] Customer service levels are also set in multiple stages. For example, customer service levels are set in four stages. Customer service levels are represented by numbers from 1 to 4, where "1" is the lowest and "4" is the highest, with the level of customer service being set according to the size of the number. In this example, if the customer service level is "1", the support level is set to "high". If the customer service level is "2" or "3", the support level is set to "medium". If the customer service level is "4", the support level is set to "low".

[0073] However, this is just one example, and the support level may be set to "High" when the customer service level is "1" or "2", the support level may be set to "Medium" when the customer service level is "3", and the support level may be set to "Low" when the customer service level is "4".

[0074] In this embodiment, when the support level is "high," the operator is notified of the content to be spoken in its entirety. When the support level is "medium," it is not necessary to convey the entire content to be spoken, so the operator is notified of matters related to the content to be spoken, in this embodiment, a short sentence or keyword. When the support level is "low," it is not necessary to convey the entire content to be spoken, nor is it necessary to convey matters related to the content to be spoken each time the operator speaks. Therefore, the operator is notified of matters related to the content to be spoken, in this embodiment, a short sentence or keyword, only if the operator forgets to convey the content to be spoken, repeats the same content, or conveys incorrect content—that is, only if there is a problem with the operator's speech (response). Specifically, the operator is notified of content that has been forgotten, repeated, or incorrect.

[0075] Furthermore, the content to be spoken is pre-set for each customer service task, according to the progress of the customer service procedure and for each support level. However, when the support level is "high," a short sentence or keyword is not pre-set, as it is only notified to the operator if there is a problem with the operator's speech, in case of something that has been forgotten, repeated, or incorrectly stated. In this embodiment, the progress of customer service by the operator is determined based on the dialogue history, and the content to be spoken to notify the operator is determined based on the determination result and the support level. The customer service procedure is pre-set according to the type of customer service task.

[0076] For example, if the customer service task involves handling the exchange or repair of a defective product, the procedure for that customer service would be: 1. Greeting and introducing oneself, 2. Hearing about the nature of the malfunction, 3. Confirming the purchase date and warranty period, 4. Explaining whether repair or exchange is possible, 5. Guiding the customer to fill out necessary documents, 6. Explaining the repair period and procedures, and 7. Expressing gratitude and seeing the customer off.

[0077] Figure 7 shows an example of a conversation history between a customer and an operator. As shown in Figure 7, the customer first says, "The headset I bought from you is broken," to which the operator replies, "I see, I'm very sorry. When did you purchase it?" Next, the customer says, "I bought it five years ago. But the sound keeps cutting out. I treated it like new!" to which the operator replies, "The sound keeps cutting out, you say. What exactly is happening?" Furthermore, the customer says, "Yes, that's right. The sound keeps cutting out intermittently. It's just been opened, so this shouldn't be happening, right? Exchange it for a new one." to which the operator replies, "I'm very sorry. When did you purchase it?" Then the customer said, "I told you it was five years ago! I don't want to hear any excuses like that," to which the operator replied, "I apologize for the inconvenience. Well, since it was five years ago, the warranty period has expired, so replacement is difficult, and you will need to either have it repaired or purchase a new one."

[0078] If the customer service task involves handling the replacement or repair of a faulty product, a prompt like the one shown in Figure 8(A) is input to the progress determination model 12a. As shown in Figure 8(A), the progress determination model 12a is input with an instruction (i.e., a command) that says, "You are the supervisor of the customer service staff. You will input the operator's utterances in order, so compare them with the following procedure for handling the replacement or repair of a faulty product, determine the current progress, and respond with the determination result," along with a prompt (prompt for progress determination) that describes the procedure for handling the replacement or repair of a faulty product. Therefore, the progress determination model 12a functions as an AI that determines the progress of the customer service procedure for a customer service task that involves handling the replacement or repair of a faulty product.

[0079] Note that the prompts shown in Figure 8(A) are examples only and should not be considered limiting. They should be modified as appropriate depending on the customer service procedure and the type of customer service task. This also applies to other prompts entered into the other models 12b, 12c, and 12d.

[0080] Furthermore, when the operator is interacting with a customer, each time the operator speaks, the content of the speech is recognized by speech recognition, transcribed into text, and the text of the speech is input into the progress determination model 12a, which then outputs the progress of the customer service interaction.

[0081] Therefore, as shown in Figure 8(B), when the first utterance from the operator in the above dialogue history, "I see, I'm sorry. When did you purchase this?", is input to the progress determination model 12a, "3. Confirmation of purchase date and warranty period" is output. The utterances from the second and subsequent operators are also sequentially input to the progress determination model 12a, and the determination result is output.

[0082] While the order of customer service procedures may vary when the operator is interacting with a customer, in this embodiment, the purpose is to provide support according to the progress, so whether the customer service is proceeding according to the procedure is irrelevant to the operator's support.

[0083] Similarly, the progress of customer service is determined for other types of customer service tasks as well. As mentioned above, the procedures differ depending on the type of customer service task, so customer service tasks are pre-configured according to the role of the operator supported by the support device 12, and the prompts to be input into the progress determination model 12a are determined according to the customer service task performed by the operator. In other words, multiple prompts to be input into the progress determination model 12a are provided for each type of customer service task.

[0084] When the support device 12 acquires the progress of customer service, it provides support to the operator according to the progress and support level. However, if the operator's support level is "low," the support device 12 will only provide support to the operator if there is an error in the operator's utterance or if the operator has forgotten to say something, i.e., if there is a problem with the operator's customer service (content of utterance). In this embodiment, if there is a problem with customer service, the operator is notified of that problem.

[0085] Whether there are problems with the operator's customer service is determined by the customer service content judgment model 12b. As shown in Figure 9(A), the customer service content judgment model 12b receives an instruction (i.e., a command) that says, "You are the supervisor of the customer service staff. You will input the conversation between the customer and the operator in order. Compare it with the checklist below, and if there are any problems based on the flow of the conversation, point them out in 10 characters or less. If there are no particular issues or no problems, answer 'No problems'." Along with this instruction, a prompt (a prompt for customer service content judgment) containing a checklist is input. Therefore, the customer service content judgment model 12b functions as an AI that determines whether there are problems with the operator's customer service. As an example, the checklist may include: 1. State your name, 2. Confirm the purchase date, and 3. Do not exchange products whose warranty has expired.

[0086] Furthermore, when an operator interacts with a customer, the conversation between the customer and the operator, which has been transcribed to track the progress, is sequentially input into the customer service content judgment model 12b, and for each conversation (i.e., interaction) between the customer and the operator, a judgment result is output regarding whether there is a problem with the operator's customer service.

[0087] Therefore, as shown in Figure 9(B), if the dialogue between the customer and the operator is "The headset I bought from you is broken," and the operator responds "I see, I'm very sorry. When did you purchase this?", then the system will output that the customer should state their name.

[0088] Next, if the conversation between the customer and the operator is, "I bought it five years ago, but the sound keeps cutting out. I treated it like new!" and the operator replies, "So the sound is cutting out. What exactly is happening?", then the system will output a message indicating that the warranty period should be checked.

[0089] Next, if the conversation between the customer and the operator is, "Yes, that's right. The sound keeps cutting out. It's brand new, so this shouldn't be happening, right? Please replace it with a new one," and the operator responds, "We apologize. When did you purchase this item?", then the system will output that the purchase date has been confirmed, meaning that the purchase date has been confirmed multiple times.

[0090] Furthermore, if the dialogue between the customer and the operator is, "I told you it was 5 years ago! I don't want to hear that excuse," to which the response is, "I apologize for the inconvenience. Well, since it was 5 years ago, the warranty period has expired, so replacement is difficult, and you will need to either have it repaired or purchase a new one," then "No problem" will be output.

[0091] In Figure 9(B), the input and output are displayed for all customer-operator conversations included in the conversation history. However, since customer-operator conversations are input sequentially, the support device 12 obtains an output for each conversation (i.e., exchange) and determines whether support is needed for operators whose support level is "low".

[0092] Similarly, the content of customer service is determined for other types of customer service tasks. As mentioned above, the customer service procedure differs depending on the type of customer service task, and therefore the checklist also differs depending on the type of customer service task. For this reason, the prompt to be input into the customer service content determination model 12b is determined according to the customer service task performed by the operator. In other words, multiple prompts for input into the customer service content determination model 12b are provided for each type of customer service task.

[0093] Furthermore, in this embodiment, when an operator interacts with a customer, the operator's customer service level is determined. As mentioned above, the support level is set by the operator when they begin interacting with a customer. However, if the customer service level changes during the interaction, the system is designed to determine the customer service level during the interaction so that the customer can receive support at a level corresponding to the changed customer service level. However, even if the customer service level changes during the interaction, the support level will not be changed for operators who do not wish to receive support at a level corresponding to the changed customer service level.

[0094] The customer service level is determined by the customer service level determination model 12c. As shown in Figure 10(A), the customer service level determination model 12c is input with the instruction (i.e., command statement) "You are the supervisor of the customer service staff. Determine the operator's customer service level from the given conversation history. Output the appropriate level. Also, please tell me the reason why you determined that the level is appropriate," along with a prompt (prompt for customer service level determination) that lists four levels of customer service. Therefore, the customer service level determination model 12c functions as an AI that determines the operator's customer service level. As an example, the customer service levels include: Level 1: Able to have a conversation, Level 2: Able to have a polite conversation throughout, Level 3: Able to hear the customer's problems, and Level 4: Able to solve the customer's problems.

[0095] Furthermore, when an operator interacts with a customer, the conversation history between the customer and the operator, which has been transcribed to track the progress, is input into the customer service level determination model 12c, and the determination result of the operator's customer service level and the reason for it are output.

[0096] Similarly, the service level is determined for other types of customer service tasks. When determining the service level, the service level to be determined is the same even if the type of customer service task is different, so only one type of prompt is input to the customer service level determination model 12c. However, this is just an example, and different prompts may be input to the customer service level determination model 12c depending on the type of customer service task.

[0097] Figure 10(B) shows an example of the output obtained when all of the dialogue history shown in Figure 7 is input into the customer service level determination model 12c. Specifically, the customer service level is listed as "1 (Level 1)," and the reason given is: "The operator is engaging in dialogue, but it lacks politeness, the questions are repetitive, and the customer's issues are not accurately understood. Furthermore, the proposed solutions are insufficient."

[0098] In this embodiment, the customer service level determination result and the reason for it are not presented to the operator, so the operator may choose not to request output regarding the reason for determining the customer service level. In other words, the last sentence of the prompt command may be deleted.

[0099] Furthermore, in this embodiment, once the operator finishes interacting with a customer, their customer service is evaluated. However, the evaluation determines whether or not the operator's utterances were problematic in terms of customer service, and also points out the reasons why they were deemed problematic. Therefore, by reviewing the evaluation, the operator can make their interactions with customers more appropriate in the future.

[0100] Customer service is evaluated by the customer service evaluation model 12d. As shown in Figure 11(A), the customer service evaluation model 12d receives an instruction (i.e., a command) that reads, "You are the supervisor of the customer service staff. Based on the given customer-operator dialogue history, point out any problematic operator utterances in terms of customer service, along with the reasons. Also, point out any items on the following checklist that have not been achieved. Furthermore, based on the problematic points in the operator's utterances, point out areas for improvement that the operator should focus on." This instruction is accompanied by a prompt containing a checklist (a prompt for customer service evaluation). Therefore, the customer service evaluation model 12d functions as an AI that judges the operator's customer service. As an example, the checklist may include items such as: 1. State your name, 2. Confirm the purchase date, and 3. Do not exchange products whose warranty has expired.

[0101] Furthermore, once the operator finishes interacting with the customer, the entire dialogue history, as shown in Figure 7, is input into the customer service evaluation model 12d, and an evaluation of the operator's customer service is output.

[0102] Specifically, as shown in Figure 10(B), the output includes problems based on the checklist, particularly problematic points in the operator's speech, and improvements based on the problems. Specifically, the problems based on the checklist include: "1. Identifying oneself: The problem is that the operator did not introduce themselves. It is important to state one's name at the beginning to clarify who the customer is speaking with." and "2. Refusal to exchange expired warranty products: The operator eventually explains that an exchange is difficult because the warranty has expired, but the initial response focused too much on confirming the warranty period, which amplified the customer's dissatisfaction. It is justified not to exchange expired warranty products, but this point should have been communicated more carefully at an earlier stage."

[0103] Furthermore, the report identifies two particularly problematic aspects of the operator's speech: "The repeated question, 'I'm sorry, when did you plan to purchase it?', wastes the customer's time," and "The final statement, 'Well, you will need to either have it repaired or purchase a new one,' may sound irresponsible to the customer. You should be more polite and offer specific solutions."

[0104] Furthermore, based on the issues identified, the following points for improvement will be output as part of the operator's evaluation of the customer service: "- Introduce yourself first," "- Immediately inform the customer that the product is outside the warranty period after confirming the purchase date," and "- Carefully provide specific solutions, such as repair procedures or discount options for purchasing a new product."

[0105] This evaluation, along with the level of support determined during the interaction, is communicated to the operator. In other words, a recommended level of support is provided to the operator. Therefore, the operator can set the recommended level of support when interacting with a customer in the future.

[0106] Similarly, customer service evaluations are also performed for other types of customer service tasks. As mentioned above, since the checklist differs depending on the type of customer service task, the prompts to be input into the customer service evaluation model 12d are determined according to the customer service task performed by the operator. In other words, multiple prompts for input into the customer service evaluation model 12d are provided for each type of customer service task.

[0107] Figures 12 and 13 show an example of the memory map 300 of the RAM 22 built into the support device 12 shown in Figure 2. As shown in Figure 12, the RAM 22 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores the information processing program executed by the support device 12 in this embodiment.

[0108] The information processing program executed by the support device 12 includes a main processing program 302a, a communication program 302b, a support program 302c, a speech recognition program 302d, a progress determination program 302e, a customer service level determination program 302g, a support level adjustment program 302h, a customer service evaluation program 302i, and a result notification program 302j.

[0109] The main processing program 302a is a program for executing the main routine of information processing of the support device 12 in this embodiment.

[0110] The communication program 302b is a program for communicating (sending and receiving data, etc.) with an external device, in this embodiment, the operator terminal 16.

[0111] Support program 302c is a program designed to support operators based on their level of assistance, the progress of customer service, and the content of customer service (i.e., whether there are any problems with the content of customer service).

[0112] The speech recognition program 302d is a program for recognizing the voice data of operators and customers and transcribing it into text. The voice data of operators and customers is transcribed into text, and the text is recorded chronologically, thereby creating a dialogue history.

[0113] The progress determination program 302e is a program that takes the current utterance of the operator or customer as input to the progress determination model 12a, which receives prompts for determining the progress of customer service according to the customer service task, and obtains a determination result regarding the progress of the customer service being performed by the operator.

[0114] The customer service content determination program 302f is a program that sequentially inputs the dialogue between the customer and the operator into the customer service content determination model 12b, which has been input with prompts to determine the content of the customer service, and obtains a determination result for each dialogue regarding whether there are any problems with the content of the customer service being provided by the operator.

[0115] The customer service level determination program 302g is a program that takes the customer service level determination model 12c, which has been given prompts to determine the customer service level, as input of the conversation history between the customer and the operator up to the present, and obtains the determination result of the operator's customer service level.

[0116] The support level adjustment program 302h is a program that, when automatic adjustment of the support level is permitted, determines the support level according to the operator's customer service level, which is obtained according to the customer service level determination program 302g, and adjusts the current support level.

[0117] The customer service evaluation program 302i is a program that inputs the entire conversation history between the customer and the operator into the customer service evaluation model 12d, which has been given prompts to evaluate the operator's customer service, in order to obtain an evaluation of the operator's customer service.

[0118] The result notification program 302j is a program that notifies the operator of the customer service evaluation obtained according to the customer service evaluation program 302i, and the support level corresponding to the customer service level ultimately determined by the interaction.

[0119] Although not shown in the diagram, the program storage area 302 also stores other programs for executing the information processing program of the support device 12.

[0120] Figure 13 is a diagram showing an example of the specific contents of the data storage area 304 of the RAM 22 shown in Figure 12. As shown in Figure 13, the data storage area 304 stores support content dictionary data 304a, prompt data 304b, dialogue history data 304c, progress data 304d, customer service level data 304e, support level data 304f, support voice data 304g, customer voice data 304h, operator voice data 304i, evaluation data 304j, and permission flag 304k, among others.

[0121] The support content dictionary data 304a is a dictionary data that describes the support content to the operator in text. However, the support content is described for each customer service task, corresponding to the customer service procedure and the support level. In this embodiment, when the support level is "high," all of what should be said is described as support content, and when the support level is "medium," the content that should be said is described as a short sentence or keyword. Therefore, the support content dictionary data 304a is a table data in which, when the support level is high, the full text of what should be said is described for each customer service procedure, and when the support level is medium, the content that should be said is described as a short sentence or keyword for each customer service procedure. However, such a table is prepared for each type of customer service task.

[0122] The prompt data 304b consists of data for prompts to be input to the progress determination model 12a, the customer service content determination model 12b, the customer service level determination model 12c, and the customer service evaluation model 12d. However, the prompt data to be input to the progress determination model 12a, the customer service content determination model 12b, and the customer service evaluation model 12d is provided separately for each type of customer service task.

[0123] Dialogue history data 304c is data recorded chronologically by transcribing the voice data of both the operator and the customer into text when the operator is interacting with the customer.

[0124] The progress data 304d is data about the progress of customer service performed by the operator, obtained from the progress determination model 12a according to the progress determination program 302e.

[0125] The customer service level data 304e is data indicating the operator's customer service level, obtained from the customer service level determination model 12c according to the customer service level determination program 302g.

[0126] The support level data 304f is data indicating the support level set by the operator, or the support level adjusted according to the support level adjustment program 302h.

[0127] Support voice data 304g is voice data of a virtual supporter assisting the operator, and is data obtained by converting the text of the support content into synthesized speech. The synthesized speech is the voice of the virtual supporter.

[0128] However, in this embodiment, the virtual supporter only generates audio of the support content and does not physically exist in the real world. The virtual supporter can also be made to physically exist in the real world by using computer graphics (CG) or a robot. In such cases, the CG of the virtual supporter is displayed on part of the operator's dialogue screen 100, or the virtual supporter robot is placed near the operator. Even when the virtual supporter is implemented using CG or a robot, the audio of the support content is output from the speaker 64 of the headset worn by the operator.

[0129] Furthermore, the support device 12 can send text of the support details to the operator terminal 16, and the operator terminal 16 can convert the text of the support details into synthesized speech. The customer voice data 304h is the customer's voice data sent (or transferred) from the operator terminal 16.

[0130] Operator voice data 304i is operator voice data transmitted from operator terminal 16.

[0131] Evaluation data 304j is data evaluating the operator's customer service in this instance, obtained from the customer service evaluation model 12d according to the customer service evaluation program 302i.

[0132] The permission flag 304k indicates whether to allow the operator to automatically adjust the assistance level. If the operator has allowed the assistance level to be automatically adjusted, the permission flag 304k is turned on. Conversely, if the operator has not allowed the assistance level to be automatically adjusted, the permission flag 304k is turned off.

[0133] Although not shown in the diagram, the data storage area 304 stores other data necessary for performing information processing, and also contains timers (counters) and flags necessary for performing information processing.

[0134] Figure 14 shows an example of the memory map 400 of the RAM 52 built into the operator terminal 16 shown in Figure 3. As shown in Figure 14, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores an interaction control program, which is an example of an information processing program executed in the operator terminal 16 of this embodiment.

[0135] The interactive control program executed on the operator terminal 16 includes a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, a shooting program 402f, an operation data generation program 402g, a sound detection program 402h, a sound output program 402i, and a support content notification program 402j, among others.

[0136] The main processing program 402a is a program for executing the main routine of the interaction control processing of the operator terminal 16 in this embodiment.

[0137] The operation detection program 402b is a program for detecting operation data 404a input from the input device 58 in accordance with the operator's actions and storing it in the data storage area 404.

[0138] The communication program 402c is a program for communicating (sending and receiving data, etc.) with external devices, in this embodiment, the support device 12 and the customer terminal 18.

[0139] The image generation program 402d is a program for generating image data corresponding to all or part of a screen or image to be displayed on the display device 60, using image generation data 404b. However, when generating image data for the operator's interaction screen 100, customer image data 404d is also used.

[0140] The image output program 402e is a program for outputting image data generated according to the image generation program 402d to the display device 60. Therefore, a screen or image corresponding to the image data is displayed on the display device 60.

[0141] The shooting program 402f is a program that causes the camera 66 to perform shooting processing and stores the operator image data 404f input from the camera 66 in the data storage area 404.

[0142] The motion data generation program 402g is a program that recognizes the operator's actions from the operator image data 404f and generates avatar motion data 404g to make the avatar move.

[0143] The sound detection program 402h is a program that detects the operator's voice input from the microphone 62 and temporarily stores the operator's voice data 404h corresponding to the detected voice in the data storage area 404.

[0144] The sound output program 402i is a program for outputting customer voice data 404e received from the customer terminal 18 to the speaker 64. Therefore, the sound corresponding to the customer voice data 404e is output from the speaker 64.

[0145] The support content notification program 402j is a program for notifying the operator of the support content received from the support device 12. In this embodiment, the operator is notified by outputting support voice data 404i, but the operator can also be notified by displaying text, or both methods may be employed. When displaying text, the support device 12 sends the text of the support content to the operator terminal 16.

[0146] Although not shown in the diagram, the program storage area 402 also stores other programs for executing the dialogue control program of the operator terminal 16.

[0147] The data storage area 404 stores operation data 404a, image generation data 404b, setting information data 404c, customer image data 404d, customer voice data 404e, operator image data 404f, avatar movement data 404g, operator voice data 404h, and support voice data 404i, among others.

[0148] Operation data 404a is data of the operator's operation detected according to the operation detection program 402b.

[0149] Image generation data 404b is data for generating a screen or image to be displayed on the display device 60 of the operator terminal 16.

[0150] Configuration information data 404c contains data about the support level set by the operator and whether automatic adjustment of the support level is permitted.

[0151] Customer image data 404d is customer image data received from customer terminal 18. Customer voice data 404e is customer voice data received from customer terminal 18.

[0152] Operator image data 404f is data about images taken of the operator. However, if the operator is not within the shooting range of camera 66, the operator will not be included in the captured image.

[0153] Avatar motion data 404g is motion data for the avatar generated based on captured images of the operator, and includes data on facial expressions and mouth movements.

[0154] Operator voice data 404h is data about the voice input by the operator or the voice obtained by converting the operator's input into the avatar's voice. Support voice data 404i is support voice data received from the support device 12.

[0155] Although not shown in the diagram, the data storage area 404 stores other data necessary for executing the dialogue control process, and also contains timers (counters) and flags necessary for executing the dialogue control process.

[0156] Figure 15 shows an example of the memory map 500 of the RAM 82 built into the customer terminal 18. As shown in Figure 15, the RAM 82 includes a program storage area 502 and a data storage area 504. The program storage area 502 stores an interaction control program, which is an example of an information processing program executed in the customer terminal 18 of this embodiment.

[0157] The interaction control program executed on the customer terminal 18 includes a main processing program 502a, an operation detection program 502b, a communication program 502c, an image generation program 502d, an image output program 502e, a shooting program 502f, an avatar control program 502g, a sound detection program 502h, and a sound output program 502i, among others.

[0158] The main processing program 502a is a program for executing the main routine of the dialogue control processing of the customer terminal 18 in this embodiment.

[0159] The operation detection program 502b is a program for detecting operation data 504a input from the input device 88 in accordance with the customer's operation and storing it in the data storage area 504.

[0160] The communication program 502c is a program for communicating with an external device, in this embodiment, the operator terminal 16.

[0161] The image generation program 502d is a program for generating image data corresponding to all or part of a screen or image to be displayed on the display device 90, using image generation data 504b. However, when generating image data corresponding to a customer-side dialogue screen 200 as shown in Figure 6, avatar motion data 504c is also used.

[0162] The image output program 502e is a program for outputting image data generated according to the image generation program 502d to the display device 90. Therefore, a screen or image corresponding to the image data is displayed on the display device 90.

[0163] The shooting program 502f is a program that causes the camera 96 ​​to perform shooting processing and stores the customer image data 504e input from the camera 96 ​​in the data storage area 504.

[0164] The avatar control program 502g is a program for operating the avatar according to the avatar operation data 504c.

[0165] The sound detection program 502h is a program that detects customer voices input from the microphone 92 and stores customer voice data 504f corresponding to the detected voices in the data storage area 504.

[0166] The sound output program 502i is a program for outputting operator voice data 504d received from the operator terminal 16 to the speaker 94. Therefore, the voice corresponding to the operator voice data 504d, or the voice obtained by converting this voice into the avatar's voice, will be output from the speaker 94.

[0167] Although not shown in the diagram, the program storage area 502 also stores other programs for executing the dialogue control program of the customer terminal 18.

[0168] The data storage area 504 stores operation data 504a, image generation data 504b, avatar motion data 504c, operator voice data 504d, customer image data 504e, and customer voice data 504f, among others.

[0169] Operation data 504a is data of customer operations detected according to the operation detection program 502b.

[0170] Image generation data 504b is data used to generate a screen or image to be displayed on the display device 90 of the customer terminal 18. Image generation data 504b also includes data for generating the avatar image 202 for the customer-side dialogue screen 200.

[0171] Avatar motion data 504c is the motion data of the avatar received from the operator terminal 16. Operator voice data 504d is data about the operator's voice received from the operator terminal 16, or the voice obtained by converting the operator's voice into the avatar's voice.

[0172] Customer image data 504e is data about images taken of customers. However, if there are no customers within the shooting range of camera 96, the images will not include customers. Customer audio data 504f is data about the customer's voice detected by microphone 92.

[0173] Although not shown in the diagram, the data storage area 504 stores other data necessary for executing the dialogue control process, and also contains timers (counters) and flags necessary for executing the dialogue control process.

[0174] Figures 16 and 17 are flowcharts showing an example of information processing by the CPU 20 of the support device 12 shown in Figure 2. Although not shown in the diagram, in parallel with information processing, a process for receiving data transmitted from the operator terminal 16 is executed. Dialogue control processing is initiated on the operator terminal 16, and in response to a connection request from the operator, the CPU 20 begins information processing.

[0175] As shown in Figure 16, when the CPU 20 starts processing information, in step S1 it establishes a connection with the operator terminal 16, and in step S3 it determines whether it has received configuration information from the operator terminal 16.

[0176] If the answer in step S3 is "NO," that is, if no configuration information has been received from the operator terminal 16, the process returns to step S3. On the other hand, if the answer in step S3 is "YES," that is, if configuration information has been received from the operator terminal 16, the support level is set in step S5, that is, the support level data 304f included in the configuration information is stored in the data storage area 304 of the RAM 22, and in step S7, it is determined whether automatic adjustment of the support level is permitted in the configuration information.

[0177] If the answer in step S7 is "NO," that is, if automatic adjustment of the support level is not permitted, proceed to step S11. On the other hand, if the answer in step S7 is "YES," that is, if automatic adjustment of the support level is permitted, turn on the permission flag 304k in step S9 and proceed to step S11. It should be assumed that the permission flag 304k is off when information processing begins.

[0178] In step S11, the progress of the customer service is determined. Here, the CPU 20 inputs the latest utterance from the operator into the progress determination model 12a and obtains output regarding the progress of the customer service. However, the progress determination model 12a is pre-programmed with prompts for progress determination according to the type of customer service task.

[0179] Next, in step S13, it is determined whether dialogue support is needed. In this step S13, the CPU 20 asks the customer service content determination model 12b to determine whether the support level is "high" or "medium," and if the support level is "low," whether a problem has occurred in the dialogue between the customer and the operator. The CPU 20 inputs the latest dialogue between the customer and the operator into the customer service content determination model 12b and obtains output indicating whether there is no problem or if a problem has occurred. If the support level is "high" or "medium," it is determined that dialogue support is needed. Also, for operators with a "low" support level, it is determined that dialogue support is needed if a problem has occurred. However, the customer service content determination model 12b is pre-programmed with prompts for determining customer service content according to the type of customer service task.

[0180] If the answer in step S13 is "NO," that is, if dialogue support is not needed, the process proceeds to step S19 as shown in Figure 17. On the other hand, if the answer in step S13 is "YES," that is, if dialogue support is needed, the content of the dialogue support is determined in step S15. Here, the CPU 20 determines the support content based on the progress and support level if the support level is "high" or "medium," and if the support level is "low," it determines the support content to be the issue regarding the operator's utterance output by the customer service content determination model 12b. In the next step, S17, the support voice data corresponding to the text of the support content determined in step S15 is sent to the operator terminal 16, and the process proceeds to step S19. The support voice data is generated by converting the text of the support content into synthesized speech using general-purpose software.

[0181] As shown in Figure 17, step S19 determines whether customer voice data or operator voice data has been received. If the answer in step S19 is "NO," that is, if neither customer voice data nor operator voice data has been received, the process proceeds to step S27.

[0182] On the other hand, if the answer in step S19 is "YES," that is, if customer voice data or operator voice data has been received, then in step S21, the customer voice data or operator voice data is recognized by voice recognition, and the process proceeds to step S23.

[0183] In step S23, the customer's or operator's speech, which was transcribed into text using speech recognition in step S21, is identified and stored in the dialogue history. In other words, the dialogue history is updated.

[0184] In the next step, S25, the operator's customer service level is determined, and the process proceeds to step S27. In step S25, the CPU 20 inputs the conversation history up to the present into the customer service level determination model 12c and obtains output regarding the customer service level. However, the customer service level determination model 12c is pre-programmed with prompts for customer service level determination.

[0185] In step S27, it is determined whether the permission flag 304k is on. If the answer in step S27 is "NO", that is, if the permission flag 304k is off, the process proceeds to step S33. On the other hand, if the answer in step S27 is "YES", that is, if the permission flag 304k is on, the process proceeds to step S29 to determine whether the support level is appropriate. Here, the CPU 20 determines whether the support level corresponding to the customer service level determined in step S25 matches the support level indicated by the support level data 304f. If the two support levels match, the support level is determined to be appropriate; if the two support levels do not match, the support level is determined to be inappropriate.

[0186] If the answer in step S29 is "YES," that is, if the support level is appropriate, the process proceeds to step S33. On the other hand, if the answer in step S29 is "NO," that is, if the support level is not appropriate, the process proceeds to step S31, where the support level is adjusted, and then the process proceeds to step S33. In step S31, the CPU 20 stores, or overwrites, the data storage area 304 with support level data 304f corresponding to the support level corresponding to the customer service level. In this way, the support level is automatically adjusted.

[0187] In step S33, it is determined whether the interaction has ended. Here, the CPU 20 determines whether it has received notification from the operator terminal 16 that the interaction has ended. If the answer in step S33 is "NO," that is, if the interaction has not ended, the process returns to step S11. In other words, operator support continues.

[0188] On the other hand, if the answer in step S33 is "YES," that is, if the conversation has ended, then in step S35, the operator's customer service (i.e., customer interaction) is evaluated. Here, the CPU 20 inputs the entire conversation history into the customer service evaluation model 12d and obtains output regarding the evaluation of the operator's customer service. However, the customer service evaluation model 12d is pre-programmed with prompts for customer service evaluation corresponding to the type of customer service task.

[0189] Then, in step S37, the customer service evaluation and the recommended support level are sent to the operator terminal, and the information processing is terminated. However, the support level recommended in step S37 is the support level indicated by the support level data 304f if the permission flag 304k is on, and the support level corresponding to the customer service level indicated by the customer service level data 304e at the end of the conversation if the permission flag 304k is off.

[0190] Figures 18-20 are flowcharts showing an example of the dialogue control processing of the CPU 50 of the operator terminal 16 shown in Figure 3. Although not shown in the diagram, the following processes are executed in parallel with the dialogue control processing: a shooting process to photograph the operator and a process to receive data transmitted from the customer terminal 18.

[0191] As shown in Figure 18, when the CPU 50 starts the interactive control process, in step S101 it establishes a connection with the support device 12, and in step S103 it determines whether there is an operation to set the support level. If the answer in step S103 is "NO", that is, if there is no operation to set the support level, it returns to step S103. On the other hand, if the answer in step S103 is "YES", that is, if there is an operation to set the support level, in step S105 it sets the support level in the setting information and proceeds to step S107.

[0192] Step S107 determines whether or not there is a setting operation for enabling or disabling automatic adjustment. If the answer in step S107 is "NO," that is, if there is no setting operation for enabling or disabling automatic adjustment, the process returns to step S107. On the other hand, if the answer in step S107 is "YES," that is, if there is a setting operation for enabling or disabling automatic adjustment, the process proceeds to step S109 to set whether or not automatic adjustment is enabled in the setting information, and then proceeds to step S111.

[0193] In step S111, the configuration information is sent to the support device 12, and the process proceeds to step S113 shown in Figure 19. As shown in Figure 19, in step S113, the connection status with the customer terminal 18 is established, and in step S115, the operator-side dialogue screen 100 as described above is displayed.

[0194] In the next step S117, avatar motion data is generated from the operator image data, and in step S119, the avatar motion data is sent to the customer terminal 18. In the next step S121, it is determined whether or not there is voice input. If the answer in step S121 is "NO", that is, if there is no voice input, the process proceeds to step S127 shown in Figure 20. On the other hand, if the answer in step S121 is "YES", that is, if there is voice input, the operator voice data 404h is sent to the customer terminal 18 in step S123, the operator voice data 404h is sent to the support device 12 in step S125, and the process proceeds to step S127.

[0195] As shown in Figure 20, step S127 determines whether data has been received from the customer terminal 18. If the answer in step S127 is "NO," that is, if no data has been received from the customer terminal 18, the process proceeds to step S137. On the other hand, if the answer in step S127 is "YES," that is, if data has been received from the customer terminal 18, the process proceeds to step S129 to determine whether customer voice data 404e exists.

[0196] If the answer in step S129 is "YES," that is, if customer voice data 404e exists, then in step S131, customer image data 404d and customer voice data 404e are output, and in step S133, customer voice data 404e is transmitted to the support device 12, and the process proceeds to step S137. On the other hand, if the answer in step S129 is "NO," that is, if customer voice data 404e does not exist, then in step S135, customer image data 404d is output, and the process proceeds to step S137.

[0197] In step S137, it is determined whether or not support voice data 404i has been received from the support device 12. If the answer in step S137 is "NO," that is, if support voice data 404i has not been received, the process proceeds to step S141. On the other hand, if the answer in step S137 is "YES," that is, if support voice data 404i has been received, the support voice data 404i is output in step S139, and the process proceeds to step S141. In other words, the support voice is notified to the operator. Therefore, the operator can hear the support voice, that is, receive appropriate support, and speak to the customer. Thus, regardless of the operator's customer service level, they can respond to customers appropriately.

[0198] In step S141, it is determined whether the interaction has ended. Here, it is determined whether the operator has turned on button image 104 to indicate the end of the interaction, or whether the interaction control process has ended on the customer terminal 18. If the answer in step S141 is "NO", that is, if the interaction has not ended, the process returns to step S117. On the other hand, if the answer in step S141 is "YES", that is, if the interaction has ended, the interaction control process is terminated. At this time, the CPU 50 notifies the support device 12 and the customer terminal 18 that the interaction has ended.

[0199] Figure 21 is a flowchart showing an example of the dialogue control processing of the CPU 80 of the customer terminal 18 shown in Figure 4. When the customer instructs the operator to start responding, or when the CPU 80 automatically determines that the operator should start responding, the CPU 80 starts the dialogue control processing. Although not shown in the diagram, in parallel with the dialogue control processing, processing is performed to receive data transmitted from the operator terminal 16.

[0200] As shown in Figure 16, when the CPU 80 starts the dialogue control process, in step S201 it establishes a connection with the operator terminal 16, and in step S203 it displays the customer-side dialogue screen 200 as shown in Figure 6. At the beginning of the dialogue control process, since the avatar operation data 504c has not been received, a still image of the avatar 202 is displayed on the customer-side dialogue screen 200. However, an avatar image 202 in which the avatar introduces itself or greets the customer may also be displayed.

[0201] In the following step S205, it is determined whether the dialogue has ended. Here, the CPU 80 determines whether the customer has instructed the dialogue to end, or whether the operator terminal 16 has notified the end of the dialogue. If the answer in step S205 is "NO," that is, if the dialogue has not ended, the process proceeds to step S207. On the other hand, if the answer in step S205 is "YES," that is, if the dialogue has ended, the dialogue control process is terminated. At this time, if the customer has instructed the dialogue to end, the CPU 80 notifies the operator terminal 16 that the dialogue has ended.

[0202] In step S207, it is determined whether or not data has been received from the operator terminal 16. If the answer in step S207 is "NO," that is, if no data has been received from the operator terminal 16, the process proceeds to step S217. On the other hand, if the answer in step S207 is "YES," that is, if data has been received from the operator terminal 16, the process proceeds to step S209 to determine whether or not operator voice data 504d exists.

[0203] If the answer in step S209 is "YES," that is, if operator voice data 504d is available, then in step S211, the operator voice data 504d is output, and in step S213, avatar image data is generated and output that performs actions according to the avatar action data 504c in accordance with the output of the operator voice data 504d, and then the process proceeds to step S217.

[0204] On the other hand, if the answer in step S209 is "NO", and there is no operator voice data 504d, then in step S215, avatar image data that performs an action according to the avatar action data 504c is generated and output, and the process proceeds to step S217.

[0205] In step S217, it is determined whether or not there is voice input. If the answer in step S217 is "YES," that is, if there is voice input, then in step S219, the customer image data 504e and customer voice data 504f are sent to the operator terminal 16, and the process returns to step S205. On the other hand, if the answer in step S217 is "NO," that is, if there is no voice input, then in step S221, the customer image data 504e is sent to the operator terminal 16, and the process returns to step S205.

[0206] According to this embodiment, the support device transmits appropriate information to the operator who is interacting with the customer, thereby providing appropriate support to the operator. Therefore, the operator can interact with the customer appropriately, regardless of their level of interaction.

[0207] Therefore, on-the-job training can be provided for novice operators, and for intermediate and advanced operators, it is possible to prevent or avoid forgetting to mention something, repeating the same information, or communicating incorrect information.

[0208] In this embodiment, the support level is set in three stages, but this is not limited to that. The support level may be set in two stages, or in four or more stages. When setting the support level in two stages, for example, it may be set in the two stages of low and normal, normal and high, or low and high, as shown in the embodiment.

[0209] Furthermore, when setting four or more levels, the normal support level can be further divided into three levels. At the normal support level, at the high level, the content to be spoken is conveyed in short sentences; at the medium level, the content to be spoken is conveyed in sentences for important topics and in keywords for less important topics; and at the low level, the content to be spoken is conveyed in keywords.

[0210] Furthermore, when setting four or more levels, the low support level can be further divided into three levels. At the low support level, if the operator ends their speech or dialogue without conveying what they were supposed to say at the high level, the operator will be given a short sentence or keyword to convey what they were supposed to say at the medium level, and if the operator hesitates, the operator will be given a short sentence or keyword to convey what they were supposed to say at the low level, and if the operator says something incorrect, the operator will be given a short sentence or keyword to point out the error.

[0211] However, both normal and low levels of support can also be divided into three stages.

[0212] Thus, when support levels are set in two or four or more stages, the customer service level is also set and determined in two or four or more stages. When support levels are set in two stages, if the support level is "high," the support content corresponding to the progress of customer service is selected from a table as described in the example, and if the support level is "low," the support content is determined by the output of the customer service content determination model if there is a problem with speech.

[0213] In this embodiment, avatar motion data is transmitted from the operator terminal and avatar image data is generated on the customer terminal. However, the avatar image data may be generated on the operator terminal and transmitted to the customer terminal.

[0214] Furthermore, although this embodiment provides separate support devices and operator terminals, the support device and operator terminal can also be combined into one unit. In this case, the process of sending and receiving data between the support device and the operator terminal can be omitted.

[0215] Furthermore, in this embodiment, a CG avatar is used to interact with customers by displaying an avatar image on the customer terminal's display device, but this is not limited to this embodiment. A robot avatar can also be used instead of a CG avatar. In such cases, a robot that also has the functions of the customer terminal is used instead of the customer terminal. The robot detects the customer's voice and transmits it to the support device and the operator terminal, and transmits a captured image including the customer to the operator terminal. However, the captured image may not include the customer. On the other hand, the robot outputs the operator's voice or the operator's voice converted into robot voice from a speaker and performs actions according to the operator's actions. However, when controlling the robot's actions, the operator's actions may be button operations or keyboard operations. Note that for robots that cannot blink and / or change facial expressions, blinking and / or changes in facial expressions are not included in the actions. In other words, in this specification, "avatar" should be understood as a term that encompasses remotely operated robots and CG agents.

[0216] Furthermore, the order in which each step in the flowchart shown in this embodiment is processed can be changed if the same result can be obtained.

[0217] Furthermore, the various screens and specific numerical values ​​shown in this specification and drawings are merely examples and can be modified as needed. [Explanation of symbols]

[0218] 10. Information Processing Systems 12...Support device 14…Network 16 ... Operator terminal 18 ... Customer terminal 20, 50, 80…CPU 22, 52, 82... RAM 24, 54, 84 ... Communication I / F 26, 56, 86… Input / Output Interfaces 28, 58, 88... Input devices 30, 60, 90…display device 62, 92... Mike 64, 94... speakers 66, 96...camera

Claims

1. Customer terminals used by customers, A terminal that is connected to the aforementioned customer terminal in a communication manner and is used by the operator who is in charge of the customer, An information processing system comprising a support device that is communicably connected to the operator terminal and assists the operator, The aforementioned support device is When the operator is online and interacting with the customer, a progress determination means determines the progress of the customer service provided by the operator based on the conversation history between the operator and the customer. A determination means for determining the content of support provided by the operator according to the level of support provided by the operator in assisting the customer, and the progress determined by the progress determination means, and The system includes a support content transmission means for transmitting the support content determined by the determination means to the operator terminal, The aforementioned operator terminal is A receiving means for receiving support content transmitted by the support content transmission means, and An information processing system comprising a notification means for notifying the operator of the support content received by the receiving means.

2. The aforementioned support device is Setting means for setting the support level according to the operator's actions, Customer service level determination means for determining the customer service level of the operator based on the aforementioned dialogue history, The information processing system according to claim 1, further comprising an adjustment means for adjusting the support level according to the customer service level determined by the customer service level determination means.

3. The aforementioned support levels are set at two or more stages. The information processing system according to claim 1 or 2, wherein the determination means determines, if the support level is the lowest, the support content to be expressed in a short sentence or keywords relating to the problem, if there is a problem with the operator's utterance, and if the support level is the highest, the support content to be expressed in a short sentence or keywords according to the progress, and if the support level is set to three or more levels, the support content to be expressed in a short sentence or keywords according to the progress.

4. When an operator is interacting with a customer online, a progress determination means determines the progress of the customer service provided by the operator based on the conversation history between the operator and the customer. A determination means for determining the content of support provided by the operator according to the level of support provided by the operator in assisting the customer, and the progress determined by the progress determination means, and An information processing device comprising a support content transmission means for transmitting the support content determined by the determination means to the operator's terminal.

5. An information processing program executed on an information processing device, The processor of the aforementioned information processing device, When an operator is interacting with a customer online, a progress determination step is performed to determine the progress of the customer service provided by the operator based on the conversation history between the operator and the customer. A decision step to determine the content of the support provided by the operator, in accordance with the level of support the operator provides to the customer and the progress determined in the progress determination step, and An information processing program that causes the program to execute a support content transmission step, which transmits the support content determined in the aforementioned determination step to the operator's terminal.

6. An information processing method for an information processing device, The processor of the aforementioned information processing device is When an operator is interacting with a customer online, the progress of the customer service provided by the operator is determined based on the conversation history between the operator and the customer. The level of support provided by the operator to assist the customer and the determined progress are used to determine the content of the support provided by the operator. An information processing method for transmitting the determined support details to the operator's terminal.

Images