Control system, control device, control program, and control method

Abstract

To allow a user to selectively perform a text chat or voice chat with an operator when a chat involving automated responses ends.SOLUTION: A dialogue control device has a function of a scenario-based chatbot and interacts with a user who uses a user terminal through an avatar. A chat screen including an avatar image is displayed on a display device of the user terminal. The dialogue control device automatically chats with the user, and when it sends the final response at the time of automated responses to the user terminal, that is, when the chat involving automated responses ends, a selection screen (200) for the user to choose a text chat or voice chat with an operator is displayed on the display device of the user terminal. The user then selects either a text chat or voice chat with the operator on the selection screen and performs the text chat or voice chat with the operator.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a control system, a control device, a control program, and a control method, and more particularly to a control system, a control device, a control program, and a control method for controlling a chat for answering questions from users, for example. [Background technology]

[0002] An example of this type of background art is disclosed in Patent Document 1. In the scenario execution device disclosed in Patent Document 1, when an input unit detects that information about a scenario to be executed and a start of execution have been input, or when a timing for executing the scenario to be executed indicated by a pre-registered execution schedule has arrived, a calling unit reads a scenario file describing the scenario to be executed from a storage unit. The first processing module described in the scenario file is then designated as the execution target module, and information about the execution target module is read from the scenario file. The information about the execution target module includes the type of the execution target module, the setting values ​​of each variable in the property information of the execution target module, and the processing module that will be next in the execution order after the execution target module. It is determined whether the type of the processing target module is a user-operated module or a user interface module, and if it is determined that the type of the processing target module is a user-operated module, the input unit is controlled to perform the user operation indicated by the processing target module. The reading unit determines whether the next execution order of the module to be processed is an end node or not, and if it determines that the next execution order is an end node, it terminates the execution of the scenario, and if it determines that the next execution order is not an end node, it sets the processing module in the next execution order as a new module to be executed and repeats the process from the process of reading information of the module to be executed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-43716 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the above-mentioned Patent Document 1, the scenario execution device terminates the execution of a scenario when it determines that the next node in the execution order is the end node. In a typical chat, when the execution of a scenario ends, if the user's problem or question is not resolved, the user selects another scenario to continue the chat, or enters contact information and requirements according to a predetermined format to make an inquiry. This makes it time-consuming for the user to resolve the problem or question. Therefore, there is room for improvement.

[0005] Therefore, a primary object of the present invention is to provide a novel control system, a novel control device, a novel control program, and a novel control method.

[0006] Another object of the present invention is to provide a control system, a control device, a control program, and a control method that allow a user to selectively engage in text chat or voice chat with an operator when an automatic response chat has ended. [Means for solving the problem]

[0007] The first invention is a dialogue control system comprising a user terminal used by a user, a control device having a chatbot function that automatically answers the user's questions, and an operator terminal used by a human operator that answers the user's questions, wherein the control device comprises a question receiving means for receiving questions sent from the user terminal, an answer sending means for generating an answer to the question received by the question receiving means and sending it to the user terminal, and a selection means for displaying a selection image on a display device of the user terminal to select text chat or voice chat with the human operator when the answer sending means sends the final answer in the automatic response to the user terminal, and the user terminal comprises a question sending means for sending a question entered by the user to the control device or the operator terminal, an answer receiving means for receiving the answer sent from the control device or the operator terminal, an answer display means for displaying on the display device the answer received by the answering means, and a chat execution means for executing text chat or voice chat with the operator terminal used by the operator when text chat or voice chat with the operator is selected in the selection image.

[0008] A second invention is dependent on the first invention, and the user terminal further comprises an avatar display means for displaying images of avatars corresponding to the chatbot and the operator on the display device, and the image of the avatar corresponding to the chatbot and the image of the avatar corresponding to the operator are images of the same character.

[0009] A third invention is a control device that controls a chat that automatically answers a user's questions, and includes a question receiving means that receives a question sent from a user terminal used by the user, an answer sending means that generates an answer to the question received by the question receiving means and sends it to the user terminal, and a selection means that, when the answer sending means sends the final answer during an automatic response to the user terminal, displays a selection image on a display device of the user terminal to select text chat or voice chat with a human operator.

[0010] A fourth invention is a control program executed by a control device that controls chat that automatically answers user questions, the control program causing a processor of the control device to execute a question receiving step of receiving a question sent from a user terminal used by the user, an answer sending step of generating an answer to the question received in the question receiving step and sending it to the user terminal, and a selection step of displaying on a display device of the user terminal a selection image for selecting text chat or voice chat with a human operator when the final answer in the automatic response is sent to the user terminal in the answer sending step.

[0011] The fifth invention is a control method for a control device that controls a chat that automatically answers a user's questions, in which the processor of the control device receives a question sent from a user terminal used by the user, generates an answer to the received question, sends it to the user terminal, and when the final answer during the automatic response has been sent to the user terminal, displays a selection image on the display device of the user terminal to select text chat or voice chat with a human operator. [Effects of the Invention]

[0012] According to this invention, when the chat by automatic response is finished, the user can selectively carry out a text chat or a voice chat with the operator.

[0013] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the preferred embodiments with reference to the drawings. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing a dialogue control system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of the electrical configuration of the dialogue control device shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an example of the electrical configuration of the user terminal shown in FIG. [Figure 4] FIG. 4 is a block diagram showing an example of the electrical configuration of the operator terminal shown in FIG. [Figure 5] Figure 5(A) is a diagram showing a first example of a web screen displayed on the display device of the user terminal shown in Figure 1, and Figure 5(B) is a diagram showing a second example of a web screen displayed on the display device of the user terminal shown in Figure 1. [Figure 6] FIG. 6 is a diagram showing an example of a chat screen for text chat displayed on the display device of the user terminal shown in FIG. [Figure 7] FIG. 7 is a diagram showing an example of a selection screen displayed on the display device of the user terminal shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a chat screen for voice chat displayed on the display device of the user terminal shown in FIG. [Figure 9] FIG. 9 is a diagram showing an example of a memory map of the RAM of the dialogue control device shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of a memory map of the RAM of the user terminal shown in FIG. [Figure 11] FIG. 11 is a diagram showing an example of a memory map of the RAM of the operator terminal shown in FIG. [Figure 12] FIG. 12 is a flowchart showing an example of a dialogue control process by the CPU of the dialogue control device shown in FIG. [Figure 13] FIG. 13 is a flowchart showing a first part of an example of the interaction control process by the CPU of the user terminal shown in FIG. [Figure 14] FIG. 14 is a flowchart showing a second part of an example of the dialogue control process by the CPU of the user terminal shown in FIG. 3, and is subordinate to FIG. [Figure 15] FIG. 15 is a flowchart showing a third part of an example of the interaction control process by the CPU of the user terminal shown in FIG. 3, and is subordinate to FIG. [Figure 16] FIG. 16 is a flowchart showing a fourth part of an example of the interaction control process by the CPU of the user terminal shown in FIG. 3, and is subordinate to FIG. [Figure 17]FIG. 17 is a flowchart showing a fifth part of an example of the interaction control process by the CPU of the user terminal shown in FIG. 3, and is subordinate to FIG. [Figure 18] FIG. 18 is a flowchart showing a first part of an example of the dialogue control process of the CPU of the operator terminal shown in FIG. [Figure 19] FIG. 19 is a flowchart showing a second part of an example of the dialogue control process by the CPU of the operator terminal shown in FIG. 4, and is a flowchart subordinate to FIG. [Figure 20] FIG. 20 is a flowchart showing a third part of an example of the dialogue control process by the CPU of the operator terminal shown in FIG. 4, and is a flowchart subordinate to FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] Referring to FIG. 1, a dialogue control system 10 of this embodiment includes a dialogue control device 12, which is communicatively connected to a user terminal 16 and an operator terminal 18 via a network 14.

[0016] In this embodiment, one dialogue control device 12, one user terminal 16, and one operator terminal 18 are shown, but in reality, multiple dialogue control devices 12, multiple user terminals 16, and multiple operator terminals 18 may be connected to the network 14. Therefore, a connection is established between the user terminal 16 and the dialogue control device 12 that provides a chat service on the website that the user terminal 16 is accessing, or the operator terminal 18 that provides a manned chat service on that website.

[0017] Although not shown, a server that operates a website is also connected to the network 14. However, the dialogue control device 12 may also operate the website.

[0018] The dialogue control device 12 is an information processing device, such as a notebook PC, a desktop PC, or a server. In this embodiment, the dialogue control device 12 provides a scenario-based chat service with automatic responses and executes an automatic dialogue program 302c (i.e., a chatbot) that responds in text to questions (utterances or statements) entered by a user (see FIG. 9).

[0019] The network 14 is composed of an IP network (or IP network) including the Internet, and an access network (or access network) for accessing this IP network. The access network may be a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, or a CATV (Cable Television) network.

[0020] The user terminal 16 is another information processing device, and examples thereof include a notebook PC, a desktop PC, a smartphone, and a tablet PC. The user terminal 16 is owned and used by a user. However, the user terminal 16 may also be a terminal installed in a store to which the dialogue control system 10 is applied and used by the user.

[0021] The operator terminal 18 is another information processing device, such as a notebook PC, a desktop PC, or a server. The operator terminal 18 provides a chat service and is used by an operator who responds to users. The operator terminal 18 can provide a text chat service and a voice chat service. In this embodiment, the user can select whether to perform a text chat (hereinafter referred to as "text chat") or a voice chat (hereinafter referred to as "voice chat"). However, the operator may also select either text chat or voice chat.

[0022] Even when using text chat, the user may input a question by voice, and the question may be recognized and converted into text, or the text of the answer to the question may be converted into voice and output.

[0023] In this embodiment, a chat screen (see FIGS. 6 and 8) including an image corresponding to an avatar of a person interacting with the user (hereinafter referred to as an "avatar image") is displayed on the display device 30 of the user terminal 16, and the user interacts with the chatbot or the operator via the chat screen. In other words, the user uses a chat service or an interaction service provided by the interaction control device 12 or the operator terminal 18.

[0024] As described above, the chatbot is a scenario-based chatbot that automatically answers user questions according to a pre-prepared scenario. In this embodiment, the user can freely input a question, keywords are extracted from the question, and an answer corresponding to the extracted keyword is selected by artificial intelligence. However, the user can also select one question from a plurality of pre-determined questions. In such a case, a pre-determined answer to the question is automatically selected.

[0025] In this embodiment, when the dialogue control device 12, i.e., the chatbot, has answered to the end of the automatic response, that is, when the chat by the automatic response has ended, the user can further use the chat service by the operator. However, in this embodiment, since the chatbot is, as an example, a scenario-type chatbot, the final answer during the automatic response is the final answer according to the scenario.

[0026] Fig. 2 is a block diagram showing an example of the electrical configuration of the dialogue control device 12 shown in Fig. 1. As shown in Fig. 2, the dialogue control device 12 includes a CPU 20, which is connected to a RAM 22, a communication interface (hereinafter referred to as "communication I / F") 24, and an input / output interface (hereinafter referred to as "input / output I / F") 26 via an internal bus.

[0027] The CPU 20 is responsible for overall control of the dialogue control device 12. However, instead of the CPU 20, a SoC (System-on-a-chip) including multiple functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided.

[0028] The RAM 22 is a main storage device and is used as a work area or buffer area for the CPU 20. Although not shown, the dialogue control device 12 is provided with a HDD and a ROM as auxiliary storage devices. However, a non-volatile memory such as an SSD may be used instead of or in addition to the HDD.

[0029] The communication I / F 24 is a wired interface for transmitting and receiving control signals and data to and from an external computer such as a user terminal 16 via the network 14 under the control of the CPU 20. However, the communication I / F 24 may also be a wireless interface for connecting to a wireless LAN.

[0030] An input device 28 and a display device 30 are connected to the input / output I / F 26. The input device 28 is a keyboard and a computer mouse. A touch panel may also be provided. The display device 30 is, for example, a liquid crystal display device.

[0031] In addition, the input / output I / F 26 outputs operation data (or operation information) input from the input device 28 to the CPU 20, and also outputs image data generated by the CPU 20 to the display device 30, causing the display device 30 to display a screen or image corresponding to the image data.

[0032] The electrical configuration of the dialogue control device 12 shown in FIG. 2 is an example and need not be limited to this.

[0033] Fig. 3 is a block diagram showing an example of the electrical configuration of the user terminal 16 shown in Fig. 1. As shown in Fig. 3, the user terminal 16 includes a CPU 50, which is connected to a RAM 52, a communication I / F 54, and an input / output I / F 56 via an internal bus.

[0034] The CPU 50 is responsible for overall control of the user terminal 16. However, instead of the CPU 50, an SoC including multiple functions such as a CPU function and a GPU function may be provided.

[0035] The RAM 52 is a main storage device and is used as a work area or buffer area for the CPU 50. Although not shown in the figure, the user terminal 16 is provided with a HDD and a ROM as auxiliary storage devices. However, a non-volatile memory such as an SSD may be used instead of or in addition to the HDD.

[0036] The communication I / F 54 is a wired interface for transmitting and receiving control signals and data to and from an external computer such as the dialogue control device 12 via the network 14 under the control of the CPU 50. However, the communication I / F 54 may also be a wireless interface for connecting to a wireless LAN.

[0037] An input device 58, a display device 60, a microphone 62, and a speaker 64 are connected to the input / output I / F 56.

[0038] The input device 58 is a keyboard and a computer mouse. A touch panel may also be provided. However, if a smartphone or a tablet PC is used as the user terminal 16, the input device 58 is a touch panel, hardware buttons, and a microphone. The display device 60 is, for example, a liquid crystal display device.

[0039] Furthermore, the input / output I / F 56 outputs operation data (or operation information) input from the input device 58 to the CPU 50, and also outputs image data generated by the CPU 50 to the display device 60, causing the display device 60 to display a screen or image corresponding to the image data. However, image data received from an external computer (for example, the dialogue control device 12 and a website server) may also be output by the CPU 50. The image data received from the dialogue control device 12 is image data of a chat screen. The image received from the website server is image data of a web screen.

[0040] Furthermore, the input / output I / F 56 converts the user's voice detected by the microphone 62 into digital voice data (hereinafter referred to as "user voice data") and outputs it to the CPU 50, and converts the voice data output by the CPU 50 into an analog voice signal and outputs it from the speaker 64. However, in this embodiment, the voice data output from the CPU 50 is, for example, voice data of the operator received from the operator terminal 18 (hereinafter referred to as "operator voice data").

[0041] The electrical configuration of the user terminal 16 shown in FIG. 3 is an example and need not be limited to this.

[0042] Furthermore, if the user terminal 16 is a smartphone, it is equipped with a call circuit for making calls via a mobile phone network, or via a mobile phone network and a public telephone network; however, in this embodiment, such calls are not made, and therefore are not shown.

[0043] Fig. 4 is a block diagram showing an example of the electrical configuration of the operator terminal 18 shown in Fig. 1. As shown in Fig. 4, the operator terminal 18 includes a CPU 80, which is connected to a RAM 82, a communication I / F 84, and an input / output I / F 86 via an internal bus.

[0044] The CPU 80 is responsible for overall control of the operator terminal 18. However, instead of the CPU 80, an SoC including multiple functions such as a CPU function and a GPU function may be provided.

[0045] The RAM 82 is a main storage device and is used as a work area or buffer area for the CPU 80. Although not shown in the figure, the operator terminal 18 is provided with a HDD and a ROM as auxiliary storage devices. However, a non-volatile memory such as an SSD may be used instead of or in addition to the HDD.

[0046] The communication I / F 84 is a wired interface for transmitting and receiving control signals and data to and from an external computer such as a user terminal 16 via the network 14 under the control of the CPU 80. However, the communication I / F 84 may also be a wireless interface for connecting to a wireless LAN.

[0047] To the input / output I / F 86, an input device 88, a display device 90, a microphone 92, a speaker 94, and a camera 96 ​​are connected.

[0048] The input device 88 is a keyboard and a computer mouse. A touch panel may also be provided. The display device 90 is, for example, a liquid crystal display device.

[0049] Furthermore, the input / output I / F 86 outputs operation data (or operation information) input from the input device 88 to the CPU 80, and also outputs image data generated by the CPU 80 to the display device 90, causing a screen corresponding to the image data to be displayed on the display device 90. However, image data received from an external computer (for example, the user terminal 16) may also be output by the CPU 80. The image data received from the user terminal 16 is user video data.

[0050] Furthermore, the input / output I / F 86 converts the voice of the operator detected by the microphone 92 into digital voice data and outputs it to the CPU 80, converts voice data output by the CPU 80 into an analog voice signal and outputs it from the speaker 94, and outputs data of an image including the operator photographed (detected) by the camera 96 ​​(hereinafter referred to as "operator image data") to the CPU 80. However, in this embodiment, the voice data output from the CPU 80 is user voice data received from the user terminal 16. Furthermore, the image of the operator is a still image or a moving image.

[0051] The electrical configuration of the operator terminal 18 shown in FIG. 4 is an example and does not need to be limited to this.

[0052] The operation of the dialogue control system 10 of this embodiment will be described below. In this dialogue control system 10, when a user accesses a website such as a general company site or an electronic commerce site (hereinafter referred to as an "EC site") on the user terminal 16, the user terminal 16 performs (or starts) a dialogue control process when the user requests a chat or when it is detected that no operation has been performed for more than a first predetermined time (for example, 60 seconds).

[0053] FIG. 5(A) shows an example of a web screen 100 displayed on the display device 60 of the user terminal 16, and FIG. 5(B) shows another example of a web screen 100 displayed on the display device 60 of the user terminal 16.

[0054] As shown in Figure 5(A), web screen 100 is an example of a web page of an e-commerce site for buying and selling products, displaying photos of multiple products with information and / or descriptions of each product displayed below the product. However, since web screen 100 itself is not essential to the present invention, in this specification, the text "Product A," "Product B," "Product C," and "Product D" are displayed instead of product photos, and the information and / or descriptions of each product are indicated by multiple dots. The same applies to Figure 5(B).

[0055] The user can refer to the web screen 100 to select and purchase a desired product and quantity. However, the purchase is not necessarily required. Since it is already well known that a product and quantity can be selected and a purchase process can be performed on the web screen 100, a detailed description thereof will be omitted.

[0056] The web screen 100 also displays a button image 102. The button image 102 is an icon for using a chat service provided by the dialogue control device 12. When the button image 102 is turned on, a connection state between the user terminal 16 and the dialogue control device 12 is established, and as shown in FIG. 5(B), the display device 60 of the user terminal 16 displays the web screen 100 with a chat window (hereinafter referred to as "chat screen") 150 displayed in the foreground. However, the chat screen 150 may be displayed instead of the web screen 100.

[0057] The operation of the chat window is controlled using JavaScript ("JAVASCRIPT" is a registered trademark) that runs on the web browser of the user terminal 16. The operation of the chat window includes operations such as displaying the text of a question entered by the user on the chat screen 150, displaying the text of a response from the dialogue control device 12 on the chat screen 150, and executing processing in response to the operation of a button image displayed on the chat screen 150.

[0058] Hereinafter, the text of the question entered by the user may be referred to as "user text," and the text of the answer given by the dialogue control device 12, i.e., the chatbot, may be referred to as "bot text."

[0059] However, as described above, even if the button image 102 is not turned on, the interaction control process can also be started if the time period during which no user operation is performed while the web screen 100 is displayed continues for a first predetermined time or longer, because there is a possibility that the user will not understand the operation on the web screen 100.

[0060] Fig. 6 is a diagram showing an example of the specific content of chat screen 150 shown in Fig. 5(B). As shown in Fig. 6, chat screen 150 includes two display areas 150a and 150b arranged one above the other. Avatar image 152 and button image 154 are displayed in display area 150a, and text display area 156, text display area 158, and button image 160 are displayed in display area 150b.

[0061] As described above, the avatar image 152 is an image of the avatar of the person who responds to the user. The avatar is the person who responds to the user, i.e., the person who is conversing with the user, and in this embodiment, it is the chatbot and the operator. In other words, the chatbot avatar and the operator avatar are the same character. The button image 154 is an icon for closing the chat screen 150, i.e., for ending the chat.

[0062] The text display area 156 is an area for displaying the text of the avatar's answer, i.e., bot text. The text display area 158 is an area for displaying the text of the user's question, i.e., user text. The button image 160 is an icon that allows the user to send user text to the chatbot, i.e., the dialogue control device 12.

[0063] When the chat screen 150 is first displayed, a message such as "Please enter your question" is displayed in the text display area 156. When the user inputs user text, the content is displayed in the text display area 158, and when the user turns on the button image 160, the data of the input user text (hereinafter referred to as "user text data") is sent to the dialogue control device 12. Then, the user text in the text display area 158 is erased, and the bot text from the dialogue control device 12 is displayed in the text display area 156. If the user has any further questions, the above process is repeated.

[0064] When the next user text is entered, the previous user text is erased from the text display area 158. However, the user texts may be displayed in chronological order, and the text display area 158 may be scrolled in the direction in which the user texts are arranged. Similarly, when the next bot text is displayed, the previous bot text is erased from the text display area 156. However, the bot texts may be displayed in chronological order, and the text display area 156 may be scrolled in the direction in which the bot texts are arranged.

[0065] Meanwhile, the dialogue control device 12 receives user text data sent from the user terminal 16, selects bot text for the user text using artificial intelligence, and sends (i.e., replies to) the data of the selected bot text (hereinafter referred to as "bot text data") to the user terminal 16.

[0066] In this embodiment, the avatar performs a preset action in response to the bot text. The avatar's action is determined by learning from captured images of actions taken by a human operator when interacting with a user. This is just one example, and the avatar's actions, i.e., gestures and movements, can be set arbitrarily. However, the avatar's actions also include changes in facial expression, mouth movements, and blinking.

[0067] When bot text for the user text is selected, an avatar action corresponding to the selected bot text is determined, and a video of the avatar image 152 performing the determined action is played back. That is, the dialogue control device 12 transmits video data of the determined avatar action to the user terminal 16, and the user terminal 16 plays back the received video data and displays the chat screen 150 including the avatar image 152 on the display device 60.

[0068] In this embodiment, after image data of the initial chat screen 150 is sent from the dialogue control device 12 at the start of the chat, the user terminal 16 generates or updates the image data of the chat screen 150 and displays the generated or updated image data on the display device 60, but it is also possible to generate or update the image data of the chat screen 150 on the dialogue control device 12 side and send it to the user terminal 16.

[0069] In this embodiment, when interacting with the chatbot, i.e., during a chat, the avatar image 152 is displayed on the chat screen 150, and the avatar is also moved. Therefore, through the avatar, the user can get the feeling that they are interacting with a person with a personality. This is thought to make it easier for the user to interact.

[0070] As described above, in this embodiment, when the dialogue control device 12, i.e., the chatbot, has answered all the questions up to the end of the scenario, i.e., when it has sent the final answer during the automatic response, the user can further use the chat service provided by the operator. At this time, the user can select whether to engage in text chat or voice chat with the operator. Figure 7 shows an example of a selection screen 200 displayed on the display device 60 of the user terminal 16.

[0071] 7, selection screen 200 includes display area 200a and display area 200b. Avatar image 202 and button image 204 are displayed in display area 200a, and button image 206 and button image 208 are displayed in display area 200b.

[0072] Avatar image 202 is the same as avatar image 152 described above, and is an image of the avatar of the person interacting with the user. However, when selection screen 200 is displayed, the avatar speaks to indicate that further chatting with the operator is possible and to select a chat method, and performs a predetermined action such as bowing. Button image 204 is an icon for closing selection screen 200, i.e., for ending the chat.

[0073] The button image 206 is an icon for selecting whether to conduct a text chat with the operator. The button image 208 is an icon for selecting whether to conduct a voice chat with the operator.

[0074] When the button image 206 is turned on, a connection state is established between the user terminal 16 and the operator terminal 18, and the operator terminal 18 is notified that a text chat will be conducted. Therefore, the display device of the user terminal 16 displays a chat screen that is the same as the chat screen 150 shown in Fig. 6. However, since the user is conducting a text chat with the operator, the text of the operator's response (hereinafter referred to as "operator text") is displayed in the text display area 156.

[0075] Furthermore, when a user engages in text chat with an operator, the avatar's movement is determined according to the operator's emotion. The avatar's movement according to the emotion is determined in advance, and video data of avatar image 202 corresponding to the determined movement is played back.

[0076] The emotion of the operator can be estimated (or detected) based on the facial image of the operator. Since a method for estimating the emotion of a person such as an operator using a facial image is already known, a description of the method will be omitted.

[0077] Known techniques can be used to estimate human emotions from facial images, such as those disclosed in "Hiroshi Kobayashi and Fumio Hara: Recognition of Basic Human Facial Expressions Using Neural Networks, Transactions of the Society of Instrument and Control Engineers, Vol. 29, No. 1, 112 / 118 (1993)," "Yosuke Koyanaka, Tsuneyasu Homma, Masao Sakai, and Kenichi Abe: Facial Expression Recognition Using Neural Networks, Bulletin of the Tohoku University School of Medicine and Health Sciences, 13(1):23-32, 2004," and "Daiki Nishime, Satoshi Endo, Yoshiaki Touma, Koji Yamada, and Yuhei Akamine: Acquisition of Facial Expressions and Analysis of Facial Features Using Convolutional Neural Networks, Transactions of the Japanese Society for Artificial Intelligence, Vol. 32, No. 5, FZ (2017)."

[0078] Another known technique for estimating human emotions based on feature points extracted from a facial image is the technique disclosed in Japanese Patent Application Laid-Open No. 2020-163660.

[0079] In this embodiment, the avatar's movement is determined in accordance with the operator's emotion, but the avatar's movement may be determined in accordance with the user's emotion. In such a case, a captured image of the user's face is sent to the operator terminal 18.

[0080] Although not shown in the figures, when the operator terminal 18 is notified that a text chat will be held, for example, a chat screen similar to the chat screen 150 shown in FIG. 6 is displayed on the operator terminal 18 as the operator's response screen. However, an image of the user is displayed instead of the avatar image 152. The user's image does not have to be displayed. Furthermore, on the operator's response screen, user text is displayed in an area corresponding to text display area 156, and operator text is displayed in an area corresponding to text display area 158.

[0081] Since text chat between humans, such as a user and an operator, is already well known, a detailed explanation of this will be omitted.

[0082] Furthermore, when the button image 208 is turned on, a connection state is established between the user terminal 16 and the operator terminal 18, and the operator terminal 18 is notified that a voice chat will be held. Therefore, a chat screen 250 such as that shown in FIG. 8 is displayed on the display device 60 of the user terminal 16.

[0083] 8, chat screen 250 for conducting a voice chat includes display area 250a and display area 250b. Avatar image 252 and button image 254 are displayed in display area 250a, and nothing is displayed in display area 250b. However, a message such as "Ask a question" may be displayed in display area 250b.

[0084] Avatar image 252 is the same as avatar image 152 and avatar image 202 described above, and is an image of the avatar of the person responding to the user (here, the operator of operator terminal 18). Button image 254 is an icon for closing chat screen 250, that is, for ending the chat.

[0085] When the operator terminal 18 is notified that a voice chat will be conducted, for example, a chat screen similar to the chat screen 250 shown in Fig. 8 is displayed on the operator terminal 18 as a response screen on the operator side, but an image of the user is displayed instead of the avatar image 252. As with the chat screen 250 shown in Fig. 8, nothing is displayed in the lower display area on the response screen on the operator side, but the voices of the operator and the user may be recognized and a dialogue history may be displayed on the display device 90 of the operator terminal 18.

[0086] Since voice chat between people such as a user and an operator is already well known, a detailed explanation will be omitted.

[0087] Furthermore, when a user engages in voice chat with an operator, the avatar's movement is determined according to the operator's emotion. The avatar's movement according to the emotion is determined in advance, and video data of avatar image 202 corresponding to the determined movement is played back.

[0088] As described above, the emotion of the operator can be estimated (or detected) based on the facial image of the operator. However, in the case of voice chat, the emotion of the operator can also be estimated based on the voice of the operator.

[0089] Known techniques can be used to estimate human emotions from speech, such as those disclosed in JP 2021-12285 A and "Mori, Daiki: Understanding Emotions and Attitudes from Speech," Journal of the Institute of Electronics, Information and Communication Engineers, Vol. 101, No. 9, 2018.

[0090] In this embodiment, the avatar's movement is determined according to the operator's emotion, but the avatar's movement may be determined according to the user's emotion. When the user's emotion is estimated from the user's facial image, a captured image of the user's face is sent to the operator terminal 18.

[0091] In the case of voice chat, instead of playing video corresponding to the avatar's movements, the avatar's movements can be controlled based on the operator's movements. In such a case, the operator terminal 18 generates movement data for controlling the avatar's movements based on the operator's movements. This generated movement data is transmitted to the user terminal 16, and the avatar's movements are controlled. However, image data of an avatar image performing a movement according to the movement data may also be generated and transmitted to the user terminal 16.

[0092] The movement data is information (data) about the position and orientation of each part of the face for controlling the avatar's gestures, facial orientation, and facial expressions. Specifically, the movement data is information about the positions (coordinate data) of each part of the face, namely the eyebrows, eyes (upper eyelids, lower eyelids), nose, and mouth (upper lip, lower lip), the positions (coordinate data) of the head, hands, arms, shoulders, waist, knees, and ankles, and the orientation of the neck (or face), wrists, and ankles. However, if the avatar is not displayed from the chest down, information about the positions and orientations of the waist, knees, and ankles does not need to be included in the movement data.

[0093] The motion data for controlling the avatar's motion based on the operator's motion can be generated using an image processing library such as MediaPipe Holistic. In another example, the method disclosed in Japanese Patent Application Laid-Open No. 2021-56940 can be used.

[0094] Fig. 9 shows an example of a memory map 300 of the RAM 22 built into the dialogue control device 12 shown in Fig. 2. As shown in Fig. 9, the RAM 22 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores a dialogue control processing program, which is an example of an information processing program executed by the dialogue control device 12 of this embodiment.

[0095] The dialogue control processing programs executed by the dialogue control device 12 include a main processing program 302a, a communication program 302b, an automatic dialogue program 302c, an action determination program 302d, and the like.

[0096] The main processing program 302a is a program for executing the main routine of the dialogue control processing of the dialogue control device 12 of this embodiment.

[0097] The communication program 302b is a program for communicating (sending and receiving data, etc.) with an external device, in this embodiment, the user terminal 16.

[0098] The automatic dialogue program 302c is a chatbot, and is a program for generating bot text data 304c in response to user text data 304b received from the user terminal 16 based on the talk script data 304a, and transmitting the generated bot text data 304c to the user terminal 16. When receiving the user text data 304b or transmitting the bot text data 304c, the communication program 302b is also executed.

[0099] The action determination program 302d is a program for determining an action preset in the bot text data 304c and transmitting video data 304d of an avatar performing the determined action to the user terminal 16. When transmitting the video data 304d, the communication program 302b is also executed.

[0100] Although not shown, the program storage area 302 stores the operating system and middleware of the dialogue control device 12, as well as a browser and other application programs other than the dialogue control program of this application.

[0101] The data storage area 304 stores talk script data 304a, user text data 304b, bot text data 304c, video data 304d, and the like.

[0102] The talk script data 304a is data about a talk script for generating answers to user questions. As an example, the talk script is data about a user's question and the attendant's answer to the user's question in a scenario for the attendant to attend to the user. Specifically, the talk script can be generated by machine learning the user's question and the human's answer to the user's question.

[0103] The user text data 304b is user text, ie, text data about a question, that is input and sent by the user on the chat screen 150 displayed on the display device 60 of the user terminal 16 and received by the dialogue control device 12.

[0104] The bot text data 304c is bot text, that is, text data of a response, generated based on the talk script data 304a in response to the user text data 304b.

[0105] The video data 304d is video data about the avatar's movements determined in accordance with the bot text data 304c. The video data about the movements of each avatar is pre-stored or registered in an auxiliary storage device such as a HDD or ROM, and the data about the determined avatar's movements is read from the auxiliary storage device into the RAM 22.

[0106] Although not shown in the figure, the data storage area 304 stores other data required to execute the dialogue control process, and is provided with a timer (counter) and flags required to execute the dialogue control process.

[0107] Fig. 10 shows an example of a memory map 400 of the RAM 52 built into the user terminal 16. As shown in Fig. 10, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores an interaction control program, which is an example of an information processing program executed by the user terminal 16 of this embodiment.

[0108] The interaction control program executed on the user terminal 16 includes a main processing program 402a, an operation detection program 402b, a communication program 402c, an image generation program 402d, an image output program 402e, and a browser program 402f. However, since the browser is a function provided in the user terminal 16, the browser program 402f does not have to be included in the interaction control program.

[0109] The main processing program 402a is a program for executing the main routine of the dialogue control processing of the user terminal 16 in this embodiment.

[0110] The operation detection program 402b is a program for detecting operation data 404a input from the input device 58 in accordance with a user's operation and storing the data in the data storage area 404.

[0111] The communication program 402c is a program for communicating with external devices, in this embodiment, the website server, the dialogue control device 12, and the operator terminal 18.

[0112] Image generation program 402d is a program for generating, using image generation data 404b, image data corresponding to all or part of a screen or image to be displayed on display device 60. However, when generating image data corresponding to chat screen 150 as shown in FIGS. 5(B) and 6, user text data 404c, bot text data 404d, operator text data 404e, and video data 404h are also used as necessary. Furthermore, when generating image data corresponding to chat screen 250 as shown in FIG. 8, video data 404h is also used as necessary.

[0113] The image output program 402e is a program for outputting image data generated in accordance with the image generation program 402d to the display device 60. Therefore, a screen or image corresponding to the image data is displayed on the display device 60.

[0114] Browser program 402f is a program for executing processes related to displaying a web page, i.e., web screen 100, scrolling web screen 100, moving web pages, and processes corresponding to operations on web screen 100. Browser program 402f is also a program for executing dialogue control processes, i.e., processes for displaying and hiding chat windows, i.e., chat screens 150 and 250, and processes corresponding to the operation of the chat windows.

[0115] Although not shown, the program storage area 402 stores the operating system and middleware of the user terminal 16, as well as other application programs other than the dialogue control processing program of the present application.

[0116] The data storage area 404 stores operation data 404a, image generation data 404b, user text data 404c, bot text data 404d, operator text data 404e, user voice data 404f, operator voice data 404g, video data 404h, and the like.

[0117] The operation data 404a is data of a user's operation detected in accordance with the operation detection program 402b. The operation data 404a is deleted from the data storage area 404 after being used for processing by the CPU 50.

[0118] Image generation data 404b is data used to generate screens or images to be displayed on display device 60 of user terminal 16. Image generation data 404b also includes data for generating avatar image 202 on selection screen 200.

[0119] The user text data 404c is user text, that is, text data of a question, input by the user on the chat screen 150. The user text data 404c is erased from the data storage area 404 when the text of the next question is input.

[0120] The bot text data 404d is bot text, i.e., text data about an answer, that is generated and transmitted by the dialogue control device 12 and received by the user terminal 16. The bot text data 404d is erased from the data storage area 404 when text data about the next answer is received.

[0121] The operator text data 404e is operator text, i.e., text data about an answer, that is input and transmitted by the operator at the operator terminal 18 and received at the user terminal 16. The operator text data 404e is erased from the data storage area 404 when text data about the next answer is received.

[0122] The user voice data 404f is data about the user's voice detected by the microphone 62. Once the user voice data 404f is transmitted to the operator terminal 18, it is deleted from the data storage area 404.

[0123] The operator voice data 404g is data about the voice of the operator that is detected and transmitted by the operator terminal 18 and received by the user terminal 16. Once the operator voice data 404g is output from the speaker 64, it is erased from the data storage area 404.

[0124] The video data 404h is video data about an avatar that is determined and transmitted by the dialogue control device 12 or the operator terminal 18 and received by the user terminal 16.

[0125] Although not shown in the figure, the data storage area 404 stores other data required to execute the dialogue control process, and is provided with a timer (counter) and flags required to execute the dialogue control process.

[0126] Fig. 11 shows an example of a memory map 500 of the RAM 82 built into the operator terminal 18 shown in Fig. 4. As shown in Fig. 11, the RAM 82 includes a program storage area 502 and a data storage area 504. The program storage area 502 stores an interaction control processing program, which is an example of an information processing program executed by the operator terminal 18 of this embodiment.

[0127] The dialogue control processing program executed by the operator terminal 18 includes a main processing program 502a, an operation detection program 502b, a communication program 502c, an image generation program 502d, an image output program 502e, a photography program 502f, an emotion detection program 502g, and an action determination program 502h.

[0128] The main processing program 502a is a program for executing the main routine of the dialogue control processing of the operator terminal 18 in this embodiment.

[0129] The operation detection program 502b is a program for detecting operation data 504a input from the input device 88 in accordance with an operator's operation and storing the data in the data storage area 504.

[0130] The communication program 502c is a program for communicating (sending and receiving data, etc.) with an external device, in this embodiment, the user terminal 16.

[0131] Image generation program 502d is a program for generating image data corresponding to all or part of a screen or image to be displayed on display device 90, using image generation data 504b.

[0132] The image output program 502e is a program for outputting image data generated in accordance with the image generation program 502d to the display device 90. Therefore, a screen or image corresponding to the image data is displayed on the display device 90.

[0133] The photographing program 502f is a program for causing the camera 96 ​​to perform photographing processing and storing the operator image data 504g input from the camera 96 ​​in the data storage area 304.

[0134] The emotion detection program 502g is a program for estimating (or detecting) the emotion of the operator from a facial image included in a captured image of the operator.

[0135] The action determination program 502h is a program for determining a preset action corresponding to the emotion detected according to the emotion detection program 502g, and transmitting video data 504i of the avatar performing the determined action to the user terminal 16. When transmitting the video data 504i, the communication program 502c is also executed.

[0136] Although not shown, the program storage area 502 stores the operating system and middleware of the operator terminal 18, as well as a browser and other application programs other than the dialogue control program of the present application.

[0137] The data storage area 504 stores operation data 504a, image generation data 504b, user text data 504c, operator text data 504d, user voice data 504e, operator voice data 504f, operator image data 504g, emotion data 504h, and video data 504i.

[0138] The operation data 504a is data of the operation of the operator detected in accordance with the operation detection program 502b. The operation data 504a is deleted from the data storage area 504 after being used for processing by the CPU 80.

[0139] Image generation data 504b is data for generating a screen or image to be displayed on display device 90 of operator terminal 18.

[0140] The user text data 504c is text data about the user text, i.e., the question, that is input and sent by the user on the chat screen 150 displayed on the display device 60 of the user terminal 16 during text chat and received by the operator terminal 18.

[0141] The operator text data 504d is operator text input by the operator in response to the user text data 504c in the text chat, that is, text data of a reply.

[0142] The user voice data 504e is voice data about a user voice, that is, a question, that is detected and transmitted by the user terminal 16 and received by the operator terminal 18 during voice chat.

[0143] The operator voice data 504f is the operator voice input by the operator in response to the user voice data 504e in the voice chat, that is, voice data of a response.

[0144] The operator image data 504g is data on a captured image of the operator.

[0145] Emotion data 504h is data for identifying the emotion of the operator detected from the face image of the operator.

[0146] The moving image data 504i is moving image data about the avatar's movement determined in accordance with the operator's emotion indicated by the emotion data 504h. The moving image data about each avatar's movement is stored or registered in advance in an auxiliary storage device such as a HDD or ROM, and the data about the determined avatar's movement is read from the auxiliary storage device into the RAM 22.

[0147] Although not shown in the figure, the data storage area 504 stores other data required to execute the dialogue control process, and is provided with a timer (counter) and flags required to execute the dialogue control process.

[0148] Figure 12 is a flow diagram showing an example of the dialogue control processing of the CPU 20 of the dialogue control device 12 shown in Figure 2. Although not shown in the figure, a process for receiving data transmitted from the user terminal 16 is executed in parallel with the dialogue control processing. When an instruction to execute a chat is given on the user terminal 16, the execution of dialogue control and connection information of the user terminal 16 are notified to the dialogue control device 12 from the server operating the website, and the CPU 20 starts the dialogue control processing.

[0149] As shown in FIG. 12, when the CPU 20 starts the dialogue control process, it establishes a connection with the user terminal 16 in step S1, and determines whether or not user text data 304b has been received from the user terminal 16 in step S3.

[0150] If step S3 is "NO", that is, if user text data 304b has not been received from user terminal 16, proceed to step S13. On the other hand, if step S3 is "YES", that is, if user text data 304b has been received from user terminal 16, in step S5, bot text data 304c for user text data 304b is generated using talk script data 304a, and in step S7, the bot text data 304c generated in step S5 is sent to user terminal 16, and proceed to step S9.

[0151] In step S9, the avatar action corresponding to the bot text data 304c generated in step S5 is determined. In the next step S11, video data 304d of the avatar performing the action determined in step S9 is read from the auxiliary storage device and transmitted to the user terminal 16, and the process proceeds to step S13.

[0152] In step S13, it is determined whether or not dialogue control has ended. Here, the CPU 20 determines whether or not responses have been completed to the end of the scenario, i.e., whether or not responses via automatic response have ended. If the answer is "NO" in step S13, that is, if dialogue control has not ended, the process returns to step S3. On the other hand, if the answer is "YES" in step S13, that is, if dialogue control has ended, the dialogue control process ends. At this time, the CPU 20 notifies the user terminal 16 that responses have been completed to the end of the scenario, that is, if responses have been completed to the end of the automatic response.

[0153] 13 to 17 are flow diagrams showing an example of the dialogue control processing of the CPU 50 of the user terminal 16 shown in Fig. 3. When the user turns on the button image 102 on the web screen 100 or when it is automatically determined that a chat will be started, the CPU 50 starts the dialogue control processing. Although not shown in the figures, in parallel with the dialogue control processing, processing for receiving data or notifications transmitted from the user terminal 16 or the operator terminal 18 is executed.

[0154] It should be noted that since it is already well known that browser functions are executed to perform browser processing, i.e., displaying web screen 100 and performing processing on web screen 100 in response to user operations, flow diagrams and explanations thereof will be omitted.

[0155] 13, when the dialogue control process is started, CPU 50 establishes a connection with dialogue control device 12 in step S101, and displays chat screen 150 in step S103. At the beginning of the dialogue control process, video data 404h has not been received, so a still avatar image 152 is displayed on chat screen 150. However, an avatar image 152 in which the avatar introduces itself or greets the user may also be displayed.

[0156] In the following step S105, it is determined whether or not a key operation has been performed. If "NO" in step S105, that is, if no key operation has been performed, the process proceeds to step S109.

[0157] On the other hand, if the answer is "YES" in step S105, that is, if a key operation has been performed, then in step S107, text is input in accordance with the key operation, and the process proceeds to step S109. In step S107, CPU 50 stores or updates user text data 404c, and displays the text about the question in text display area 158 of chat screen 150.

[0158] In the next step S109, it is determined whether or not user text is to be transmitted. Here, the CPU 50 determines whether or not the button image 160 has been turned on. If "NO" in step S109, that is, if user text is not to be transmitted, the process proceeds to step S113. On the other hand, if "YES" in step S109, that is, if user text is to be transmitted, the user text data 404c is transmitted to the dialogue control device 12 in step S111, and the process proceeds to step S113.

[0159] In step S113, it is determined whether or not the bot text data 404d has been received. If "NO" in step S113, that is, if the bot text data 404d has not been received, the process proceeds to step S117 shown in FIG. 14. On the other hand, if "YES" in step S113, that is, if the bot text data 404d has been received, the bot text is displayed in step S115, and the process proceeds to step S117. In step S115, the CPU 50 displays the text of the answer corresponding to the bot text data 404d in the text display area 156 of the chat screen 150.

[0160] As shown in Fig. 14, in step S117, it is determined whether or not video data 404h has been received. If "NO" in step S117, that is, if video data 404h has not been received, the process proceeds to step S121. On the other hand, if "YES" in step S117, that is, if video data 404h has been received, the process starts playing the video in step S119, and proceeds to step S121. In step S119, CPU 50 starts playing video data 404h, and displays the played video in display area 150a. Therefore, the avatar makes gestures and other movements.

[0161] In step S121, it is determined whether the chat has ended. Here, the CPU 50 determines whether the button image 154 has been turned on. If the determination is "YES" in step S121, that is, if the chat has ended, in step S125, the dialogue control device 12 is notified of the end of the chat, and the dialogue control process is terminated. In other words, the chat screen 150 displayed in front of the web screen 100 is hidden, and the button image 102 is displayed.

[0162] On the other hand, if the answer is "NO" in step S121, that is, if the chat has not ended, it is determined in step S123 whether the scenario has ended. Here, the CPU 50 determines whether a notification has been received from the dialogue control device 12 that the responses have been completed to the end of the scenario, that is, a notification that the responses have been completed to the end of the automatic response.

[0163] If "NO" in step S123, that is, if the scenario has not ended, the process returns to step S105. On the other hand, if "YES" in step S123, that is, if the scenario has ended, selection screen 200 such as that shown in Fig. 7 is displayed on display device 60 in step S127 shown in Fig. 15, and it is determined in step S129 whether or not a selection has been made. Here, CPU 50 determines whether button image 206 or button image 208 has been turned on.

[0164] If "NO" in step S129, that is, if no selection has been made, a connection state with the operator terminal 18 is established in step S131, and it is determined whether or not a text chat is being performed in step S133. That is, the CPU 50 determines whether or not the button image 206 has been turned on.

[0165] If the determination in step S133 is "YES," that is, if the button image 206 is turned on, then in step S135, the chat screen 150 for text chat as shown in Fig. 6 is displayed on the display device 60, and the process proceeds to step S139 shown in Fig. 16. At this time, the CPU 50 notifies the operator terminal 18 that a text chat will be executed.

[0166] On the other hand, if the result in step S133 is "NO", that is, if the button image 208 is turned on, then in step S137, the chat screen 250 for the voice chat as shown in Fig. 8 is displayed on the display device 60, and the process proceeds to step S157 shown in Fig. 17. At this time, the CPU 50 notifies the operator terminal 18 of the execution of the voice chat.

[0167] 16, in step S139, it is determined whether or not a key operation is performed. If "NO" in step S139, the process proceeds to step S143. On the other hand, if "YES" in step S139, text is input in accordance with the key operation in step S141.

[0168] In the next step S143, it is determined whether or not user text is to be transmitted. If "NO" in step S143, the process proceeds to step S147. On the other hand, if "YES" in step S143, the user text data 404c is transmitted to the operator terminal 18 in step S145, and the process proceeds to step S147.

[0169] In step S147, it is determined whether or not the operator text data 404e has been received. If "NO" in step S147, that is, if the operator text data 404e has not been received, the process proceeds to step S151. On the other hand, if "YES" in step S147, that is, if the operator text data 404e has been received, the operator text is displayed in step S149, and the process proceeds to step S151. In step S149, the CPU 50 displays the operator text corresponding to the operator text data 404e, that is, the text of the reply, in the text display area 156 of the chat screen 150.

[0170] In step S151, it is determined whether or not video data 404h has been received. If "NO" in step S151, the process proceeds to step S155. On the other hand, if "YES" in step S151, playback of the video starts in step S153, and the process proceeds to step S155.

[0171] In step S155, it is determined whether or not the chat has ended. Here, CPU 50 determines whether or not the user has turned on button image 154. If "NO" in step S155, that is, if the chat has not ended, the process returns to step S139. If "YES" in step S155, that is, if the chat has ended, the dialogue control process ends.

[0172] 17, in step S157, it is determined whether or not there is voice input. If "NO" in step S157, that is, if there is no voice input, the process proceeds to step S161. On the other hand, if "YES" in step S157, that is, if there is voice input, the process transmits the user voice data 404f to the operator terminal 18 in step S159, and the process proceeds to step S161.

[0173] In step S161, it is determined whether or not the operator voice data 404g has been received. If "NO" in step S161, that is, if the operator voice data 404g has not been received, the process proceeds to step S165. On the other hand, if "YES" in step S161, that is, if the operator voice data 404g has been received, the operator voice data 404g is output in step S163, and the process proceeds to step S165. In step S163, the CPU 50 outputs the operator voice data 404g to the speaker 64.

[0174] In step S165, it is determined whether or not video data 404h has been received. If "NO" in step S165, proceed to step S169. On the other hand, if "YES" in step S165, playback of the video starts in step S167, and proceed to step S169.

[0175] In step S169, it is determined whether or not the chat has ended. If "NO" in step S169, the process returns to step S157. If "YES" in step S169, that is, if the chat has ended, the dialogue control process ends.

[0176] 18 to 20 are flow diagrams showing an example of the interaction control processing of the CPU 80 of the operator terminal 18 shown in Fig. 4. Although not shown in the figures, an imaging process for imaging the operator and a process for receiving data or a notification transmitted from the user terminal 16 are executed in parallel with the interaction control processing.

[0177] As shown in FIG. 18, when the CPU 80 starts the interaction control process, it establishes a connection with the user terminal 16 in step S201.

[0178] In the next step S203, it is determined whether a text chat is to be performed. Here, the CPU 80 determines whether there is a notification from the user terminal 16 that a text chat or a voice chat is to be performed.

[0179] If "NO" in step S203, that is, if voice chat is to be performed, the process proceeds to step S227 shown in Fig. 20. On the other hand, if "YES" in step S203, that is, if text chat is to be performed, the above-mentioned operator's response screen is displayed in step S205.

[0180] In the next step S207, it is determined whether or not user text data 504c has been received. If "NO" in step S207, that is, if user text data 504c has not been received, the process proceeds to step S211. On the other hand, if "YES" in step S207, that is, if user text data 504c has been received, the user text is displayed in step S209, and the process proceeds to step S211.

[0181] In step S211, it is determined whether or not a key operation has been performed. If "NO" in step S211, the process proceeds to step S215 shown in Fig. 19. On the other hand, if "YES" in step S211, text is input in accordance with the key operation in step S213, and the process proceeds to step S215.

[0182] As shown in Fig. 19, in step S215, it is determined whether or not the operator is to transmit text, where the CPU 80 determines whether or not the operator has instructed to transmit a response.

[0183] If "NO" in step S215, that is, if the operator text is not to be transmitted, the process proceeds to step S225. On the other hand, if "YES" in step S215, that is, if the operator text is to be transmitted, the operator text data 504d is transmitted to the user terminal 16 in step S217, and the operator's emotion is detected based on the operator's facial image in step S219.

[0184] In the next step S221, the avatar's behavior is determined according to the operator's emotion detected in step S219, and in step S223, video data 504i of the avatar performing the behavior determined in step S221 is sent to the user terminal 16, and the process proceeds to step S225.

[0185] In step S225, it is determined whether or not the dialogue control has ended. Here, the CPU 80 determines whether or not the operator has instructed the chat to end, or whether or not a chat end notification has been received from the user terminal 16. If the determination in step S225 is "NO", the process returns to step S207. On the other hand, if the determination in step S225 is "YES", the dialogue control process ends.

[0186] As described above, if the determination in step S203 is "NO", then in step S227 shown in FIG. 20, it is determined whether or not the user voice data 504e has been received. If the determination in step S227 is "NO", that is, if the user voice data 504e has not been received, the process proceeds to step S231. On the other hand, if the determination in step S227 is "YES", that is, if the user voice data 504e has been received, the user voice data 504e is output in step S229, and the process proceeds to step S231. In step S229, the CPU 80 outputs the user voice data 504e to the speaker 94.

[0187] In step S231, it is determined whether or not there is voice input. If "NO" in step S231, that is, if there is no voice input, the process proceeds to step S241. On the other hand, if "YES" in step S231, that is, if there is voice input, in step S233, the operator voice data 504f is transmitted to the user terminal 16, and in step S235, the operator's emotions are detected based on the operator's facial image.

[0188] In the following step S237, the avatar's movement is determined according to the operator's emotion detected in step S235, and in step S239, video data 504i of the avatar performing the movement determined in step S237 is transmitted to the user terminal 16, and the process proceeds to step S241.

[0189] In step S241, it is determined whether or not the dialogue control is to be ended. If "NO" in step S241, the process returns to step S227. On the other hand, if "YES" in step S241, the dialogue control process is ended.

[0190] According to this embodiment, when a user ends a chat using automatic response, a selection screen is displayed that allows the user to select whether to conduct a text chat or a voice chat with the operator, and the user can conduct a text chat or a voice chat with the operator depending on their selection.

[0191] Furthermore, according to this embodiment, the chatbot and the avatar, which is the alter ego of the person interacting with the user, such as the operator, use the same character, so the user can chat with the chatbot and the operator without feeling uncomfortable.

[0192] Furthermore, according to this embodiment, once the user has chatted up to the final answer of the automatic response, they can select text chat or voice chat with the operator to continue the chat. Therefore, if the user has any questions or doubts that are not resolved by chatting with the chatbot, they can be resolved by continuing to chat with the operator after the chat with the chatbot ends.

[0193] Furthermore, since the user continues chatting with the operator, the chat history between the user and the operator is stored and learned, which can be used to add to and modify the scenario.

[0194] In this embodiment, answers to user questions are generated based on talk scripts generated by machine learning, but this is not limited to this. Instead of talk scripts generated by machine learning, anticipated Q&As may be prepared in advance, and answers to user questions may be generated using the Q&As.

[0195] In addition, in this embodiment, when chatting with the chatbot, text chat is used, but it is also possible to operate an icon containing a pre-prepared question, or to be able to both input text and operate an icon.

[0196] Furthermore, in this embodiment, when a chat according to one scenario is ended, a text chat or a voice chat with the operator can be selectively executed, but this is not necessarily limited to this. If there are multiple scenarios, when a chat according to a certain scenario is ended, it may be possible to select whether to execute a chat according to another scenario or to execute a chat with the operator, and when chatting with the operator is selected, it may be possible to selectively execute a text chat or a voice chat with the operator. In this case, when executing a chat according to another scenario is selected, when the chat according to this other scenario is ended, it may be possible to select whether to execute a chat according to yet another scenario or to execute a chat with the operator.

[0197] In addition, in this embodiment, a scenario-type chatbot is used to automatically answer user questions, but it is also possible to have an artificial intelligence (AI) chatbot such as ChatGPT or Marvin automatically answer user questions.

[0198] In such cases, the end of the chat with the automated response may be determined by the AI, or when a predetermined number of interactions with the user have occurred, or when an error has occurred. The end of the chat with the automated response may be determined by the AI ​​when the AI ​​determines that the user's question has been resolved or when the AI ​​determines that it is unable to answer the question.

[0199] In addition, the chat can be terminated via automatic response when an operator becomes available, at the discretion of the operator monitoring the chatbot exchange, when an exchange of more than a specified amount of text (number of characters) occurs, or when a specific keyword is detected.

[0200] Furthermore, in this embodiment, text chat or voice chat is selectively performed with the operator of one operator terminal, but this is not necessarily limited to this. The operator terminal and its operator performing the text chat and the operator terminal and its operator performing the voice chat may be different terminals and different operators. In such a case, when text chat is selected, the user terminal establishes a connection with the operator terminal performing the text chat, and the user performs text chat with the operator of this operator terminal. Also, when voice chat is selected, the user terminal establishes a connection with the operator terminal performing the voice chat, and the user performs voice chat with the operator of this operator terminal.

[0201] Furthermore, the order of the steps in the flowchart shown in this embodiment can be changed if the same results are obtained.

[0202] Furthermore, the various screens and specific numerical values ​​shown in this specification and drawings are merely examples and can be changed as needed. [Explanation of symbols]

[0203] 10...Dialogue control system 12...Dialogue control device 14...Network 16...User terminal 18...Operator terminal 20, 50, 80...CPU 22, 52, 82...RAM 24, 54, 84...Communication I / F 26, 56, 86... Input / Output I / F 28, 58, 88...input device 30, 60, 90…display device 62, 92...Mike 64, 94...speakers 96...Camera

Claims

1. a user terminal used by a user; a control device having a chatbot function that automatically answers questions from the user; A dialogue control system including an operator terminal used by a human operator who answers questions from the user, The control device a question receiving means for receiving the question transmitted from the user terminal; an answer sending means for generating an answer to the question received by the question receiving means and sending the answer to the user terminal; and a selection means for displaying a selection image on a display device of the user terminal for selecting text chat or voice chat with a human operator when the final response of the automatic response is transmitted to the user terminal by the response transmission means; The user terminal a question transmission means for transmitting the question input by the user to the control device or the operator terminal; a response receiving means for receiving the response transmitted from the control device or the operator terminal; an answer display means for displaying the answer received by the answering means on the display device; and A control system comprising a chat execution means for executing the text chat or the voice chat with the operator terminal used by the operator when the text chat or the voice chat with the operator is selected in the selection image.

2. the user terminal further comprises an avatar display means for displaying images of avatars corresponding to the chatbot and the operator on the display device; The control system according to claim 1 , wherein the avatar image corresponding to the chatbot and the avatar image corresponding to the operator are images of the same character.

3. A control device that controls a chat that automatically answers questions from users, a question receiving means for receiving the question transmitted from the user terminal used by the user; an answer sending means for generating an answer to the question received by the question receiving means and sending the answer to the user terminal; and A control device comprising a selection means for displaying a selection image on a display device of the user terminal for selecting text chat or voice chat with a human operator when the final response during an automatic response is sent to the user terminal by the response sending means.

4. A control program executed by a control device that controls a chat that automatically answers questions from users, a processor of the control device; a question receiving step of receiving the question transmitted from a user terminal used by the user; an answer sending step of generating an answer to the question received in the question receiving step and sending the answer to the user terminal; and A control program that executes a selection step in which, when the final response during an automatic response is sent to the user terminal in the response sending step, a selection image for selecting text chat or voice chat with a human operator is displayed on a display device of the user terminal.

5. A control method for a control device that controls a chat that automatically answers questions from users, comprising: The processor of the control device receiving the question transmitted from a user terminal used by the user; generating and transmitting the answer to the received question to the user terminal; and A control method in which, when the final response during an automatic response is sent to the user terminal, a selection image for selecting text chat or voice chat with a human operator is displayed on a display device of the user terminal.

Images