Information processing system, information processing apparatus, information processing program, and information processing method

Abstract

To make it possible to make a response to suit the preferences of an interaction agent.SOLUTION: In an information processing system 10, a server 12, an operator terminal 16, and a robot 18 cooperate with each other over a network 14. The robot displays an avatar image 28 of an operator on a display device 30, and provides a predetermined service to a service user who is the operator. The operator interacts with an interaction agent through the robot. Customer information for the interaction agent, which includes at least taste information on an avatar of the interaction agent, is displayed on the operator terminal. The operator makes a response while viewing the customer information of the interaction agent, and updates importance of individual information included in the customer information in accordance with a response of the interaction agent.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing program, and an information processing method, and more particularly to an information processing system, an information processing device, an information processing program, and an information processing method for providing a predetermined service using a robot that functions as an avatar of an operator, for example. [Background technology]

[0002] An example of this type of conventional information processing system is disclosed in Patent Document 1. The dialogue support system disclosed in Patent Document 1 includes a robot that faces a user, a database server that manages dialogue support information, and a terminal device operated by an operator. The operator operates the terminal device, and the user faces the robot. Furthermore, when a dialogue is conducted between the operator and the user, dialogue support information extracted from the database by the database server is presented to the operator through the terminal device. A message issued by the operator with reference to the dialogue support information is conveyed to the user through the robot. Furthermore, a message issued by the user to the robot is conveyed to the operator through the terminal device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4798577 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, the operator refers to the dialogue support information and the user faces the robot, realizing an active dialogue led by the operator, but it is unclear whether the appearance of the robot or the way the operator responds using the robot matches the user's preferences. Therefore, there is room for improvement in the content of the dialogue support information and the way it is presented.

[0005] Therefore, a primary object of the present invention is to provide a novel information processing system, an information processing device, an information processing program, and an information processing method.

[0006] Another object of the present invention is to provide an information processing system, an information processing device, an information processing program, and an information processing method that are capable of responding in accordance with the preferences of a dialogue agent. [Means for solving the problem]

[0007] A first invention is an information processing system comprising: an avatar of a robot or computer graphics; an operator terminal used by an operator controlling the movements and speech of the avatar; and an information processing device communicably connected to each of the avatar and the operator terminal, wherein the information processing device comprises dialogue partner information acquisition means for acquiring dialogue partner information from a database about a dialogue partner who faces the avatar and dialogues with the operator, the dialogue partner information including at least preference information regarding the avatar; a first transmission means for transmitting the dialogue partner information acquired by the dialogue partner information acquisition means to the operator terminal; a first reception means for receiving at least one of a control signal from the operator terminal and an operator speech content which is the speech content of the operator; an avatar voice generation means for generating avatar voice to make the avatar speak based on the operator speech content received by the first reception means; a second transmission means for transmitting at least one of the control signal and the avatar voice to the avatar; a second reception means for receiving at least one of an image and an audio of the dialogue partner from the avatar; a third transmission means for transmitting at least one of the image and an audio of the dialogue partner received by the second reception means to the operator terminal; and the operator terminal comprises a third receiving means for receiving the dialogue partner information transmitted by the first transmitting means, a dialogue partner information display means for displaying the dialogue partner information received by the third receiving means, a fourth transmitting means for detecting a control signal for controlling an operation of the avatar input by the operator and transmitting the control signal to the information processing device, a fifth transmitting means for detecting an utterance content for controlling the avatar's utterance input by the operator and transmitting the control signal to the information processing device, a fourth receiving means for receiving at least one of the video and audio of the dialogue partner transmitted by the second transmitting means, a dialogue partner video display means for displaying the video of the dialogue partner received by the fourth receiving means, and an output means for outputting the audio of the dialogue partner received by the fourth receiving means,The information processing system includes a fifth receiving means for receiving at least one of the control signal and the avatar voice transmitted by the second transmitting means, an operation executing means for executing an operation according to the control signal received by the fifth receiving means, an avatar voice output means for outputting the avatar voice received by the fifth receiving means, and a sixth transmitting means for detecting at least one of an image and a voice of a conversation partner and transmitting the image and the voice of the conversation partner to the information processing device.

[0008] A second invention is according to the first invention, and the interlocutor information includes individual information on preferences regarding at least an avatar of the interlocutor.

[0009] A third invention is according to the first or second invention, and the reaction detection means detects the emotion of the dialogue partner toward the avatar voice based on at least one of the video and audio of the dialogue partner, and the update means updates the importance of the individual information extracted by the individual information extraction means based on the emotion of the dialogue partner.

[0010] A fourth invention is according to the third invention, and the updating means increases the importance of the individual information extracted by the individual information extraction means when the emotion of the conversation partner is joy, and decreases the importance of the individual information or does not update it when the emotion of the conversation partner is sadness or anger.

[0011] A fifth invention is an information processing device communicatively connected to each of an avatar of a robot or computer graphics, and an operator terminal used by an operator who controls the movement and speech of the avatar, comprising: a dialogue partner information acquisition means for acquiring dialogue partner information from a database, the dialogue partner information being information about a dialogue partner who faces the avatar and converses with the operator, the dialogue partner information including at least preference information about the avatar; a first transmission means for transmitting the dialogue partner information acquired by the dialogue partner information acquisition means to the operator terminal; a first reception means for receiving at least one of a control signal from the operator terminal and an operator speech content which is the speech content of the operator; an avatar voice generation means for generating an avatar voice to be uttered by the avatar based on the operator speech content received by the first reception means; and at least one of the avatar voice and the video of the dialogue partner to the avatar, second receiving means for receiving at least one of an image and a video of the dialogue partner from the avatar, third transmitting means for transmitting at least one of the image and a video of the dialogue partner received by the second receiving means to an operator terminal, reaction detection means for detecting a reaction of the dialogue partner to the avatar voice based on at least one of the image and a video of the dialogue partner received by the second receiving means, individual information extraction means for extracting individual information included in the dialogue partner information from the latest dialogue history, and update means for updating the importance of the individual information extracted by the individual information extraction means and the individual information set in the avatar based on the reaction of the dialogue partner detected by the reaction detection means.

[0012] A sixth invention is an information processing program for an information processing device communicably connected to each of an avatar of a robot or computer graphics and an operator terminal used by an operator who controls the movement and speech of the avatar, the information processing program including a processor of the information processing device, the processor including: a dialogue partner information acquisition step of acquiring, from a database, dialogue partner information about a dialogue partner who faces the avatar and converses with the operator, the dialogue partner information including at least preference information about the avatar; a first transmission step of transmitting the dialogue partner information acquired in the dialogue partner information acquisition step to the operator terminal; a first reception step of receiving at least one of a control signal from the operator terminal and an operator speech content which is the speech content of the operator; and an avatar voice generation step of generating an avatar voice to be uttered by the avatar based on the operator speech content received in the first reception step. a second transmission step of transmitting at least one of a control signal and an avatar voice to the avatar; a second reception step of receiving at least one of an image and an audio of the dialogue partner from the avatar; a third transmission step of transmitting at least one of the image and an audio of the dialogue partner received in the second reception step to an operator terminal; a reaction detection step of detecting a reaction of the dialogue partner to the avatar voice based on at least one of the image and an audio of the dialogue partner received in the second reception step; an individual information extraction step of extracting individual information included in the dialogue partner information from the latest dialogue history; and an update step of updating the importance of the individual information extracted in the individual information extraction step and the individual information set in the avatar based on the reaction of the dialogue partner detected in the reaction detection step.

[0013] A seventh invention is an information processing method for an information processing system including an avatar of a robot or computer graphics, an operator terminal used by an operator who controls the movements and speech of the avatar, and an information processing device communicatively connected to each of the avatar and the operator terminal, in which the information processing device acquires, from a database, dialogue partner information about a dialogue partner who faces the avatar and dialogues with the operator, the dialogue partner information including at least preference information about the avatar, transmits the acquired dialogue partner information to the operator terminal, receives at least one of a control signal and an operator utterance content which is the speech content of the operator from the operator terminal, generates avatar voice to make the avatar speak based on the received operator utterance content, transmits at least one of the control signal and the avatar voice to the avatar, receives at least one of an image and an audio of the dialogue partner from the avatar, transmits at least one of the received image and audio of the dialogue partner to the operator terminal, and an information processing method in which the reaction of the dialogue partner to the avatar voice is detected based on the one of the signals, individual information contained in the dialogue partner information is extracted from the latest dialogue history, and importance of the extracted individual information and the individual information set for the avatar is updated based on the detected reaction of the dialogue partner; an operator terminal receives the dialogue partner information from the information processing terminal, displays the received dialogue partner information, detects a control signal input by the operator for controlling the operation of the avatar and transmits it to the information processing device, detects utterance content input by the operator for controlling the avatar's utterance and transmits it to the information processing device, receives at least one of an image and an audio of the dialogue partner from the information processing device, displays the received image of the dialogue partner, and outputs the received audio of the dialogue partner; and the avatar receives at least one of a control signal and an avatar audio from the information processing device, performs an operation in accordance with the received control signal, outputs the received avatar audio, and detects at least one of the image and audio of the dialogue partner and transmits it to the information processing device. Effect of the Invention

[0014] According to the present invention, it is possible to respond in a manner that matches the preferences of the dialogue agent.

[0015] The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of the embodiments with reference to the drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of an electrical configuration of the server shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing an example of an electrical configuration of the operator terminal shown in FIG. [Figure 4] FIG. 4 is a block diagram showing an example of the electrical configuration of the robot shown in FIG. [Diagram 5] FIG. 5 is a diagram showing an example of a control screen displayed on the operator terminal. [Figure 6] FIG. 6 is a diagram showing an example of a customer information screen displayed on the operator terminal. [Figure 7] FIG. 7 is a diagram showing an example of customer information stored in the customer information DB. [Figure 8] FIG. 8 is a diagram showing an example of a memory map of the RAM of the server shown in FIG. [Figure 9] FIG. 9 is a diagram showing an example of a memory map of the RAM of the operator terminal shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of a memory map of the RAM of the robot shown in FIG. [Figure 11] FIG. 11 is a flow diagram showing a first part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. [Figure 12] FIG. 12 is a flowchart showing a second part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and follows FIG. [Figure 13] FIG. 13 is a flow chart showing a third part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and follows FIG. [Figure 14]FIG. 14 is a flowchart showing a fourth part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and follows FIG. [Figure 15] FIG. 15 is a flow chart showing a fifth part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and is a sequel to FIG. [Figure 16] FIG. 16 is a flow chart showing a sixth part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and is a sequel to FIG. [Figure 17] FIG. 17 is a flow chart showing a seventh part of an example of information processing during execution of a predetermined service by the CPU of the server shown in FIG. 2, and is a sequel to FIG. 11 and FIG. [Figure 18] FIG. 18 is a flowchart showing a first part of an example of information processing by the CPU of the operator terminal shown in FIG. [Figure 19] FIG. 19 is a flowchart showing a second part of an example of information processing by the CPU of the operator terminal shown in FIG. 3, and follows FIG. [Figure 20] FIG. 20 is a flowchart showing a third part of an example of information processing by the CPU of the operator terminal shown in FIG. 3, and follows FIG. [Figure 21] FIG. 21 is a flow chart showing a part of an example of control processing by the CPU of the robot shown in FIG. [Figure 22] FIG. 22 is a flowchart showing another part of an example of the control process of the CPU of the robot shown in FIG. 4, which follows FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Referring to FIG. 1, an information processing system 10 of this embodiment includes a server 12, which is communicatively connected to a plurality of operator terminals 16 and a robot 18 via a network 14.

[0018] Each of the multiple operator terminals 16 is operated by an operator and is mainly used to remotely control the robot 18. In this embodiment, the operator is a person (service user) who uses a predetermined service using an avatar provided by the server 12. The robot 18 functions as an avatar of the operator. In the information processing system 10 of this embodiment, one of the multiple operator terminals 16 and the robot 18 are linked via the network 14, and the operator of the one operator terminal 16 interacts with a dialogue agent, which will be described later, through the robot 18. However, the operator terminal 16 that is linked with the robot 18 is the operator terminal 16 that is assigned to log in to the server 12 and use the robot 18.

[0019] The predetermined service is a service in which the robot 18 placed in a predetermined environment (for example, a corporate office, a store, a museum, an art gallery, an aquarium, or a science museum) functions as an avatar for a remote operator, and the avatar, i.e., the robot 18, is used by moving freely around the predetermined environment. However, when the robot 18 performs a reception service, it may be fixedly placed at a predetermined location in the predetermined environment.

[0020] In this case, the operator interacts with a human in the office through the robot 18. The operator also listens to explanations from and interacts with human guides in stores, museums, art galleries, aquariums, science museums, etc. through the robot 18. In other words, the human interacting with the operator is the operator's conversation partner (i.e., a conversation agent).

[0021] 1 shows one robot 18, but in reality, multiple robots 18 are provided. When an operator uses a specific service, one robot 18 is assigned to the operator (operator terminal 16) by the server 12. The following description will be given using one robot 18, but the same content applies to the other robots 18.

[0022] The server 12 is an example of an information processing device, and may be a general-purpose server. In this embodiment, the server 12 operates a website that provides a predetermined service using avatars. A history database (hereinafter referred to as "history DB") 12a and a customer information database (hereinafter referred to as "customer information DB") 12b are connected to the server 12.

[0023] The history DB 12a stores image data and voice data of the dialogue agent, which are transmitted and received between the operator terminal 16 and the robot 18 via the server 12, as well as the operator's speech data and the history of operation commands instructed by the operator.

[0024] The customer information DB 12b stores information about each dialogue agent (hereinafter referred to as "customer information"). In this embodiment, the customer information (corresponding to "dialogue partner information") includes a plurality of pieces of individual information, which are classified into personal attributes, social attributes, preference information, and important dialogues. The customer information will be described in detail later.

[0025] The history DB 12a and the customer information DB 12b may be provided in a HDD built into the server 12, or may be provided so as to be communicable via the network .

[0026] The network 14 is composed of an IP network including the Internet, and an access network for accessing the IP network. The access network may be a public telephone network, a mobile phone network, a wired LAN, a wireless LAN, or a CATV (Cable Television) network.

[0027] Each of the multiple operator terminals 16 is an information processing device other than the server 12, and is, for example, a desktop PC. In another example, a general-purpose terminal such as a smartphone, a tablet PC, or a notebook PC can be used as the operator terminal 16. The operator terminals 16 are information processing devices of the same type or different types.

[0028] The robot 18 is a so-called telepresence (or teleexistence) robot capable of autonomous behavior, and a commercially available mobile robot can be used. As an example, the robot 18 can be a robot Double 3 sold by Double Robotics, Inc. However, the robot 18 is not limited to such a specific robot, and a robot such as that disclosed in JP 2020-004182 A can also be used.

[0029] As an example of a method for making the robot 18 act autonomously, the method disclosed in Japanese Patent Application Laid-Open No. 2009-131914 can be adopted. However, autonomous behavior means that the robot 18 performs a communication behavior including at least one of an action including movement and speech without being operated by an operator. In this embodiment, the communication behavior is performed according to a scenario generated according to the role of the robot 18 or a predetermined service to be provided. In this embodiment, the communication behavior is determined by the server 12, and the server 12 transmits at least one of an action command and voice data ("avatar voice data" to be described later) for the communication behavior to the robot 18. When a known technology is used as a method for making the robot 18 act autonomously, the necessary circuit components and data are appropriately provided in the server 12. In this embodiment, when there is no operator, the robot 18 performs the autonomous behavior under the control of the server 12.

[0030] The robot 18 can also be remotely operated by an operator. In this embodiment, the operator operates the operator terminal 16 to send commands (hereinafter, referred to as "operation commands") for causing the robot 18 to execute a task (operation) to the robot 18 via the server 12.

[0031] Furthermore, the operator terminal 16 detects the operator's voice with the microphone 82, and transmits the detected voice to the robot 18 via the server 12. In this embodiment, a voice based on the operator's voice is output from the speaker 112 of the robot 18. The voice based on the operator's voice is the operator's voice converted into a predetermined male voice, female voice, or robot-like voice. However, the operator's voice can also be output directly from the speaker 112 of the robot 18.

[0032] The robot 18 displays an image of a predetermined CG (Computer Graphics) character on the display device 30. In this embodiment, an image of the upper three figures or the upper body including the face of the CG character is displayed on the display device 30. As an example, a plurality of CG characters are prepared, each having the appearance of a man, a woman, or a robot.

[0033] Meanwhile, the robot 18 takes an image including the dialogue agent with the camera 122, and transmits the taken image (hereinafter referred to as "robot image") to the operator terminal 16 via the server 12. Therefore, in the operator terminal 16, an image seen from the robot 18, i.e., an image of the dialogue agent, is displayed on the display device 80. In addition, the robot 18 detects surrounding sounds including the voice of the dialogue agent with the microphone 110, and transmits the detected sounds to the operator terminal 16 via the server 12. Therefore, sounds including the voice of the dialogue agent are output from the speaker 112 of the operator terminal 16.

[0034] Therefore, the operator can interact with the interactive agent while looking at its face, and can operate the robot 18 while looking at the environment.

[0035] The robot 18 also includes a base 20, and an ultrasonic sensor 22, which is one of the movement sensors, is provided on the front side of the base 20. A wheel (left wheel) 24L and a wheel (right wheel) 24R are provided on both ends of the base 20 in a rotatable manner.

[0036] An upwardly extending pole 26 is provided on the upper surface of the base 20, and at the upper end of the pole 26 is a display device 30 that displays an image (hereinafter referred to as an "avatar image") 28 (the image of the CG character mentioned above) corresponding to the operator.

[0037] The display device 30 is provided with a touch panel, a camera, a microphone, a speaker, etc., and details will be described with reference to the block diagram shown in Fig. 4. The robot 18 is a battery-powered robot, and the battery is housed in the pole 26, for example.

[0038] Fig. 2 is a block diagram showing the electrical configuration of the server 12 shown in Fig. 1. As shown in Fig. 2, the server 12 includes a CPU 50, which is connected to a RAM 52, a communication unit 54, and an input / output interface (hereinafter referred to as "input / output I / F") 56 via an internal bus.

[0039] The CPU 50 is responsible for the overall control of the server 12. However, a SoC (System-on-a-chip) including multiple functions such as a CPU function and a GPU (Graphics Processing Unit) function may be provided instead of the CPU 50. The RAM 52 is the main memory of the server 12, and is used as a work area and a buffer area for the CPU 50.

[0040] Although not shown in the drawings, the server 12 also includes auxiliary storage devices such as a HDD and a ROM. However, instead of the HDD and the ROM, or in addition to the HDD and the ROM, a non-volatile memory such as an SSD may be used.

[0041] Under the control of the CPU 50, the communication unit 54 transmits and receives control signals (including operation commands) and data to and from external computers such as the operator terminal 16 and the robot 18 via the network 14. However, as the communication unit 54, not only a wired LAN but also a wireless LAN, Bluetooth (registered trademark), etc. can be used.

[0042] An input device 58 and a display device 60 are connected to the input / output I / F 56. A keyboard and a computer mouse are used as the input device 58. The display device 60 is an LCD or an organic EL display.

[0043] The input / output I / F 56 outputs operation data (or operation information) input from the input device 58 to the CPU 50, and also outputs image data generated by the CPU 50 to the display device 60, causing the display device 60 to display a screen corresponding to the image data.

[0044] The electrical configuration of the server 12 shown in FIG. 2 is merely an example and is not necessarily limited to this.

[0045] Fig. 3 is a block diagram showing the electrical configuration of the operator terminal 16 shown in Fig. 1. As shown in Fig. 3, the operator terminal 16 includes a CPU 70, which is connected to a RAM 72, a communication unit 74, and an input / output I / F 76 via an internal bus.

[0046] The CPU 70 is responsible for overall control of the operator terminal 16. However, an SoC including multiple functions such as a CPU function and a GPU function may be provided instead of the CPU 70. The RAM 72 is a main storage device of the operator terminal 16, and is used as a work area and a buffer area for the CPU 70.

[0047] Although not shown in the drawings, the operator terminal 16 also includes auxiliary storage devices such as a HDD and a ROM. However, instead of the HDD and the ROM, or in addition to the HDD and the ROM, a non-volatile memory such as an SSD may be used.

[0048] Under the control of the CPU 70, the communication unit 74 transmits and receives control signals (including operation commands) and data to and from external computers such as the server 12 and the robot 18 via the network 14. However, as the communication unit 74, not only a wired LAN but also a wireless LAN, Bluetooth (registered trademark), etc. can be used.

[0049] An input device 78, a display device 80, a microphone 82, and a speaker 84 are connected to the input / output I / F 76. A keyboard and a computer mouse are used as the input device 78. A touch panel may also be provided. The display device 80 is an LCD or an organic EL display.

[0050] However, when a smartphone is used as the operator terminal 16, the input device 78 is a touch panel and hardware buttons. The touch panel is a general-purpose touch panel, and any type of touch panel can be used, such as a capacitive type, an electromagnetic induction type, a resistive film type, or an infrared type.

[0051] The input / output I / F 76 converts the operator's voice detected by the microphone 82 into digital voice data (hereinafter referred to as "operator voice data") and outputs it to the CPU 70, and also converts the voice data output by the CPU 70 into an analog voice signal and outputs it from the speaker 84. However, in the embodiment, the voice data output from the CPU 50 is voice data received from the robot 18 via the server 12 (hereinafter referred to as "robot voice data"). In addition, the input / output I / F 76 outputs operation data (or operation information) input from the input device 78 to the CPU 70, and outputs image data generated by the CPU 70 to the display device 80, causing the display device 80 to display a screen or image corresponding to the image data. However, in this embodiment, the robot image data received from the robot 18 via the server 12 is also output to the control screen 200 (see FIG. 5) displayed on the display device 80. However, the robot image data is image data of an image (moving image or still image) captured by the camera 122 of the robot 18.

[0052] The operator terminal 16 also includes a sensor I / F 86 and a camera 88. The CPU 70 is connected to the camera 88 via the bus and the sensor I / F 86. The camera 88 is a camera that uses an imaging element such as a CCD or CMOS. Although not shown in the figure, the camera 88 is provided near the display device 80 and captures an image of a predetermined range (a range determined by the angle of view) in front of the display device 80. Therefore, the camera 88 can capture an image of the operator facing the display device 80.

[0053] 3 is an example, and does not need to be limited to the electrical configuration of the operator terminal 16. As described above, when the operator terminals 16 are different types of information processing devices, the electrical configurations may also be different.

[0054] In addition, if the operator terminal 16 is a smartphone, it is equipped with a calling circuit for making calls via a mobile phone network, or a mobile phone network and a public telephone network. However, in this embodiment, such calls are not made, so they are not shown in the figure.

[0055] Fig. 4 is a block diagram showing the electrical configuration of the robot 18 shown in Fig. 1. As shown in Fig. 4, the robot 18 includes a CPU 100, which is connected to a RAM 102, a communication unit 104, and an input / output I / F 106 via an internal bus.

[0056] The CPU 100 is responsible for the overall control of the robot 18. However, an SoC including multiple functions such as a CPU function and a GPU function may be provided instead of the CPU 100. The RAM 52 is a main storage device of the operator terminal 16, and is used as a work area and a buffer area for the CPU 100.

[0057] Although not shown in the drawings, the robot 18 also includes auxiliary storage devices such as a HDD and a ROM. However, instead of the HDD and the ROM, or in addition to the HDD and the ROM, a non-volatile memory such as an SSD may be used.

[0058] Under the control of the CPU 100, the communication unit 104 transmits and receives control signals (including operation commands) and data to and from external computers such as the server 12 and the operator terminal 16 via the network 14. However, the communication unit 104 may use wireless LAN, Bluetooth (registered trademark), or the like.

[0059] The input / output I / F 106 is connected to an input device 108, a display device 30, a microphone 110, and a speaker 112. An emergency stop hardware button or the like is provided as the input device 108. Further, as the input device 108, a touch panel is provided over the display surface of the display device 30. The display device 30 is an LCD or an organic EL display. As an example, the touch panel is a general-purpose touch panel, and any type such as a capacitive type, an electromagnetic induction type, a resistive film type, or an infrared type can be used.

[0060] The input / output I / F 106 converts the voice of the dialogue agent detected by the microphone 110 into digital voice data and outputs it to the CPU 100, and also converts the voice data output by the CPU 100 into an analog voice signal and outputs it from the speaker 112.

[0061] However, in this embodiment, the voice data output from the CPU 100 is voice data received from the server 12. The server 12 generates voice data based on the speech data received from the operator terminal 16 (hereinafter referred to as "operator speech data") and transmits it to the robot 18. In this embodiment, the operator speech data is the voice of the operator or text data of the speech content uttered in accordance with a speech command input by the operator.

[0062] In addition, the input / output I / F 106 outputs operation data (or operation information) input from the input device 108 to the CPU 100, and outputs image data generated (or output) by the CPU 100 to the display device 30, causing the display device 30 to display a screen or image corresponding to the image data.

[0063] However, in this embodiment, image data of an avatar image 28 for a specified CG character that performs an action in response to an action command transmitted from the server 12 is generated by the robot 18, and the avatar image 28 corresponding to the generated image data is displayed on the display device 30.

[0064] The actions corresponding to the action commands include actions that are performed unconsciously (hereinafter referred to as "unconscious actions"). Unconscious actions refer to physiological actions such as blinking and breathing, and actions due to habits such as touching hair or face. The action commands are transmitted from the server 12 when the robot 18 performs autonomous actions, and are transmitted from the operator terminal 16 via the server 12 when the robot 18 is remotely controlled by an operator.

[0065] The robot 18 also includes a motor driver 114, which is connected to the CPU 100 via a bus. The motor driver 114 controls the driving of a right wheel motor 116 that rotates the right wheel 24R of the robot 18, and also controls the driving of a left wheel motor 118 that rotates the left wheel 24L of the robot 18, in accordance with instructions from the CPU 100.

[0066] The robot 18 also includes a sensor I / F 120 , a camera 122 , a laser range finder (LRF) 124 , an LRF 126 , an ultrasonic sensor 22 and an inertial measurement unit 128 .

[0067] The CPU 100 is connected to a camera 122, an LRF 124, an LRF 126, an ultrasonic sensor 22, and an inertial measurement unit 128 via a bus and a sensor I / F 120, respectively.

[0068] The camera 122 is a camera using an imaging element such as a CCD or CMOS. Although omitted in Fig. 1, the camera 122 is provided at the upper or lower end of the display device 30, and captures an image of a predetermined range (a range determined by the angle of view) in front of the robot 18. The camera 122 therefore captures an image of a dialogue agent that faces the robot 18 and communicates with the operator through the robot 18 and / or an environment in which the robot 18 is placed or moves.

[0069] The LRF 124 generally measures distance by irradiating a laser beam and measuring the time it takes for the laser beam to reflect off an object (such as an object or a human body) and return. The LRF 124 of this embodiment (the same applies to the LRF 126 described later) includes a mirror (not shown) that rotates around an axis within a range of, for example, ±45 degrees, and can perform measurement (scan with the laser beam) while changing the path of the laser beam by, for example, 0.6 degrees at a time with this rotating mirror. Hereinafter, the plane scanned with the laser beam by the LRF 124 is called the scan plane. In addition, the distance that can be measured by the LRF 124 is limited to a predetermined distance R (for example, 15 m) or less so that the laser beam does not affect the human eye. For this reason, the measurement area (i.e., the scan plane) of the LRF 124 is, for example, a sector, that is, a sector with a central angle of 90 degrees with a radius of the predetermined distance R. The central angle of the scan plane (sector) is not limited to 90 degrees, and may be, for example, 180 degrees or 360 degrees.

[0070] In this embodiment, LRF 124 is provided for the purpose of tracking a specific human, etc. LRF 126 is for the autonomous movement of this robot 18. LRF 124 may be used in combination with LRF 126. In other words, one LRF may be used when robot 18 is moving and when robot 18 has stopped moving and is engaged in a conversation using an avatar image (hereinafter referred to as "avatar image") 28.

[0071] Based on the output of the LRF 124, the robot 18 can detect the distance and orientation of an object relative to the robot 18 or the LRF 124. Therefore, the robot 18 can detect (calculate) the current position of the object based on its own detected current position and the distance and orientation of the object detected using the LRF 124.

[0072] Furthermore, the robot 18 creates a map of the environment in which the robot 18 moves by SLAM (Simultaneous Localization and Mapping) based on the output of the LRF 126, estimates the current position in the created map of the environment, and navigates itself based on the results. In this embodiment, the LRF 126 measures the distance to an obstacle in the horizontal direction, so a two-dimensional map of obstacles around the robot 18 is created. Therefore, the map of the environment created by the robot 18 is a two-dimensional map of the robot 18 (LRF 126) viewed from directly above. Hereinafter, the map of the environment created by the robot 18 will be referred to as a "created map." Furthermore, the robot 18 stores a map of the environment in which the robot 18 is placed (in this embodiment, a "set map") that has been created in advance. The current position of the robot 18 in the set map (or the created map) is estimated by matching the origin in the created map with the origin in the set map and matching the orientation in the created map with the orientation in the set map.

[0073] The ultrasonic sensor 22 is a general-purpose distance sensor that uses ultrasonic waves. The ultrasonic sensor 22 measures the distance to an object (human or object) present in front of the robot 18. As an example, the detection range of the ultrasonic sensor 22 in the horizontal direction is 10 degrees to 45 degrees, and the measurable distance is 5 m to 15 m. As described above, since the LRF 128 is provided, the ultrasonic sensor 22 does not need to be provided.

[0074] The inertial measurement unit 128 is a device that detects three-dimensional inertial motion (translational motion and rotational motion in three orthogonal axis directions), and detects translational motion using an acceleration sensor and rotational motion using an angular velocity (gyro) sensor.

[0075] It should be noted that the electrical configuration of the robot 18 shown in FIG. 4 is merely an example and is not necessarily limited to this.

[0076] As described above, the robot 18 is capable of autonomous behavior, and can move from a start point (movement starting point) to a goal point (target point) that is determined in advance in the environment in which it is placed, while searching for a route. However, one or more waypoints may be set between the start point and the goal point. In addition, standard methods, such as the Floyd-Warshall algorithm, Dijkstra's algorithm, A* algorithm, potential method, and RRT (Rapidly exploring Random Tree), can be used as a method for searching the route. The start point, the goal point, and one or more waypoints are designated in advance by an administrator or operator of the server 12.

[0077] The robot 18 can also move freely in the environment in which it is placed, according to remote control by an operator. Controlling the movement of the robot 18 using the operator terminal 16 is already well known, and detailed explanation of the remote control will be omitted here.

[0078] 5 is a screen for remotely controlling the robot 18. The control screen 200 is provided with display areas 202, 204, 206, and 208. The control screen 200 is also provided with a display area 250 and a button 252 below the display areas 206 and 208, and further provided with a button 260 below these.

[0079] The display area 202 is an area for displaying images (moving images or still images) captured by the camera 124 provided on the robot 18.

[0080] The display area 204 displays a number of buttons for inputting commands (i.e., motion commands) that control the movement of the robot 18. In this embodiment, the display area 204 displays buttons 210, 212, 214, 216, and 218. The button 210 is a button for moving the robot 18 forward. The button 212 is a button for making the robot 18 turn left or rotate left. The button 214 is a button for making the robot 18 turn right or rotate right. The button 216 is a button for stopping the robot 18. The button 218 is a button for making the robot 18 move backwards.

[0081] Although a detailed description will be omitted, when button 212 is turned on while button 210 or button 218 is turned on, the robot 18 turns left while moving forward or backward. When button 214 is turned on while button 210 or button 218 is turned on, the robot 18 turns right while moving forward or backward. When button 212 is turned on while the robot 18 is stopped, the robot 18 changes its direction of movement counterclockwise. When button 214 is turned on while the robot 18 is stopped, the robot 18 changes its direction of movement clockwise.

[0082] In this embodiment, when buttons 210 , 212 , 214 , 216 , 218 , and the like displayed in the display area 204 are operated, the operation commands assigned to the respective buttons 210 , 212 , 214 , 216 , and 218 are transmitted to the server 12 .

[0083] The display area 206 displays a number of buttons for inputting motion commands that control the communication behavior of the robot 18. In this embodiment, buttons 220, 222, 224, 226, 228, and the like are displayed in the display area 206. The button 220 is a button for making the robot 18 nod. The button 222 is a button for making the robot 18 shake its head. The button 224 is a button for making the robot 18 bow. The button 226 is a button for making the robot 18 wave its hand. The button 228 is a button for making the robot 18 raise its right hand.

[0084] However, since the robot 18 in this embodiment does not have parts corresponding to arms, nor does it have joints corresponding to the waist and neck, communication actions such as nodding, shaking the head, bowing, waving, and raising the right hand can be performed by moving the avatar image 28.

[0085] Although not shown in the figure, a plurality of other buttons are provided for inputting operation commands for controlling other communication actions, and the other plurality of buttons are displayed by scrolling the display area 206 .

[0086] In this embodiment, when buttons 220 , 222 , 224 , 226 , 228 , etc. displayed in the display area 206 are operated, the operation commands assigned to the respective buttons 220 , 222 , 224 , 226 , and 228 are transmitted to the server 12 .

[0087] The display area 208 displays a plurality of buttons for inputting commands (i.e., speech commands) for controlling the speech of the robot 18. Speech commands for making the robot 18 speak standard words or sentences such as greetings and calls are assigned to the plurality of buttons displayed (provided) in the display area 208. In this embodiment, buttons 230, 232, 234, 236, 238, and the like are displayed in the display area 208. The button 230 is a button for making the robot 18 speak the greeting "Good morning." The button 232 is a button for making the robot 18 speak the greeting "Hello." The button 234 is a button for making the robot 18 speak the greeting "It's been a while." The button 236 is a button for making the robot 18 speak "Please take my temperature." The button 238 is a button for making the robot 18 speak "Please disinfect with alcohol." When a button 230-238 or the like is turned on, an instruction is given to the server 12 to make the robot 18 speak the text of a standard word or sentence assigned to the turned-on button (230-238 or the like) and to make the robot 18 speak the text.

[0088] Although not shown in the figure, a plurality of other buttons are provided for inputting speech commands for controlling other speech, and the plurality of other buttons are displayed by scrolling the display area 208 .

[0089] In this embodiment, when buttons 230 , 232 , 234 , 236 , 238 , and the like displayed in the display area 208 are operated, the speech commands assigned to the respective buttons 230 , 232 , 234 , 236 , and 238 are transmitted to the server 12 .

[0090] The display area 250 is an area for inputting speech content other than standard words and sentences as text. The button 252 is a button for inputting a speech command for making the robot 18 speak the speech content input or displayed in the display area 250, i.e., the text content.

[0091] In this embodiment, when the button 252 is operated, a speech command is sent to the server 12 to cause the robot 18 to speak the contents of the text displayed in the display area 250 .

[0092] The button 260 is a button for instructing the end of remote operation of the robot 18. As an example, when the button 260 is turned on, an instruction (or a notification) is given to the server 12 that the operator of the operator terminal 16 on which the control screen 200 is displayed is to end the remote operation of the robot 18.

[0093] 6 is a diagram showing an example of a customer information screen 300. This customer information screen 300 is displayed alongside the above-mentioned control screen 200. The customer information screen 300 is a screen for displaying customer information about a customer who faces the robot 18, i.e., a dialogue agent.

[0094] 6, customer information screen 300 is provided with display areas 302, 304, 306, and 308. In addition, at the bottom of customer information screen 300, an input field (display area) 310, a button 312, and a button 314 are provided.

[0095] The display area 302 is an area for displaying a face image of the customer, i.e., the dialogue agent. The display area 304 is an area for displaying basic information of the dialogue agent. The basic information is information such as personal attributes and social attributes of the dialogue agent. The display area 306 is an area for displaying response information of the dialogue agent. The response information is information about preferences of the dialogue agent. The display area 308 is an area for displaying important dialogue history.

[0096] The input field 310 is a field or display area for inputting an important dialogue history. The button 312 is a button for registering the content entered or displayed in the input field 310, i.e., the text of the important dialogue history, as an important dialogue in the customer information. The button 314 is a button for deleting the content entered or displayed in the input field 310, i.e., the text of the important dialogue history, from the important dialogue in the customer information.

[0097] In this embodiment, the server 12 authenticates the dialogue agent based on the robot image data transmitted from the robot 18, acquires customer information of the successfully authenticated dialogue agent from the customer information DB 12b, and transmits the acquired customer information to the operator terminal 16 that remotely operates the robot 18. The operator terminal 16 displays the above-mentioned customer information screen 300 based on the customer information received from the server 12. In this embodiment, the server 12 authenticates the dialogue agent by determining whether customer information having a face image matching a face image included in the robot image data is registered in the customer information DB 12b. The server 12 determines that the authentication was successful when customer information having a face image matching a face image included in the robot image data is registered in the customer information DB 12b.

[0098] However, if the result of authentication shows that the customer information of the dialogue agent is not registered in the customer information DB 12b, the server 12 determines that the authentication has failed, registers the customer information of the dialogue agent as a new customer, and transmits the registered customer information to the operator terminal 16. When registering a dialogue agent as a new customer, the server 12 issues a customer ID, generates customer information including a facial image associated with the issued customer ID, and registers it in the customer information DB 12b. The facial image included in the newly registered customer information is the facial image included in the robot image data.

[0099] As shown in Fig. 7, the customer information includes a plurality of individual pieces of information classified into personal attributes, social attributes, preference information, and important conversations for each customer, i.e., each dialogue agent. In addition, identification information of the customer, i.e., the dialogue agent (hereinafter, referred to as "customer ID") is added to each piece of customer information. As described above, the customer ID is issued by the server 12 when registering customer information for a new customer.

[0100] The personal attributes are individual information about the customer, that is, the dialogue agent, and include a face image, name, sex, age, means of transportation, place of residence, destination, and date of previous visit (the most recent date in the dialogue history).

[0101] The social information is individual information about the affiliation and social status of the customer, that is, the dialogue agent, and includes the name and classification of the organization to which the customer belongs, the job title, and the like.

[0102] The preference information is individual information about the customer's preferences regarding the avatar of the conversation agent or the operator's response using the avatar, and includes preferences for how to be called, voice preferences, dialect preferences, clothing preferences, and the like.

[0103] However, for preference information, there may be multiple individual information for one piece of preference information. For example, preferences for how to be addressed include individual information such as "san" or "sama" and job title. Therefore, as preferences for how to be addressed, "san", "sama" and job title are each registered as separate individual information. The same is true for other preference information. Preferences for voice include individual information such as male voice, female voice and robotic voice. Preferences for dialect include individual information such as Kanto dialect, Kinki dialect and standard Japanese. Preferences for clothing include individual information such as uniform and casual clothes.

[0104] The important dialogue history is individual information about important dialogues among the dialogue history with the customer, i.e., the dialogue agent. In this embodiment, important dialogues are registered and deleted by the operator.

[0105] The customer information and its details are merely an example and need not be limited. The customer information may further include other individual information. For example, biometric information is provided as another classification, and the biometric information includes individual information such as the health condition, audibility, and emotions of the dialogue agent. It is considered that by increasing the individual information, the dialogue agent can respond more appropriately. However, when biometric information is included, it is necessary to provide sensors for detecting the health condition and audibility in the robot 18 or in the environment in which the robot 18 is placed, and to provide a program for analyzing the health condition and audibility from the detected values ​​in the robot 18 or the server 12 as appropriate.

[0106] In this embodiment, a robot 18 is used to provide a predetermined service, and when a conversation takes place with a conversation agent, the robot responds based on customer information about the conversation agent.

[0107] In addition, an importance level is added to each piece of customer information. The importance level indicates the degree of interest the dialogue agent has in the corresponding individual information. When the robot 18 acts autonomously, its communication behavior is controlled by the server 12 in accordance with the individual information having a high importance level. When the robot 18 is remotely controlled, the robot 18 presents the individual information having a high importance level to the operator in priority, thereby encouraging the operator to control the robot in accordance with the individual information having a high importance level.

[0108] However, since the facial image is information that is always displayed in the display area 302 of the customer information screen 300, no importance level is set for this facial image. Also, since an important conversation is important, no importance level is set for it.

[0109] As described above, when the customer information of a new customer is registered, it contains only a face image, so in this embodiment, other individual information is added (registered) to the customer information by using the robot 18 to respond to the conversation agent by autonomous action or remote control, and the individual information extracted from the content of the conversation and the individual information related to the avatar's preferences set during the conversation are added to the customer information. At this time, importance (initial value=1 in this embodiment) is added to each individual information.

[0110] As an example of a method for extracting and classifying individual information from the contents of a conversation, the following method can be adopted: The text of the conversation is subjected to morphological analysis to separate it into morphemes, and the character strings of the morphemes are classified by referring to preregistered dictionary data for classification.

[0111] However, if the dialogue is voice data, it is converted into text by voice recognition. When recognizing the voice of the dialogue agent and the voice of the operator, the inference device disclosed in Japanese Patent Application Laid-Open No. 2021-157145 described later can be used.

[0112] The dictionary data for classification is data in which a plurality of character strings that have been determined in advance to be registered as individual information are stored in association with the classification in the customer information and the type of individual information.

[0113] Incidentally, if names were included in the dictionary data for classification, the amount of data would become enormous, so for names, the autonomously acting robot 18 asks questions according to a scenario, or a remote operator asks questions, and the voice spoken by the dialogue agent in response is recognized, and the name, which is individual information, is extracted and classified into personal attributes.

[0114] In addition, as a method for extracting individual information from the contents of the dialogue, a known method such as the method disclosed in JP 2020-071675 A can be adopted. In addition, as a method for classifying the extracted individual information, a known method such as the method disclosed in JP 2021-018489 A can be adopted. When using a known technology as a method for extracting or classifying individual information from the contents of the dialogue, the necessary circuit components and data are appropriately provided in the server 12.

[0115] Furthermore, since the history of the dialogue is registered in the history DB 12a, the administrator or operator of the information processing system 10 or the server 12 may add (register) individual information.

[0116] When the robot 18 performs a communicative action in accordance with remote control (hereinafter referred to as "remote mode"), as described above, the robot image data and robot voice data transmitted from the robot 18 are received by the operator terminal 16 used by the operator remotely operating the robot 18 via the server 12, and the robot image data and robot voice data are output by this operator terminal 16, enabling the operator to see the surroundings of the robot 18, grasp the situation around the robot 18, and interact with the interactive agent facing the display device 30 of the robot 18.

[0117] However, the robot image data and the robot voice data are added with identification information (robot ID) of the robot 18. Therefore, the server 12 identifies the robot 18 to which the robot ID has been assigned from the robot image data and the robot voice data, and also identifies the operator who operates the robot 18 and the operator terminal 16 used by this operator.

[0118] Similarly, the operator utterance data includes identification information (operator ID) of the operator, so that the server 12 identifies the operator from the operator ID added to the operator utterance data.

[0119] In the server 12, by identifying the robot 18 and the operator, when storing image data, voice data or operator utterance data, and operation commands in the RAM 52, the image data of the interactive agent during an interaction, the voice data of the interactive agent during an interaction and the operator utterance data, and the operation commands instructed during an interaction can be recorded in the history DB 12a as a collective history. However, the image data and voice data of the interactive agent during an interaction are robot image data and robot voice data transmitted from the identified robot 18.

[0120] In this embodiment, as described above, when an operator, i.e., a service user, utilizes a predetermined service, the operator interacts with the dialogue agent. That is, the predetermined service includes a service (dialogue service) in which the dialogue agent and the operator interact with each other via the robot 18. The server 12 also stores in the history DB 12a a history of the dialogue between the dialogue agent and the operator, a history of the operator's operations (operation commands in this embodiment), and a history of images of the dialogue agent and the operator.

[0121] As an example, the dialogue history, operation history, and image history of the dialogue agent are recorded when the operator is interacting with the dialogue agent, in order to extract individual information contained in the dialogue content between the operator or robot and the dialogue agent through autonomous behavior or remote control, and not only to register the extracted individual information but also to update the importance of the extracted individual information.

[0122] In this embodiment, as described above, the robot 18 performs operations in a case where it acts autonomously without the assistance of an operator (hereinafter referred to as an "autonomous mode") or in a remote mode. The autonomous mode or the remote mode is set by the server 12.

[0123] The server 12 sets the robot 18 to the autonomous mode when there is no operator remotely operating the robot 18 that provides a predetermined service. In this case, the server 12 instructs the robot 18 to perform operations, etc. in the autonomous mode. Therefore, the robot 18 performs operations, etc. in the autonomous mode.

[0124] In the autonomous mode, the server 12 authenticates (or identifies) the interactive agent from the facial image contained in the robot image data, obtains customer information about the authenticated interactive agent from the customer information DB 12b, and uses the obtained customer information to automatically control the movement and communication behavior of the robot 18.

[0125] As an example, in the autonomous mode, the avatar is controlled using information of high importance among the customer information. The server 12 controls the robot 18 to converse using information of high importance among the personal information, social information, and preference information. At this time, it is set that an acoustic model according to the voice preference included in the preference information is used, that text about the speech content according to the dialect preference is used, and that an avatar dressed in clothes according to the clothing preference is displayed. However, in this embodiment, since the avatar image 28 is generated by the robot 18, the robot 18 is instructed to display an avatar dressed in clothes according to the clothing preference, which is an avatar according to the gender, etc., indicated by the voice preference. Other individual information is reflected in the text when avatar voice data is generated.

[0126] In this embodiment, important dialogue history is not stored or deleted in the autonomous mode, but this is not necessary. Important dialogues can also be extracted from the robot voice data and stored as important dialogue history. As an example, the method disclosed in Japanese Patent Application Laid-Open No. 2020-071675 can be used as a method for extracting important dialogue history from dialogue. When using a known technique for extracting important dialogue history from dialogue, the necessary circuit components and data are appropriately provided in the server 12.

[0127] Furthermore, when the server 12 assigns an operator who remotely operates the robot 18 that provides a predetermined service to the robot 18, the server 12 sets the remote mode. However, the operator is an operator who logs in to the server 12 using his / her own operator terminal and selects to use the predetermined service.

[0128] As described above, in the remote mode, the server 12 authenticates (or identifies) the dialogue agent from the face image included in the robot image data, acquires customer information about the authenticated dialogue agent from the customer information DB 12b, and transmits the acquired customer information to the operator terminal 16. The operator terminal 16 displays a customer information screen 300 using the customer information. At this time, the individual information included in the customer information is displayed in order of importance. The operator controls the robot 18 by referring to the customer information screen 300. The server 12 controls the movement of the robot 18 according to the movement command transmitted from the operator terminal 16 of the operator assigned to remotely operate the robot 18, controls the speech of the robot 18 according to the operator speech data transmitted from the operator terminal 16, and controls the display of the avatar image 28 according to the operator image data transmitted from the operator terminal 16.

[0129] As an example, in the remote mode, the avatar can be controlled using individual information of high importance among the customer information. It is up to the operator to decide whether to control the avatar according to the customer information. However, in this embodiment, since customer information of high importance is preferentially displayed on the customer information screen 300, it is considered that if the avatar is controlled according to the customer information, it will be possible to smoothly respond to the dialogue agent.

[0130] In the remote mode, the server 12 controls the robot 18 according to the operation of the operator. At this time, when the operator terminal 16 instructs that an acoustic model according to the voice preference included in the preference information of the dialogue agent is used, that text about the speech content according to the dialect preference is used, and that an avatar dressed in accordance with the clothing preference is displayed, the following are set: that an acoustic model according to the voice preference is used, that text about the speech content according to the dialect preference is used, and that an avatar dressed in accordance with the clothing preference is displayed. However, it is not necessary to set all of these at the same time. It is sufficient to set one or more of them. However, in this embodiment, since the avatar image 28 is generated by the robot 18, the robot 18 is instructed to display an avatar with an avatar of gender, etc., according to the voice preference and dressed in accordance with the clothing preference. Other customer information is reflected in the text when generating avatar voice data.

[0131] Although detailed explanation and illustrations are omitted, when the customer information screen 300 is displayed on the operator terminal 16, the operator can select an option to individually instruct (or select) that an acoustic model according to the voice preference included in the preference information of the dialogue agent is used, that text about the speech content according to the dialect preference is used, and that an avatar dressed in clothes according to the clothing preference is displayed. The operator can also cancel the instructed (or selected) contents.

[0132] As described above, the voice preference is a male voice, a female voice, or a robotic voice, and when the server 12 causes the robot 18 to speak in a voice according to the voice preference, the server 12 generates a synthetic voice using an acoustic model for the voice according to the preference. As an example, this synthetic voice is generated using a trained inference device disclosed in JP 2021-157145 A. In this trained inference device, a synthetic voice is generated from an input text. Therefore, in the remote mode, when the operator speech data is operator voice data, a text that is a result of voice recognition of the operator voice corresponding to the operator voice data is input to the inference device.

[0133] The inference device can be trained using the method disclosed in JP 2021-157145 A. In addition, the inference device can be used to recognize the operator's voice.

[0134] Furthermore, avatar image 28 is lip-synchronized when outputting a sound based on avatar voice data transmitted from server 12.

[0135] In this embodiment, as described above, the importance of the individual information included in each customer information is updated. As an example, the individual information is updated according to the reaction of the dialogue agent when the robot 18 speaks in the autonomous mode or the remote mode. In this embodiment, the server 12 estimates (or detects) the emotion of the dialogue agent from the face image of the dialogue agent included in the robot image data received after the robot 18 speaks. When the emotion of the dialogue agent is positive, the importance of each of the one or more pieces of individual information included in the latest dialogue and the one or more pieces of individual information included in the individual information set in the avatar is increased by a predetermined value. On the other hand, when the emotion of the dialogue agent is negative, the importance of each of the one or more pieces of individual information included in the latest dialogue and the one or more pieces of individual information included in the individual information set in the avatar is decreased by a predetermined value, or is not updated.

[0136] However, in this embodiment, the latest dialogue refers to the latest dialogue history stored in the history DB 12a, and the latest dialogue history is the voice data of the dialogue agent and the voice data of the operator in response thereto, or only the voice data of the operator.

[0137] The importance level is set as a number between 1 and 10, and the predetermined value for increasing or decreasing is 0.2. The importance level is set to an initial value (for example, 1) when the individual information is registered in the customer information. Furthermore, the importance level cannot be increased or decreased beyond the maximum value (10) or the minimum value (1).

[0138] The estimated emotions are any one of joy, fear, sadness, disgust, and anger, of which joy is a positive emotion and the remaining emotions, fear, sadness, disgust, and anger, are negative emotions.

[0139] In addition, a known technique can be used as a method for estimating emotions from the face image of the operator. For example, the techniques disclosed in "Hiroshi Kobayashi, Fumio Hara: Recognition of basic human facial expressions using neural networks, Transactions of the Society of Instrument and Control Engineers, Vol. 29, No. 1, 112 / 118 (1993)", "Yosuke Koyanaka, Tsuneyasu Homma, Masao Sakai, Kenichi Abe: Facial expression recognition using neural networks, Bulletin of the School of Medicine and Health Sciences, Tohoku University, 13(1):23-32, 2004" and "Taiki Nishime, Satoshi Endo, Aiko Touma, Koji Yamada, Yuhei Akamine: Acquisition of facial expressions and analysis of facial features using convolutional neural networks, Transactions of the Japanese Society for Artificial Intelligence, Vol. 32, No. 5, FZ (2017)" can be used.

[0140] In addition, in another known technology, the technology disclosed in JP 2020-163660 A can also be used as a method for estimating human emotions based on feature points extracted from a face image.

[0141] It is also possible to estimate human emotions from voice. As a method for estimating human emotions from voice, known technologies can be used. For example, the technologies disclosed in JP 2021-12285 A and "Daiki Mori: Understanding Emotions and Attitudes from Voice, Journal of the Institute of Electronics, Information and Communication Engineers Vol. 101, No. 9, 2018" can be used.

[0142] However, when using a known technique for estimating emotions based on a facial image or voice, the necessary circuit components and data for that purpose are appropriately provided in the server 12. However, it is also possible to provide a device that estimates emotions based on a facial image or voice (hereinafter referred to as an "estimation device") on the cloud, transmit the facial image or voice to the estimation device, and receive the emotion estimation result from the estimation device.

[0143] In this embodiment, the emotion of the dialogue agent is estimated as the reaction of the dialogue agent, but this is not limited to this. The length of time that the dialogue agent directs its gaze at the robot 18 and the length of time that it looks away may be detected as the reaction of the dialogue agent. In this case, the longer the gaze time is, the higher the importance level can be, and the longer the gaze time is, the lower the importance level can be. However, even if the gaze time is long, the importance level does not have to be updated.

[0144] Fig. 8 shows an example of a memory map 400 of the RAM 52 built into the server 12. As shown in Fig. 8, the RAM 52 includes a program storage area 402 and a data storage area 404. The program storage area 402 stores an information processing program executed by the server 12 of this embodiment.

[0145] The information processing programs include a communication program 402a, an authentication program 402b, a customer information transmission program 402c, a history storage program 402d, an operation mode setting program 402e, an emotion estimation program 402f, an individual information registration program 402g, and an importance update program 402h.

[0146] The communication program 402a is a program for communicating (sending and receiving data) with external devices, in this embodiment, the operator terminal 16 and the robot 18 via the network 14.

[0147] The authentication program 402b is a program for authenticating whether or not the customer information of the dialogue agent is registered in the customer information DB 12b. In this embodiment, the authentication program 402b judges whether or not customer information having a face image that matches the face image of the robot image data is registered in the customer information DB 12b, and if it is registered, it determines that the authentication is successful. However, if the customer information of the dialogue agent is not registered in the customer information DB 12b, the authentication program 402b determines that the authentication is unsuccessful, issues a customer ID, generates customer information data 404f including face image data associated with the issued customer ID, and registers the generated customer information data 404f in the customer information DB 12b as a new customer. In addition, the authentication program 402b transmits the generated customer information data 404f to the operator terminal 16.

[0148] The customer information transmission program 402c is a program for transmitting, in the remote mode, customer information on the interactive agent authenticated by the authentication program 402b or customer information on a new customer to the operator terminal 16.

[0149] The history storage program 402d is a program for storing a response history for each operator. In this embodiment, as the response history, operation commands, operator utterance data, robot image data, and robot voice data when responding to a dialogue agent are stored in chronological order.

[0150] Although a detailed description is omitted, an administrator of the information processing system 10 or the server 12 can also generate and update customer information based on the interaction history.

[0151] The operation mode setting program 402e is a program for setting and managing the operation mode of the robot 18. In this embodiment, when there is no operator to remotely operate the robot, the autonomous mode is set as the operation mode, and when an operator to remotely operate the robot is assigned, the remote mode is set. However, when the operator terminal 16 logs out in the remote mode, that is, when none of the operator terminals 16 is logged in, there is no operator to remotely operate the robot, and therefore the autonomous mode is set as the operation mode.

[0152] The emotion estimation program 402f is a program for estimating the emotion of the dialogue agent from the face image included in the robot image data 404a. The individual information registration program 402g is a program for extracting individual information from the utterance contents of the dialogue agent included in the robot voice data 404b, and registering the individual information when the extracted individual information is not included in the customer information of the dialogue agent. When the individual information is registered, the importance of the individual information is set to 1.

[0153] The importance update program 402h is a program for updating the importance of the individual information contained in the latest dialogue and the individual information set for the avatar (e.g., voice preference, dialect preference, clothing preference) according to the dialogue agent's reaction (in this embodiment, emotion) when the avatar speaks in autonomous mode or remote mode.

[0154] When the customer information data 404f or the avatar voice data 404h is transmitted, the communication program 402a is also executed.

[0155] Although not shown, the program storage area 402 also stores other programs necessary for providing predetermined services in addition to the operating system and middleware of the server 12.

[0156] In addition, the data memory area 404 stores robot image data 404a, robot voice data 404b, operator speech data 404c, operation command data 404d, operation mode data 404e, customer information data 404f, avatar setting data 404g, avatar voice data 404h, and important dialogue data 404i.

[0157] The robot image data 404a is data of a camera image captured by the camera 124 provided on the robot 18 and received from the robot 18. The robot image data 404a is erased from the RAM 52 after being transmitted to the operator terminal 16.

[0158] The robot voice data 404b is data of voice detected by the microphone 110 provided on the robot 18 and received from the robot 18. This voice includes the voice of the dialogue agent. The robot voice data 404b is erased from the RAM 52 after being transmitted to the operator terminal 16.

[0159] The operator speech data 404c is voice data detected by a microphone 82 provided on the operator terminal 16 used by an operator remotely operating the robot 18 and received from the operator terminal 16, or text data instructed to be spoken using the control screen 200 displayed on the display device 80 of the operator terminal 16. However, the voice data is basically voice data of the operator remotely operating the robot 18, i.e., operator voice data.

[0160] In the remote mode, the action command data 404d is data of an action command received from the operator terminal 16 and input by an operator remotely operating the robot 18. In the autonomous mode, the action command data 404d is data of an action command for an action included in a communication action determined by the server 12 or an action command for an unconscious action. The operation mode data 404e is data for identifying the operation mode of the robot 18, and in this embodiment, is data indicating whether the operation mode is the autonomous mode or the remote mode.

[0161] The customer information data 404f is data on the customer information of the dialogue agent, acquired from the customer information DB 12b.

[0162] The avatar setting data 404g is data on individual information set for the avatar by the server 12 itself or according to an instruction from an operator. In this embodiment, this includes voice preference, dialect preference, and clothing preference. If the avatar setting data 404g includes data on voice preference and / or clothing preference, the robot 18 is instructed to display an avatar with gender, etc., according to the voice preference and / or to display an avatar with clothing according to the clothing preference (i.e., the type of avatar).

[0163] The avatar voice data 404h is data of a synthetic voice to be uttered by the avatar image 28. However, in the autonomous mode, the synthetic voice is generated using an acoustic model set as a default, and in the remote mode, the synthetic voice is generated using an acoustic model set according to the voice of the avatar specified by the operator remotely operating the robot 18.

[0164] The important dialogue data 404i is text data on important dialogues for the dialogue agent, which are designated to be registered by the operator.

[0165] Although not shown in the figure, the data memory area 404 stores other data necessary for executing information processing, such as dictionary data for classifying individual information, and is provided with timers (counters) and flags necessary for executing information processing.

[0166] Fig. 9 shows an example of a memory map 500 of the RAM 72 built into the operator terminal 16. As shown in Fig. 9, the RAM 72 includes a program storage area 502 and a data storage area 504. The program storage area 502 stores an information processing program executed by the operator terminal 16 of this embodiment.

[0167] The information processing programs include a communication program 502a, an operation detection program 502b, an image display program 502c, a sound detection program 502d, and a sound output program 502e.

[0168] The communication program 502a is a program for communicating (sending and receiving data) with external devices, in this embodiment, the server 12 and the robot 18, either wired or wirelessly.

[0169] The operation detection program 502b is a program for detecting operation data input from the input device 78 in accordance with an operator's operation and storing the detected operation data in the data storage area 504.

[0170] However, when buttons 210-238, 252, and 260 provided on control screen 200 are operated, data of operation information (including operation commands) assigned to the operated buttons 210-238, 252, and 260 is detected as operation data.

[0171] Similarly, when button 312 or button 314 provided on customer information screen 300 is operated, data of the operation information assigned to the operated button 312 or 314 is detected as operation data.

[0172] In addition, data of operation information on the customer information screen 300, such as selecting an option or individually instructing (or selecting) that an acoustic model in accordance with the voice preference included in the dialogue agent's preference information be used, that text about the speech content in accordance with the dialect preference be used, and that an avatar dressed in clothing in accordance with the clothing preference be displayed, is detected as operation data.

[0173] The image display program 502c is a program for generating display image data for various screens (in this embodiment, the control screen 200 and the customer information screen 300) and outputting the generated display image data to the display device 80. However, the display image data for the control screen 200 also includes robot image data 504d received from the server 12. In addition, the display image data for the customer information screen 300 also includes customer information data 504c received from the server 12.

[0174] The sound detection program 502d is a program for detecting the voice of the operator inputted from the microphone 82 and storing in the data storage area 504 operator utterance data 504f corresponding to the detected voice.

[0175] The sound output program 502e is a program for outputting the robot voice data 504e received from the server 12 to the speaker 84. The sound output program 502e is also a program for transmitting (or outputting) the operator utterance data 504f to the server 12. At this time, the communication program 502a is also executed. However, when the button 252 is turned on, the sound output program 502e also transmits (or outputs) the operator utterance data 504f corresponding to the text displayed in the display area 250 to the server 12.

[0176] Although not shown, the program storage area 502 also stores programs for executing the operating system of the operator terminal 16, any middleware, and any applications.

[0177] The data storage area 504 also stores operation data 504a, image generation data 504b, customer information data 504c, robot image data 504d, robot voice data 504e, operator speech data 504f, action command data 504g, and important dialogue data 504h.

[0178] The operation data 504a is operation data detected according to the operation detection program 502b. The operation data 504a is deleted after being transmitted to the server 12 or used for processing by the operator terminal 16.

[0179] Image generation data 504b is image data for generating and displaying screens to be displayed on display device 80 (in this embodiment, control screen 200 shown in FIG. 5 and customer information screen 300 shown in FIG. 6).

[0180] The customer information data 504c is data received from the server 12 about customer information regarding the currently serving dialogue agent.

[0181] The robot image data 504d is robot image data received from the server 12. The robot image data 504d is deleted after being used in the processing of the CPU 70.

[0182] The robot voice data 504e is robot voice data received from the server 12. The robot voice data 504e is deleted after being used in the processing of the CPU 70.

[0183] The operator speech data 504f is voice data detected by the microphone 82 or data of an input speech command, and is deleted after being transmitted to the server 12.

[0184] The operation command data 504g is data of the input operation command, and is deleted after being transmitted to the server 12.

[0185] The important dialogue data 504h is text about an important dialogue that has been entered, and is added to or deleted from the customer information data 504c, and is deleted after being transmitted to the server 12.

[0186] Although not shown, the data storage area 504 stores other data necessary for executing information processing, and is provided with a timer (counter) and a flag necessary for executing information processing.

[0187] Fig. 10 shows an example of a memory map 600 of the RAM 102 built into the robot 18. As shown in Fig. 10, the RAM 102 includes a program storage area 602 and a data storage area 604. The program storage area 602 stores a control program executed by the robot 18 of this embodiment.

[0188] The control programs include a communications program 602a, a movement control program 602b, a photographed image acquisition program 602c, an avatar image generation program 602d, an image display program 602e, a sound detection program 602f, and a sound output program 602g.

[0189] The communication program 602a is a program for wirelessly communicating (transmitting and receiving data) with external devices, in this embodiment, the server 12 and the operator terminal 16.

[0190] The movement control program 602b is a program for moving and stopping within the environment in accordance with the action command data 604e for movement received from the server 12.

[0191] The photographed image acquisition program 602c is a program for acquiring photographed image data captured by the camera 122.

[0192] Avatar image generation program 602d is a program for generating avatar image data 604f in accordance with avatar type and action command data 604e received from server 12, using image generation data 604a.

[0193] Image display program 602e is a program for outputting on display device 30 avatar image 28 corresponding to avatar image data 604f generated in accordance with avatar image generation program 602d.

[0194] The sound detection program 602f is a program for detecting the voice of the dialogue agent or the like inputted from the microphone 110, and storing in the data storage area 604 the robot voice data 604c corresponding to the detected voice.

[0195] The sound output program 602g is a program for outputting the avatar voice data 604d received from the server 12 to the speaker 112. The sound output program 602g is also a program for transmitting (or outputting) the robot voice data 604c detected by the sound detection program 602f to the server 12. At this time, the communication program 602a is also executed.

[0196] Although not shown, the program storage area 602 stores an operating system for the robot 18 and any middleware, as well as a program for detecting distance and a program for detecting inertial information.

[0197] Furthermore, data storage area 604 stores image generation data 604a, robot image data 604b, robot voice data 604c, avatar voice data 604d, action command data 604e, avatar image data 604f, and the like.

[0198] Image generation data 604a is data such as polygon data and texture data for generating avatar images 28 for a default avatar, a male avatar, and a female avatar.

[0199] The robot image data 604b is data of a camera image captured by the camera 122 equipped to the robot 18. The robot image data 604b is deleted after being transmitted to the server 12.

[0200] The robot voice data 604c is data of the voice of the dialogue agent or the like detected by the microphone 110 provided in the robot 18. The robot voice data 604c is deleted after being transmitted to the server 12.

[0201] Avatar voice data 604d is avatar voice data received from server 12. Avatar voice data 604d is used for processing by CPU 100 and then deleted.

[0202] The action command data 604e is data of an action command received from the server 12. The movement of the robot 18 and the action of the avatar image 28 are controlled in accordance with this action command. The action command data 604e is deleted after being used in processing by the CPU 100.

[0203] Avatar image data 604f is image data of avatar image 28 generated in accordance with avatar image generation program 602d.

[0204] Although not shown, the data storage area 604 stores other data required for executing the control processing, and is provided with a timer (counter) and a flag required for executing the control processing.

[0205] 11 to 17 are flow charts showing an example of information processing when a predetermined service is executed by the CPU 50 of the server 12 shown in Fig. 2. Although not shown, the CPU 50 of the server 12 executes login and logout processing of the operator terminal 16 or the operator in parallel with the information processing in response to a request from the operator terminal 16.

[0206] 11, when the CPU 50 starts information processing, in step S1, the CPU 50 determines whether or not the operation mode of the robot 18 is the autonomous mode. Here, the CPU 50 refers to the operation mode data 404e to determine whether or not the autonomous mode is set as the operation mode.

[0207] The CPU 50 sets the autonomous mode when there is no operator operating the robot 18, that is, when none of the operator terminals 16 are logged in. When the autonomous mode is set, the CPU 50 instructs the robot 18 to display the default avatar image 28.

[0208] However, when an operator who remotely operates the robot 18 has already been assigned when starting information processing, the CPU 50 sets the remote mode as the operation mode.

[0209] If the result in step S1 is "NO", that is, if the operation mode is not the autonomous mode, the process proceeds to step S51 shown in Fig. 14. On the other hand, if the result in step S1 is "YES", that is, if the operation mode is the autonomous mode, the process proceeds to step S3, where it is determined whether the robot 18 is on standby. Here, the CPU 50 determines whether the robot 18 is not responding to the dialogue agent. As an example, the CPU 50 determines whether a state in which no voice data is received from the robot 18 has continued for a certain period of time or more.

[0210] If the answer is "YES" in step S3, that is, if the robot 18 is on standby, then in step S5 it is determined whether or not to execute an unconscious action. Here, the CPU 50 determines whether or not it is time for the avatar (avatar image 28) to change the direction of its line of sight or turn its face to the left or right in accordance with the program that causes the robot 18 to act autonomously.

[0211] If "NO" in step S5, that is, if the unconscious movement is not to be executed, the process proceeds to step S117 shown in Fig. 17. On the other hand, if "YES" in step S5, that is, if the unconscious movement is to be executed, the process transmits movement command data for the unconscious movement to the robot 18 in step S7, and proceeds to step S117.

[0212] As described above, if the result is "NO" in step S3, then in step S9 shown in Fig. 12 it is determined whether or not the robot image data has been received. If the result is "NO" in step S9, that is, if the robot image data has not been received, the process proceeds to step S27 shown in Fig. 13. On the other hand, if the result is "YES" in step S9, that is, if the robot image data has been received, then in step S11 the robot image data 404a is stored in the RAM 52, and further, in step S13 the robot image data 404a is stored in the history DB 12a.

[0213] In the next step S15, the dialogue agent is recognized. Here, the CPU 50 recognizes a human face image in the robot image data 404a, and searches the customer information DB 12b to see whether customer information including a face image that matches the recognized face image is registered.

[0214] Next, in step S17, it is determined whether or not the customer information of the dialogue agent has been registered. If it is "NO" in step S17, that is, if the customer information of the dialogue agent has not been registered, in step S19, the dialogue agent is registered in the customer information DB 12b as a new customer, and the process proceeds to step S27. In this step S19, the CPU 50 issues a customer ID, and registers the issued customer ID in the customer information DB 12b in association with the customer information including the face image recognized in step S15. The same applies to step S71 described later.

[0215] On the other hand, if the answer is "YES" in step S17, and if the customer information of the dialogue agent is registered, then in step S21 it is determined whether the dialogue agent has been changed. That is, the CPU 50 determines whether the corresponding dialogue agent has been changed. However, when the information processing is started, if the robot 18 is corresponding to the first dialogue agent, the CPU 50 determines that the dialogue agent has not been changed.

[0216] If "YES" in step S21, that is, if the dialogue agent is to be changed, proceed to step S25. On the other hand, if "NO" in step S21, that is, if the dialogue agent is not to be changed, in step S23, it is determined whether or not customer information has been acquired.

[0217] If "YES" in step S23, that is, if the customer information has been acquired, the process proceeds to step S27. On the other hand, if "NO" in step S23, that is, if the customer information has not been acquired, in step S25, the customer information of the dialogue agent recognized in step S15 is acquired from the customer information DB 12b, and the customer information data 404f is stored in the RAM 52, and the process proceeds to step S27.

[0218] As shown in Fig. 13, in step S27, it is determined whether or not robot voice data has been received. If "NO" in step S27, that is, if robot voice data has not been received, the process proceeds to step S33. Although not shown, if robot voice data has not been received, the time is measured. On the other hand, if "YES" in step S27, that is, if robot voice data has been received, in step S39, the robot voice data 404b is stored in RAM 52, and the process proceeds to step S33.

[0219] In step S33, the emotion of the dialogue agent is estimated. The method for estimating the emotion of the dialogue agent is as described above. In the next step S35, the individual information included in the latest dialogue and the individual information set for the avatar are extracted (or acquired).

[0220] In the next step S37, it is determined whether the extracted individual information is registered in the customer information of the dialogue agent. If the answer is "NO" in step S37, that is, if the extracted individual information is not registered in the customer information of the dialogue agent, in step S39, the extracted individual information and its importance (here, the initial value=1) are registered in the customer information of the dialogue agent, and the process proceeds to step S43.

[0221] On the other hand, if step S37 is "YES", that is, if the extracted individual information is registered in the customer information of the dialogue agent, the importance of the extracted individual information is updated in step S41, and the process proceeds to step S43. The method of updating the importance of individual information is as described above. Also, if multiple pieces of individual information are extracted in step S35, the processes of steps S37-S41 are performed for each of the multiple pieces of individual information.

[0222] In step S43, a communication behavior of the robot 18 is determined. However, the communication behavior is determined according to the method disclosed in the above-mentioned Japanese Patent Application Laid-Open No. 2009-131914.

[0223] In the next step S45, action command data for the gesture, i.e., the action, included in the communication action determined in step S43 is transmitted to the robot 18. However, if no action is included in the communication action, the action command data is not transmitted.

[0224] In the following step S47, avatar voice data corresponding to the speech content included in the communication behavior determined in step S43 is transmitted to the robot 18. However, if the communication behavior does not include speech, the avatar voice data is not transmitted.

[0225] Then, in step S49, the action command data and the avatar voice data are stored in the history DB 12a, and the process proceeds to step S117. However, if the communication action does not include an action, the action command data is not stored in step S49. Also, if the communication action does not include a speech, the avatar voice data is not stored in step S49.

[0226] As described above, if the result of step S1 is "NO", that is, in the remote mode, in step S51 shown in Fig. 14, it is determined whether or not operator utterance data has been received from the operator terminal 16. If the result of step S51 is "NO", that is, if the operator utterance data has not been received, the process proceeds to step S59. On the other hand, if the result of step S51 is "YES", that is, if the operator utterance data has been received, in step S53, the operator utterance data 404c is stored in the RAM 52, in step S55, the avatar voice data 404h based on the operator utterance data 404c is transmitted to the robot 18, and in step S57, the operator utterance data 404c is stored in the history DB 12a, and the process proceeds to step S59.

[0227] In step S59, it is determined whether or not robot image data has been received from the robot 18. If step S59 is "NO", that is, if robot image data has not been received, the process proceeds to step S67 shown in Fig. 15. On the other hand, if step S59 is "YES", that is, if robot image data has been received, in step S61, the robot image data 404a is stored in the data storage area 404 of the RAM 52, in step S63, the robot image data 404a is transmitted to the operator terminal 16, in step S65, the robot image data 404a is stored in the history DB 12a, and the process proceeds to step S67.

[0228] 15, the dialogue agent is recognized, and in step S69, it is determined whether the customer information of the dialogue agent is registered. If the answer is "NO" in step S69, that is, if the customer information of the dialogue agent is not registered, in step S71, the dialogue agent is registered in the customer information DB 12b as a new customer, and the process proceeds to step S79.

[0229] On the other hand, if "YES" in the step S69, and if the customer information of the dialogue agent is registered, it is determined in a step S73 whether the dialogue agent is to be changed.

[0230] If "YES" in step S73, that is, if the dialogue agent is to be changed, the process proceeds to step S77. On the other hand, if "NO" in step S73, that is, if the dialogue agent is not to be changed, in step S75, it is determined whether or not the customer information has been transmitted.

[0231] If "YES" in step S75, that is, if the customer information has already been transmitted, the process proceeds to step S81 shown in Fig. 16. On the other hand, if "NO" in step S75, that is, if the customer information has not already been transmitted, the customer information of the dialogue agent recognized in step S67 is obtained from the customer information DB 12b in step S77, and the customer information of the dialogue agent is transmitted to the operator terminal 16 in step S79, and the process proceeds to step S81.

[0232] 16, in step S81, it is determined whether or not robot voice data has been received from the robot 18. If "NO" in step S81, that is, if robot voice data has not been received, the process proceeds to step S89. On the other hand, if "YES" in step S81, that is, if robot voice data has been received, the process stores the robot voice data 404b in the data storage area 404 in step S83, transmits the robot voice data 404b to the operator terminal 16 in step S85, stores the robot voice data 404b in the history DB 12a in step S87, and proceeds to step S89.

[0233] In step S89, the emotion of the dialogue agent is estimated. The method for estimating the emotion of the dialogue agent is as described above. In the next step S91, individual information is extracted (or acquired) from the latest dialogue and the response information set for the avatar.

[0234] In the next step S93, it is determined whether the extracted individual information is registered in the customer information of the dialogue agent. If the answer is "NO" in step S93, that is, if the extracted individual information is not registered in the customer information of the dialogue agent, in step S95, the extracted individual information and its importance (initial value=1 in this case) are registered in the customer information of the dialogue agent, and the process proceeds to step S99. In other words, the customer information of the dialogue agent of the new customer is registered.

[0235] On the other hand, if step S93 is "YES", that is, if the extracted individual information is registered in the customer information of the dialogue agent, the importance of the extracted individual information is updated in step S97, and the process proceeds to step S99. That is, by updating the importance of the individual information, the customer information including the individual information is updated. However, the method of updating the importance of the individual information is as described above. Furthermore, if multiple pieces of individual information are extracted in step S91, the processes of steps S93-S97 are performed for each of the multiple pieces of individual information.

[0236] In step S99, the updated or registered customer information is transmitted to the operator terminal 16, and the process proceeds to step S101 shown in FIG. 17. Accordingly, in the operator terminal 16, the customer information screen 300 using the registered customer information is displayed, or the customer information screen 300 using the updated customer information is displayed (updated). Therefore, when almost no individual information is displayed on the customer information screen 300, the operator can know that he is dealing with a new customer or a dialogue agent with whom he has hardly had a dialogue. In addition, by updating the individual information during a dialogue, the operator can always refer to new customer information when dealing with the dialogue agent. In other words, the operator can deal with the dialogue agent appropriately.

[0237] 17, in step S101, it is determined whether or not operation command data has been received from the operator terminal 16. If "NO" in step S101, that is, if operation command data has not been received, the process proceeds to step S117. On the other hand, if "YES" in step S101, that is, if operation command data has been received, the operation command data 404d is stored in the data storage area 404 of the RAM 52 in step S103, the operation command data 404d is transmitted to the robot 18 in step S105, and the operation command data 404d is stored in the history DB 12a in step S107.

[0238] In the next step S109, it is determined whether or not there is an instruction to register an important dialogue from the operator terminal 16. If the answer is "YES" in step S109, that is, if there is an instruction to register an important dialogue, then in step S111, the important dialogue is registered in the customer information data 404f and the customer information data of the dialogue agent stored in the customer information DB 12b, and the process proceeds to step S117. On the other hand, if the answer is "NO" in step S109, that is, if there is no instruction to register an important dialogue, then in step S113, it is determined whether or not there is an instruction to delete an important dialogue.

[0239] If the answer is "YES" in step S113, that is, if there is an instruction to delete the important dialogue, then in step S115 the important dialogue instructed to be deleted is deleted from the customer information data 404f and the customer information data of the dialogue agent stored in the customer information DB 12b, and the process proceeds to step S117. On the other hand, if the answer is "NO" in step S113, that is, if there is no instruction to delete the important dialogue, then in step S117 it is determined whether the service has ended. Here, the CPU 50 determines whether the end time of the service has arrived or whether the provision of the service has been completely terminated.

[0240] If "NO" in step S117, that is, if the service is not to be ended, the process returns to step S1. On the other hand, if "YES" in step S117, that is, if the service is to be ended, in step S119, a service end notification is sent to the operator terminal 16 and the robot 18, and the information processing is ended.

[0241] Fig. 18 to Fig. 20 are flow charts showing an example of information processing of the CPU 70 of the operator terminal 16 shown in Fig. 3. Although not shown in the figures, the CPU 70 of the operator terminal 16 detects operation data 504a and acquires operator utterance data 504f, and executes a process of transmitting each data to the server 12 (i.e., a transmission process), and executes a process of receiving the robot image data 404a and the robot voice data 404b transmitted from the server 12 (i.e., a reception process).

[0242] As shown in Fig. 18, when the CPU 70 starts information processing, in step S201, it establishes a connection with the server 12. In the next step S203, the control screen 200 as shown in Fig. 5 is displayed (or updated) on the display device 80. However, at the beginning of the information processing, since no robot image data has been received, no image is displayed in the display area 202. Also, at this time, the remote mode is set as the operation mode.

[0243] In the next step S205, it is determined whether or not customer information has been received from the server 12. If "NO" in step S205, that is, if customer information has not been received from the server 12, the process proceeds to step S209. On the other hand, if "YES" in step S205, that is, if customer information has been received from the server 12, in step S207, a customer information screen 300 as shown in Fig. 6 is displayed (or updated) on the display device 80, and the process proceeds to step S209. As described above, the customer information screen 300 is displayed side by side on the control screen 200.

[0244] In step S209, it is determined whether or not an operation command has been inputted. Here, the CPU 70 determines whether or not any of the buttons 210-224, 220-228, . . . has been operated.

[0245] If "NO" in step S209, that is, if no operation command has been input, proceed to step S213. On the other hand, if "YES" in step S209, that is, if a video command has been input, in step S211, the operation command data 504g is transmitted to the server 12, and proceed to step S213.

[0246] In step S213, it is determined whether or not there is an utterance. Here, the CPU 70 determines whether or not the operator's voice or a speech command has been input.

[0247] If "YES" in step S213, that is, if there is an utterance, in step S215, the operator utterance data is transmitted to the server 12, and the process proceeds to step S217 shown in Fig. 19. On the other hand, if "NO" in step S213, that is, if there is no utterance, the process proceeds to step S217.

[0248] 19, in step S217, it is determined whether or not dialogue data has been received from the server 12. If "NO" in step S217, that is, if dialogue data has not been received from the server, the process proceeds to step S227. On the other hand, if "YES" in step S217, that is, if dialogue data has been received from the server 12, it is determined in step S219 whether or not robot voice data is present.

[0249] If the result in step S219 is "YES", that is, if the robot voice data 504e is present, the robot voice data 504e is output in step S221, the robot image data 504d is output in step S223, and the process proceeds to step S227. Therefore, the voice of the interactive agent that converses with the operator through the robot 18 is output from the speaker 84 of the operator terminal 16, and a camera image including the interactive agent that converses with the operator through the robot 18 is displayed in the display area 202 of the control screen 200.

[0250] On the other hand, if "NO" in the step S219, that is, if the robot voice data 504e does not exist, in a step S225, the robot image data 504d is output, and the process proceeds to a step S227.

[0251] In step S227, it is determined whether or not the text of the utterance content is input. If "NO" in step S227, that is, if the text of the utterance content is not input, the process proceeds to step S231. On the other hand, if "YES" in step S227, that is, if the text of the utterance content is input, the input text is displayed in the display area 250 in step S229, and the process proceeds to step S231.

[0252] In step S231, it is determined whether or not an important dialogue is to be registered. Here, CPU 70 determines whether or not button 312 is turned on in a state in which text is input (displayed) in input field 310.

[0253] If "NO" in step S231, that is, if it is not an important dialogue to be registered, the process proceeds to step S235 shown in Fig. 20. On the other hand, if "YES" in step S231, that is, if it is an important dialogue to be registered, in step S233, an instruction to register the important dialogue is sent to the server 12, and the process proceeds to step S235. However, in step S233, the registration instruction and the text data of the important dialogue are sent to the server 12. Although not shown in the figure, at this time, the text data of the important dialogue is classified as an important dialogue and added to the customer information data 504c. Therefore, when the customer information screen 300 is subsequently displayed (S207), the added important dialogue is displayed.

[0254] 20, in step S235, it is determined whether or not an important dialogue is to be deleted. Here, CPU 70 determines whether or not button 314 is turned on in a state where text is input (displayed) in input field 310.

[0255] If "NO" in step S235, that is, if the important dialogue is not to be deleted, proceed to step S239. On the other hand, if "YES" in step S235, that is, if the important dialogue is to be deleted, in step S237, an instruction to delete the important dialogue is sent to the server 12, and proceed to step S239. However, in step S237, the deletion instruction and the text data of the important dialogue are sent to the server 12. Although not shown in the figure, at this time, the text data of the important dialogue is deleted from the customer information data 504c. Therefore, when the customer information screen 300 is displayed thereafter (S207), the deleted important dialogue is not displayed.

[0256] In step S239, it is determined whether or not the text input is for an important dialogue. If "NO" in step S239, that is, if the text input is not for an important dialogue, the process proceeds to step S243. On the other hand, if "YES" in step S239, that is, if the text input is for an important dialogue, the input text is displayed in input field 310 in step S241, and the process proceeds to step S243.

[0257] In step S243, it is determined whether or not the service has ended. Here, CPU 70 determines whether or not a service end notification has been received from server 12. If "YES" in step S243, that is, if the service has ended, the information processing is ended.

[0258] On the other hand, if "NO" in step S243, that is, if the service is not to be ended, then in step S245 it is determined whether or not to log out. Here, the CPU 70 determines whether or not the operator has instructed to log out. If "NO" in step S245, that is, if logging out is not to be performed, the process returns to step S203. On the other hand, if "YES" in step S245, that is, if logging out is to be performed, then in step S247 a logout instruction is sent to the server 12, and the information processing is terminated.

[0259] Figures 21 and 22 are flow charts showing an example of control processing of the CPU 100 of the robot 18 shown in Figure 4. Although not shown in the figures, the CPU 100 of the robot 18 detects robot image data 604b, detects robot voice data 604c, executes a transmission process for transmitting each data to the server 12, and executes a reception process for receiving avatar voice data 604d and action command data 604g transmitted from the server 12.

[0260] 21, when CPU 100 starts the control process, in step S301, a connection is established with server 12. In the next step S303, avatar image data is generated and output. That is, avatar image 28 is displayed on display device 30.

[0261] However, the avatar image 28 to be displayed is an image of a type (male, female, or a robot 18-like appearance) instructed by server 12. This also applies to steps S317 and S319 described below.

[0262] Although not shown, when CPU 100 receives an instruction from server 12 to change the type of avatar during execution of the control process, CPU 100 displays avatar image 28 of the instructed type.

[0263] Next, in step S305, it is determined whether the service has ended. Here, CPU 100 determines whether a service end notification has been received from server 12. If "YES" in step S305, that is, if the service has ended, the control process ends. On the other hand, if "NO" in step S305, that is, if the service has not ended, it is determined in step S307 whether dialogue data has been received from server 12.

[0264] If "NO" in step S307, that is, if dialogue data has not been received from the server 12, the process proceeds to step S323 shown in Fig. 22. On the other hand, if "YES" in step S307, that is, if dialogue data has been received from the server 12, it is determined in step S309 whether or not avatar voice data 604d is present.

[0265] If "NO" in step S309, that is, if avatar voice data 604d does not exist, the process proceeds to step S313 shown in Fig. 22. If "YES" in step S309, that is, if avatar voice data 604d exists, the process outputs avatar voice data 604d to speaker 112 in step S311, and the process proceeds to step S313.

[0266] 22, in step S313, it is determined whether or not there is action command data 604g. If "NO" in step S313, that is, if there is no action command data 604g, the process proceeds to step S317. On the other hand, if "YES" in step S313, that is, if there is action command data 604g, it is determined in step S315 whether or not the action command data 604g includes a movement instruction.

[0267] If step S315 is "NO", that is, if action command data 604g does not include an instruction to move, then in step S317, image generation data 604a is used to generate and output avatar image data 604h for performing an action in accordance with action command data 604g, and the process proceeds to step S323. On the other hand, if step S315 is "YES", that is, if action command data 604g includes movement, then in step S319, image generation data 604a is used to generate and output avatar image data 604h for performing an action in accordance with action command data 604g, and in step S321, movement processing is performed in accordance with action command data 604g, and the process proceeds to step S323.

[0268] In step S323, it is determined whether or not voice input has been made. If "YES" in step S323, that is, if voice input has been made, in step S325, the robot image data 604b and the robot voice data 604c are sent to the server 12, and the process returns to step S305. On the other hand, if "NO" in step S323, that is, if voice input has not been made, in step S327, the robot image data 604b is sent to the server 12, and the process returns to step S305.

[0269] According to this embodiment, the individual information included in the customer information is registered, and the importance of the individual information included in the dialogue and the individual information set in the avatar is updated based on the response of the dialogue agent, so that the use of the individual information makes it possible to respond in accordance with the preferences of the dialogue agent.

[0270] Furthermore, according to this embodiment, since the individual information includes important dialogue history, important dialogue can be appropriately handled even if a different operator handles the dialogue.

[0271] In this embodiment, a robot having an avatar image displayed on a display device is used, but the present invention is not limited to this.

[0272] The applicant may use Robovie (registered trademark) of Communication Lot, which is sold by the applicant. In such cases, the movements of the hands and head (neck) can be realized by the robot. However, the appearance (clothing and gender) cannot be changed.

[0273] It is also possible to use Erica, an android (registered trademark) sold by the applicant of the present application. In such a case, the hand movements, head (neck) movements, facial expressions, and mouth movements can be realized by the robot. However, the appearance (clothing and gender) cannot be changed.

[0274] Therefore, the robot described in the embodiment and other robots may be used depending on the type of service to be provided.

[0275] In this embodiment, image data and voice data are stored as the dialogue history, but it is also possible to store only the voice data. In such a case, the emotion of the dialogue agent is estimated based on the voice data.

[0276] In addition, in this embodiment, when the robot speaks, the importance of the individual information included in the latest dialogue is updated, but the importance of the individual information included in all dialogues may be updated after the dialogue agent has finished all responses.

[0277] Furthermore, in this embodiment, the operator terminal transmits the operator voice data and the action command data to the server, and the server generates the avatar voice data according to the voice preference, but this is not limited to this. The operator terminal or the robot may generate the avatar voice data according to the voice preference.

[0278] When avatar voice data according to the voice preference is generated at the operator terminal, the avatar voice data is converted into an acoustic model according to the voice preference at the operator terminal and transmitted to the robot 18.

[0279] In addition, when a robot generates avatar voice data according to a voice preference, the robot generates avatar voice data by converting the operator speech data received from the operator terminal into an acoustic model according to the voice preference.

[0280] In this embodiment, a predetermined CG character is displayed as the avatar image 28, but an image of the operator may be displayed instead. In such a case, an image of the operator captured by a camera provided in the operator terminal is transmitted to the robot via a server and output to a display device.

[0281] In addition, in this embodiment, the interactive service using a robot avatar has been described, but it is also applicable to an interactive service using a CG avatar. In such a case, instead of a robot, a display device for displaying the CG avatar is provided, and a computer further including a camera, a microphone, and a speaker is provided. As an example, a general-purpose PC, a smartphone, a tablet PC, or the like can be used as the computer. In this case, the movement and speech of the CG avatar are controlled by an operator.

[0282] Furthermore, if the same results can be obtained from the steps in the flow charts shown in the above embodiments, the order in which the steps are processed can be changed.

[0283] Furthermore, the various screens and specific numerical values ​​given in the above embodiment are merely examples and can be appropriately changed as necessary. [Explanation of symbols]

[0284] 10. Information processing systems 12 ...Server 14 … Network 16 … Operator terminal 18. Robot 50, 70, 100...CPU 52, 72, 102...RAM 54, 74, 104 ... Communication I / F 56, 76, 106... Input / Output I / F 58, 78, 108 ... Input device 30, 60, 80…display device 82, 110...Mike 84, 112 ... Speaker 86, 120 ... Sensor I / F 88, 122 ... Camera 124, 126...LRF 128 ... Inertial Measurement Unit

Claims

1. robot or computer graphic avatar, an operator terminal used by an operator who controls the movement and speech of the avatar; and An information processing system comprising an information processing device communicably connected to each of the avatar and the operator terminal, The information processing device includes: a dialogue partner information acquiring means for acquiring, from a database, dialogue partner information about a dialogue partner who faces the avatar and converses with the operator, the dialogue partner information including at least preference information about the avatar; a first transmission means for transmitting the dialogue partner information acquired by the dialogue partner information acquisition means to the operator terminal; A first receiving means for receiving at least one of a control signal from the operator terminal and an operator utterance content which is an utterance content of the operator; an avatar voice generating means for generating an avatar voice to be uttered by the avatar based on the operator utterance content received by the first receiving means; a second transmitting means for transmitting at least one of the control signal and the avatar voice to the avatar; a second receiving means for receiving at least one of an image and a voice of the conversation partner from the avatar; a third transmission means for transmitting at least one of the video and audio of the conversation partner received by the second reception means to the operator terminal; a reaction detection means for detecting a reaction of the dialogue partner to the avatar voice based on at least one of the video and audio of the dialogue partner received by the second receiving means; individual information extraction means for extracting individual information included in the dialogue partner information from a latest dialogue history; and an update means for updating importance of the individual information extracted by the individual information extraction means and the individual information set for the avatar based on a reaction of the conversation partner detected by the reaction detection means, The operator terminal includes: a third receiving means for receiving the dialogue partner information transmitted by the first transmitting means; a dialogue partner information display means for displaying the dialogue partner information received by the third receiving means; a fourth transmission means for detecting the control signal for controlling the movement of the avatar input by the operator and transmitting the control signal to the information processing device; a fifth transmission means for detecting speech content input by the operator for controlling the speech of the avatar and transmitting the detected speech content to the information processing device; a fourth receiving means for receiving at least one of the video and audio of the conversation partner transmitted by the second transmitting means; a conversation partner image display means for displaying the image of the conversation partner received by the fourth receiving means; and an output means for outputting the voice of the conversation partner received by the fourth receiving means, The avatar is a fifth receiving means for receiving at least one of the control signal and the avatar voice transmitted by the second transmitting means; an operation execution means for executing an operation according to the control signal received by the fifth receiving means; an avatar voice output means for outputting the avatar voice received by the fifth receiving means; and An information processing system comprising a sixth transmitting means for detecting at least one of an image and an audio of the conversation partner and transmitting the detected image and audio to the information processing device.

2. The information processing system according to claim 1 , wherein the interlocutor information includes individual information on at least preferences of the interlocutor regarding the avatar.

3. the reaction detection means detects an emotion of the conversation partner with respect to the avatar voice based on at least one of an image and a voice of the conversation partner; 3. The information processing system according to claim 1, wherein said update means updates the importance of the individual information extracted by said individual information extraction means based on an emotion of said conversation partner.

4. 4. The information processing system according to claim 3, wherein the update means increases the importance of the individual information extracted by the individual information extraction means when the emotion of the dialogue partner is joy, and decreases the importance of the individual information or does not update it when the emotion of the dialogue partner is sadness or anger.

5. An information processing device communicably connected to a robot or computer graphics avatar and an operator terminal used by an operator who controls the movements and speech of the avatar, a dialogue partner information acquiring means for acquiring, from a database, dialogue partner information about a dialogue partner who faces the avatar and converses with the operator, the dialogue partner information including at least preference information about the avatar; a first transmission means for transmitting the dialogue partner information acquired by the dialogue partner information acquisition means to the operator terminal; A first receiving means for receiving at least one of a control signal from the operator terminal and an operator utterance content which is an utterance content of the operator; an avatar voice generating means for generating an avatar voice to be uttered by the avatar based on the operator utterance content received by the first receiving means; a second transmitting means for transmitting at least one of the control signal and the avatar voice to the avatar; a second receiving means for receiving at least one of an image and a voice of the conversation partner from the avatar; a third transmission means for transmitting at least one of the video and audio of the conversation partner received by the second reception means to the operator terminal; a reaction detection means for detecting a reaction of the dialogue partner to the avatar voice based on at least one of the video and audio of the dialogue partner received by the second receiving means; individual information extraction means for extracting individual information included in the dialogue partner information from a latest dialogue history; and an update means for updating importance of the individual information extracted by the individual information extraction means and the individual information set for the avatar based on the reaction of the conversation partner detected by the reaction detection means.

6. An information processing program for an information processing device communicably connected to a robot or computer graphics avatar and an operator terminal used by an operator who controls the movement and speech of the avatar, A processor of the information processing device a dialogue partner information acquisition step of acquiring, from a database, dialogue partner information about a dialogue partner who faces the avatar and dialogues with the operator, the dialogue partner information including at least preference information about the avatar; a first transmission step of transmitting the dialogue partner information acquired in the dialogue partner information acquisition step to the operator terminal; a first receiving step of receiving at least one of a control signal from the operator terminal and an operator utterance content which is an utterance content of the operator; an avatar voice generating step of generating an avatar voice to be spoken by the avatar based on the operator utterance content received in the first receiving step; a second transmitting step of transmitting at least one of the control signal and the avatar voice to the avatar; a second receiving step of receiving at least one of a video image and a voice of the conversation partner from the avatar; a third transmission step of transmitting at least one of the video and audio of the conversation partner received in the second reception step to the operator terminal; a reaction detection step of detecting a reaction of the dialogue partner to the avatar voice based on at least one of the video and the voice of the dialogue partner received in the second receiving step; an individual information extraction step of extracting individual information included in the dialogue partner information from a latest dialogue history; and an information processing program that executes an update step of updating the importance of the individual information extracted in the individual information extraction step and the individual information set for the avatar based on the reaction of the conversation partner detected in the reaction detection step.

7. robot or computer graphic avatar, an operator terminal used by an operator who controls the movement and speech of the avatar; and An information processing method for an information processing system including an information processing device communicably connected to each of the avatar and the operator terminal, The information processing device includes: acquiring, from a database, dialogue partner information about a dialogue partner who faces the avatar and dialogues with the operator, the dialogue partner information including at least preference information about the avatar; Transmitting the acquired dialogue partner information to the operator terminal; receiving at least one of a control signal from the operator terminal and an operator utterance content which is an utterance content of the operator; generating an avatar voice to be spoken by the avatar based on the received operator utterance content; transmitting at least one of the control signal and the avatar voice to the avatar; receiving at least one of a video image and a voice of the conversation partner from the avatar; Transmitting at least one of the received video and audio of the conversation partner to the operator terminal; detecting a reaction of the conversation partner to the avatar voice based on at least one of the received video and audio of the conversation partner; extracting individual information included in the dialogue partner information from a latest dialogue history; updating the importance of the extracted individual information and the individual information set for the avatar based on the detected reaction of the conversation partner; The operator terminal includes: receiving the dialogue partner information from the information processing terminal; Displaying the received dialogue partner information; Detecting the control signal input by the operator to control the movement of the avatar and transmitting the control signal to the information processing device; detecting speech content input by the operator for controlling the speech of the avatar and transmitting the detected speech content to the information processing device; receiving at least one of an image and an audio of the conversation partner from the information processing device; Displaying the received image of the conversation partner; outputting the received voice of the conversation partner; The avatar is receiving at least one of the control signal and the avatar voice from the information processing device; Performing an operation according to the received control signal; outputting the received avatar voice; An information processing method comprising: detecting at least one of an image and an audio of the conversation partner and transmitting the detected image and audio to the information processing device.

Images