Device, and program and the like

Abstract

To provide a device which has a function to perform communication, for example.SOLUTION: A robot 1 has a function to output voice and a function to perform communication with a user. The robot 1 interacts with the user by using an interaction engine and while being in interaction converts a content that has been spoken by the user immediately before that into text data and displays it in a touch panel part 7 that serves as a display part. Thus, the user can visually confirm the content that he / she has spoken, which contributes to correction or directionality of subsequent communication.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a device and a program having a function of performing communication or the like. [Background technology]

[0002] Patent Document 1 discloses a technique relating to an interactive communication robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-000681 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional communication robots have a problem in that they do not have sufficient capabilities. Therefore, the object of the present invention is to provide a device, program, etc. that has superior capabilities to conventional communication robots. The purpose of the present invention is not limited to this, and the present invention intends to obtain rights for configurations that aim to obtain the effects achieved from the configurations disclosed in this specification and drawings, etc., through divisional applications, amendments, etc. For example, in this specification, the phrase "can..." may be changed to "the problem is..." The present specification discloses a problem in which the problem is read as "the problem is solved by the present invention". Each problem is described as an independent problem, and the applicant intends to obtain rights for the configuration for solving this problem separately through a divisional application, amendment, etc. Even if the problem is implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in the present specification in the scope of the patent claim through amendment or divisional application. Problems that combine these independent problems are also disclosed. [Means for solving the problem]

[0005] (1) A device having a function of communicating by outputting output information to at least one of a user or another device, and preferably having a function of controlling the generation of the output information for the communication or a function of controlling the timing of output of the output information for the communication. In this way, at least one of the user and the other device can obtain at least one of output information whose generation is controlled and output information whose timing is controlled, thereby providing a device that is superior to the conventional devices. The device may control the generation of the output information for communication and generate output information according to the communication, for example. In this way, the convenience of communicating with the device, particularly with a user or other devices, is increased. The communication may be performed by outputting any kind of output information, but in particular, history information of past communication may be stored and the communication may be performed based on the history information. In addition, the communication may be performed based on history information of communication with a plurality of different users or other devices. In particular, the output information may be output from one output means, but it is preferable to configure the device to be capable of using a plurality of different output means and to select one of these output means for output. The output means may be, for example, an audio output means, a display means, a communication means, etc. The communication may be configured to appeal to voice, sight, or other five senses, such as touch.

[0006] The "apparatus" may have a function for outputting output information by voice. In this way, for example, a device that operates by inputting voice can be controlled. The apparatus may have an interface for communication and a determination means for executing communication. Note that the part included in the configuration of the "device" may be composed of multiple housings, but it is particularly preferable for it to be composed of a single housing. The device may be, for example, a system having a function of accessing a network via wired or wireless communication. In particular, the device may be, for example, a smartphone, a tablet terminal, a smart speaker, a smart camera, or the like. The appearance is not limited, but the device may be, in particular, a device that appears to communicate with others. In particular, the device may be a robot. For example, the robot may be a robot that imitates a human or animal, or may be an anthropomorphic robot other than these.

[0007] The device may have an input interface for communication, such as an input device like a keyboard, or an interface with an optical character recognition (OCR) that reads characters and converts them into data, but it may also have an audio input interface, such as a function to acquire audio data based on an audio signal converted into an electrical signal by a microphone. The output side interface may be an audio interface, such as a speaker device, earphones, etc. The output side interface may be a visual interface, such as a display whose display content can be changed, such as a liquid crystal display (LCD), a plasma display (PDP), an organic electroluminescence display, a cathode ray tube, etc. It is preferable to provide a device for outputting information by means of, for example, printed matter. It is especially preferable to provide an interface on the output side that actually generates movement. It is especially preferable to provide an actuator as an interface for actually generating movement. For example, it is especially preferable to provide a motor or the like. In particular, the device is preferably a robot provided with a member that actually generates movement. In particular, the device preferably outputs output information as the movement of a member that actually generates movement. In particular, it is especially preferable to provide an interface on the output side that is all audio, visual, and actually generates movement. A "user" is, for example, a person who can operate a device, and may be one person, but it is preferable to have multiple people. The "other device" may be the same as or different from one of the specific devices described above in terms of, for example, appearance, function, etc. The other device does not have to have a function for outputting audio. It is preferable that the other device has an audio output function. Also, the other device does not have to have a function for inputting audio, but it is preferable that the other device has an audio input function. The other devices may not have access to a network, but may have access to a network, particularly the Internet. The output information may be configured to cause a certain output from the output means. A "certain output" may be, for example, a notification to the outside. Even if it is not in the form of an obvious "notification," it can be interpreted as a "notification" if it results in some change. A "certain output" does not necessarily have to be intended to be a notification. For example, it may be the output of sound or light that contains some information or no information, or a change in some physical quantity, the movement of an object, etc.

[0008] (2) The device may have a display function that changes the display mode on a display unit in response to the voice communication.

[0009] When communicating by voice, the display mode on the display unit is changed in relation to the voice communication, so that the voice can be changed to a visual display for communication, thereby improving convenience when communicating. The "display unit" is preferably a device that changes the display mode on the display unit in response to the voice communication, and is preferably a display device such as a liquid crystal display (LCD), a plasma display (PDP), an organic electroluminescence display, a cathode ray tube, etc. In particular, the display unit is preferably provided on the device. "Displaying so that the display mode on the display unit is changed by the voice communication" may be either one of the cases where the voice of the user or another device is displayed on the display unit and the mode is changed, or the voice of the device itself is also displayed on the display unit and the mode is changed, but it is particularly preferable to have both, and in this case, either one or both may be displayed, but it is preferable to have both a state where only one is displayed and a state where both are displayed, and to have a configuration where they can be switched. For example, an example of displaying only one is a state where the face screen S1 is displayed on the robot 1 in the following embodiment 1, and an example of displaying both is a state where the chat screen S2 is displayed on the robot 1 in the following embodiment 1. As a display mode, for example, the screen may be changed in various ways in relation to the sound, for example, an animation may be executed in which an object displayed on the display screen moves in response to the sound. For example, an image may be changed in response to the output of the sound, or another image may be superimposed on an image in response to a change in the sound. Also, for example, the sound data may be converted into character data and displayed. The display of the character data may be changed moment by moment in response to changes in the sound. The communication by voice is performed by a function of controlling generation of the output information or a function of controlling timing of output of the output information for the communication. Preferably, the communication is controlled by a function that controls the generation of the output information and a function that controls the timing of output of the output information for the communication. In addition, the function of changing the display mode on the display unit may be controlled by a function of controlling the generation of the output information or a function of controlling the timing of outputting the output information for the communication, and in particular, the communication may be controlled by a function of controlling the generation of the output information and a function of controlling the timing of outputting the output information for the communication. Similarly, in the following (3) and onwards, the configuration for output from the device may be controlled by a function for controlling the generation of the output information or a function for controlling the timing of outputting the output information for the communication, and in particular, the communication may be controlled by a function for controlling the generation of the output information and a function for controlling the timing of outputting the output information for the communication.

[0010] (3) It is preferable that the change in the display mode on the display unit is based only on an utterance from at least one of the user and the other device.

[0011] By changing the display state on the display unit based on speech from the user or other devices, the user or other device can visually confirm whether their speech is being recognized by the device, which increases the convenience of communication. Also, this is a unique form of communication using a different type of human interface, which feels fresh and interesting. The speech may be directly provided by the user or another device, or may be indirectly provided by using, for example, text data of the speech. The speech may also be converted into some corresponding information, for example, other sounds or visualized patterns, and the display mode may be changed on the display unit based on the converted information. (3) is based on the speech of at least one of the user and another device, so for example, the device itself may have a voice communication function. As a configuration based on speech of at least one of the user and the other device, for example, a configuration may be provided in which speech is converted into a string of characters using a voice recognition function to identify the content of speech of at least one of the user and the other device.

[0012] (4) The change in the display mode on the display unit is alternated with the audio output from the device. Communication is not one-sided, and communication can be achieved with stable mutual understanding. "Alternate" means, for example, that communication between the device and the user or other device basically proceeds in a dialogue format, and that it is not a configuration in which only one side speaks unilaterally.

[0013] (5) The display mode may include text information converted from speech of at least one of the user and the other device.

[0014] In this way, the user or other device can see from the displayed text information how their speech is being recognized by the device, and can judge whether communication is correct from the content of the displayed text information. In addition, the content of what was said can be visually confirmed. If the recognition is incorrect, the user can repeat the statement or rephrase it in a different way, leading to correct communication. "Text information" refers to, for example, Japanese, characters based on the speech of a user or other device, such as normal kanji, hiragana, katakana, etc., and if the speech constitutes a sentence, it is preferable that the sentence has a phrase that is a mixture of kanji, hiragana, katakana, and foreign language notation. If the speech is in a foreign language, such as English or Chinese, it may be displayed in those characters. stomach. For example, it is particularly preferable to have a voice recognition function and a voice output function, and to have a function of displaying the content of speech by at least one of the user and the other device, which has been recognized by the voice recognition function and converted into text information, generating reply text information based on that content, converting the reply text information into voice information by a voice synthesis function, and outputting it as voice.

[0015] (6) It is preferable that the characters into which the speech is converted are displayed on the display unit simultaneously in their entirety from the start of the speech to its end. Since the device receives the entire speech of a certain length from the user or from other devices, it is expected that the device will respond correctly to the speech. In addition, the user can instantly visually confirm what they have said, which will contribute to correcting and directing future communication. The time from the start to the end of speech may be set taking into consideration the time it takes a person to speak in one breath. The end of speech may be, for example, the point at which a sound volume that is deemed to be no speech continues for a predetermined time. The start of speech may be, for example, when the sound volume exceeds a predetermined level. The start of speech may be, for example, when a sudden change in sound volume above a predetermined level is detected.

[0016] (7) The device may have a function of displaying, on the display unit, a dialogue history of voice communication between the device and at least one of the user and the other device as text information.

[0017] It becomes easy to understand how a conversation has taken place between the device and a user or other devices, and the convenience of voice communication is improved. The dialogue history should be displayed in a way that makes it clear which party has spoken. For this purpose, for example, speech bubbles should be provided to distinguish which utterance the text information is based on. It is preferable that past dialogues can be checked by scrolling on the screen. To indicate that the format is a dialogue, different avatar characters should be displayed on the device side and on the user or other device side. It is also preferable to simultaneously display the date and time when the utterance was made, as this will encourage the user to recall the past from the dialogue history.

[0018] (8) When the device displays the dialogue history as text information, if there is a change in the user or the other device and the dialogue target of the device changes, it is preferable that the device has a function of displaying this fact on the display unit.

[0019] If the conversation partner of the device changes, the conversation content also changes. By displaying the change in conversation partner, for example when viewing the past conversation history, the reader can see that the conversation content has changed before and after the change, and can therefore see the break in the conversation content. The device may have a function for detecting a change in the user or the other device. The change may be detected based on a change in voice characteristics, but it is preferable to have a function for detecting the change by acquiring the status of surrounding people or devices using a camera, and it is particularly preferable to have a function for detecting the change based on both.

[0020] (9) The device may have a function for displaying, on a display unit of the device, the status of recognition of at least one of the voice of the user or the other device using a voice recognition function.

[0021] For example, when a user or another device is speaking, the user can be made to understand that the device is definitely listening to what the user is saying, thereby making the user understand that a smooth dialogue is taking place. For example, as a visual function, it is preferable to change the color of the display screen or the object displayed on the display screen according to the degree of voice recognition, to display different objects according to the voice recognition situation, or to display a quantity according to the degree of voice recognition, for example, a high number if the recognition is good. As an auditory function, it is preferable to make the sound louder or quieter according to the degree of voice recognition, for example, to change the tone. In particular, it is preferable to display the character on the display unit in an anthropomorphic manner and change its facial expression.

[0022] (10) It is preferable that the device has a function for performing the communication by outputting the voice using the result of recognizing voice and converting it into a character string, and that if the result of recognizing voice and converting it into a character string contains a portion that matches a character string stored in a storage means that has stored in advance the correspondence between the resultant character string and output content, the device has a function for outputting the output content corresponding to the character string by voice.

[0023] If the character string converted by recognizing the voice matches the character string stored in the voice storage means, and the necessary voice output can be performed only within the device, the device can quickly respond to speech from the user or other devices. In addition, since there is no need to connect to an external server, the connection cost can be reduced. For example, it is preferable to prepare a large number of built-in scenarios as character strings stored in the storage means. The built-in scenario is a planned dialogue, and may be, for example, a simple greeting such as "How are you?" from the device side in response to "Good morning" from the user, or a scenario for executing a certain process such as "Open the setting screen" (user), "Are you sure?" (device), "Yes" (user), "Okay, I'll open the setting screen" (device). The storage means may be, for example, a ROM or SSD inside the computer, an external SD card, microSD card, CD-ROM, etc.

[0024] Here, "if there is a part that matches the character string" refers to a case where it matches a completely stored character string, and a case where it is a regular expression that may differ in some parts. A regular expression is one of the language processing methods that expresses a set of character strings as one character string. For example, in the case of "XXX, turn up the volume XX", if there are different parts, such as "Yupibo, turn up the volume", "Hey, turn up the volume", "Please turn up the volume", etc., the expression is recognized as "turn up the volume" as an interpretation if the essential parts match. And "outputting the output content corresponding to the character string by voice" may be, for example, a voice output such as "Yes, I will turn up the volume" in response to the character string "turn up the volume". In addition, following such a voice output, the device may perform a certain process. For example, after the utterance of "Yes, I will turn up the volume", the device may actually turn up the volume of its own utterance in the subsequent dialogue. The process of recognizing voice and converting it into a character string may be performed by the device, or may be performed by transmitting voice data to a voice recognition server connected to a network and receiving the converted character string at the voice recognition server. It is preferable to have both, and to have a function for determining which result to use depending on the communication situation, etc.

[0025] (11) It is preferable that the device has a function of connecting to a server equipped with a dialogue engine and outputting voice data when the result of recognizing voice and converting it into a character string does not match any character string stored in a storage means that stores in advance the correspondence between the resultant character string and the output content.

[0026] When the character string converted by recognizing the voice does not match any part of the character string stored in the voice storage means, the server equipped with the dialogue engine is connected, so that the cost for connection can be reduced. Cut. The server may be, for example, a device having a computer function as a storage unit and a control unit connected via an Internet line. In the present invention, it is preferable that the server is equipped with a dialogue engine. In the case of an external server, connection is possible, for example, by an ID, password, or electronic authentication. The external server is preferably a cloud server. The server is preferably equipped with a voice recognition engine, and is preferably capable of converting voice into character string data by the voice recognition engine. The converted character string data is preferably transmitted to the device via an Internet line. The function of connecting to a server equipped with a dialogue engine and outputting voice data may, for example, send a voice-recognized character string to the dialogue engine, receive a character string containing the dialogue content corresponding to that character string from the dialogue engine, and convert the character string containing the dialogue content into voice data using a voice synthesis function. The conversion of a character string into voice data may be performed by a voice synthesis engine installed in the device, but it is also more preferable to convert the character string into voice data by sending the character string to a voice recognition server that converts the character string into voice data and receiving the voice data corresponding to the converted character string from the voice recognition server.

[0027] (12) Even if the result of recognizing voice and converting it into a character string contains a portion that matches a character string stored in a storage means that stores in advance the correspondence between the resultant character string and the output content, it is preferable to have a function of connecting to a server equipped with a voice recognition engine and outputting the voice data when certain conditions are met.

[0028] When a character string converted by recognizing voice contains parts that match those stored in the memory means, outputting a fixed voice that the user can predict can discourage dialogue, so by deliberately connecting to an external server in this way, a more human-like dialogue can be achieved. For example, when the user says "Hello" and the device recognizes it, if the original scenario is a built-in scenario that has a dialogue such as "Hello, how are you?", it is preferable to request voice data of "Hello" from an external server without using that scenario, and to request the creation of response data to "Hello" using the dialogue engine of the external server. A certain condition may be, for example, once every several times or at random timing.

[0029] (13) It is preferable that the device has a function of detecting when speech is interrupted and silence occurs after speech recognition, a storage means for storing speech data from the speech recognition to the silence state, and a function of connecting the speech data stored in the storage means to a server equipped with a speech recognition engine when the silence state occurs and outputting the speech data.

[0030] Silences often occur during conversations. However, if the device remains connected to a server equipped with an external voice recognition engine even during silent periods, unnecessary costs will be incurred. Therefore, by storing voice data from voice recognition to the silent period in advance in a storage device and sending the voice data at the time of the silent period instead of in real time, the silent period can be cut, thereby reducing costs.

[0031] (14) The device may have a recording function and a function of connecting to a server equipped with a voice recognition engine and outputting voice data upon detection of a voice at a predetermined sound pressure level. It would be an unnecessary cost to always be connected to a server equipped with an external voice recognition engine. This reduces the connection cost because the system does not connect to an external server unless the conversation is silent or near silent, and only connects to the external server when a necessary conversation starts. The server equipped with a voice recognition engine receives recordings from the device over a given period of time. It may receive data and return a character string corresponding to the recorded data, but it is particularly preferable to receive voice data in real time as streaming data and return a character string. In many cases, a pay-per-use charge is made based on the number of yen per hour of voice data reception, but this makes it possible to significantly reduce costs.

[0032] (15) It is preferable to provide a function for outputting to the user a dialogue example selected from dialogue data stored in a storage means when the server is busy when connected to a server equipped with a voice recognition engine and voice data is output.

[0033] When the device is busy, it is normal to announce that fact, but such an announcement in the middle of a conversation is abrupt and unrelated to the conversation, and may put a damper on the conversation. Therefore, if the conversation is continued by a connecting utterance such as "Can you say that again?" or "Oh, I see" instead of announcing that the device is busy, it will be possible to connect to the voice recognition engine and continue the appropriate conversation without the conversation sounding unnatural.

[0034] (16) It is preferable that the system has a function of outputting a dialogue example selected from voice data stored in a storage means, without connecting to a server equipped with a voice recognition engine, when it is determined that the recognized utterance of the user is too long.

[0035] If the user's speech is too long, the voice recognition engine may misrecognize it, resulting in an irrelevant response. Therefore, when the sentence is longer than a certain length, it is often better to eliminate such a possibility and restart the conversation by selecting and outputting a dialogue example such as "Yes," "Really?", or "Really?", which can be interpreted in any way, and then making it possible to continue the appropriate conversation.

[0036] (17) In the interactive communication, when the user misses an audio signal from the device, the device may re-output the previous audio signal based on an utterance from the user. For example, if you cannot hear or have forgotten to hear the words the device just spoke, such as "Say it again" or "Speak again," you can make the device speak the words just spoken. This makes it possible to continue the conversation that was going on up until that point without interruption.

[0037] (18) If the voice cannot be recognized, the device may output a voice to prompt the user to speak again. This makes it possible to continue the conversation that was taking place up until that point without interruption.

[0038] (19) When the recognized voice content satisfies certain conditions, a certain display may be made on the display unit. For example, when a speech containing a specific word is made and the speech is recognized, the display unit is made to display a "certain display" corresponding to the word. The "specific word" may be, for example, the user's birthday, the user's child's name, a nickname for the device, the name of the company, a specific advertising catchphrase, etc. It is preferable to have a function for setting in advance the correspondence between the specific word and a certain display. This allows communication that goes beyond simple dialogue and includes visual contact, increasing the number of modes of communication with the device and improving convenience when communicating.

[0039] (20) The device has a function of moving the housing or a part connected to the housing, and detects the recognized voice When the content satisfies a certain condition, the housing or a part connected to the housing may perform a certain movement. For example, when a speech containing a specific word is made and the speech is recognized, the housing or a part connected to the housing is made to make a certain movement, such as a gesture, corresponding to the word. This allows communication that includes the movement of the device rather than just a simple conversation, and increases the number of ways to communicate with the device, improving the convenience of communication. This is particularly effective when combined with the above-mentioned "certain display."

[0040] (21) The device may include an eye portion that is a part that the user can recognize as an eye, a user position recognition function that recognizes the position of the user, and a function of moving the eye portion, and the device may include a function of moving the eye portion to face in the direction of the user's position recognized by the position recognition function as the communication function.

[0041] By having the eye parts, which can be recognized as eyes, facing the user's position, the user can obtain a pseudo-sensation of actually talking to a person, which increases the desire to communicate with the device and increases the usefulness of the device. The "eye portion that the user can recognize as an eye" may be an eye as an object displayed on a display screen, or may be an eye that actually operates mechanically and is not a virtual image. The device itself may be controlled to face the user's position in synchronization with the eye portion. Only the device for directing the eye portion toward the user may be controlled to face the user's position.

[0042] (22) The device may be equipped with a facial recognition function to recognize a user's face. Since the face of each person can be identified, it becomes possible to communicate according to each person's personality. For example, by associating the authenticated face of each person with a name, the person whose face is recognized during conversation can be called by that name. It is particularly good to configure the system to conduct a conversation specialized for the person whose face is recognized based on the past conversation history.

[0043] (23) The device may include a display unit and a function for displaying the recognition status of the user's face by the face recognition function on the display unit. In this way, the user can grasp the recognition status of the device by looking at the display unit. In particular, it is good to display whether the recognition status is complete or not yet recognized as a person's face, and in this way, if the recognition is not yet complete, the user can cooperate by not moving his / her face as much as possible so that the device can easily recognize him / her.

[0044] (24) The position recognition function may be a sound source direction identification function including three microphones arranged at the vertices of a triangle, and an identification unit that identifies a sound source direction from a position of the sound source projected onto a plane including the triangle along a direction perpendicular to the plane including the triangle based on a difference in arrival time of sound from the sound source to each of the three microphones, toward a reference position inside the area enclosed by the triangle on the plane. This allows the direction of the sound source to be identified using three microphones. If the direction of the sound source can be identified, the device can be directed in that direction when the user speaks, giving the user the feeling that they are communicating through dialogue.

[0045] (25) The device may include an infrared remote control signal output unit, and the communication may be communication with the other device that has an infrared remote control receiving function. This allows easy communication with the device via the infrared remote control signal output section of the device. For example, it is possible to output an infrared remote control signal from the device to a receiving device equipped with an infrared remote control signal receiving unit, such as a television, audio equipment, air conditioner equipment, etc., to execute control such as ON / OFF, etc. In particular, it is preferable that the device has a voice dialogue function, and in response to the utterance of a command phrase such as "Turn on the TV" or "Turn off the TV", the device controls the infrared remote control signal output unit based on the command.

[0046] (26) It is preferable that the device be remotely controlled from the other device via the Internet. The device can be remotely controlled from another device, which increases the convenience of the device. For example, the other device may be a smartphone. The device may be equipped with a camera. The device may be equipped with a mechanism for changing the direction of the camera. By accessing the device from another device, for example, it is possible to view the camera video on the device side or change the direction of the camera. This makes it possible to control the device even if you are not near the device. Also, for example, it is possible to access the device from a smartphone, turn on the monitoring function of the device, and send an e-mail to the smartphone when a person (object) moves. Also, for example, when watching over a sick person or a person receiving care, it is assumed that the person is always moving, and it is possible to send a notification when the person does not move for a certain period of time. The other device may be, for example, a tablet terminal or a personal computer.

[0047] (27) The device may perform the voice output using text information transmitted from the other device via the Internet. For example, it would be good to have a function that reads out the text of received e-mails. By setting it up so that e-mails from someone arrive, the contents of the e-mail will be output as voice from the device, eliminating the need to visually check one's own device. The other device could be, for example, a smartphone, tablet, or computer, but it could also be another "device."

[0048] (28) It is preferable that the duration, time or number of times the voice output is performed can be changed depending on the content of the text information. For example, you may want to have an email that says "I took my medicine" spoken at a specific time. For example, if the subject line matches, the device can speak at a specific time, or if you make it speak important content twice with a time gap, you can ensure that the content of the email is carried out by a user who is close to the device.

[0049] (29) The device may have a function of transmitting text information generated by voice recognition to the other device via the Internet. By using the device's voice communication function to convert voice into text and send it to another device as text data, you can, for example, send an e-mail without having to manually enter the text into your own device.

[0050] (30) The device may have a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character strings from the plurality of different servers, and selecting the longest output string for dialogue. The longest reply makes the conversation feel more like a conversation, breaks the monotony of the conversation, and allows the listener (user) to enjoy the conversation.

[0051] (31) The device may have a function of transmitting a voice-recognized character string as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character string from the plurality of different servers, and selecting the output character string with a question mark at the end for dialogue. When a question mark is added to the end of a sentence, the conversation flows in a way that further answers the question, making it easier for the conversation to continue and allowing the listener (user) to enjoy the dialogue.

[0052] (32) It is preferable that the device has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character string from the plurality of different servers, and combining affirmative sentences and then interrogative sentences to engage in a dialogue. By arranging in this way, the response will seem well thought out, so the user will feel that their speech is being seriously listened to and will want to continue the conversation, which will make the conversation easier to continue and the listener (user) will be able to enjoy the dialogue. It also allows you to save output time and ask for a response from the listener (user).

[0053] (33) It is preferable that the device has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character string from the plurality of different servers, and combining the character strings by placing a character string changing the topic at the end to have a dialogue. By arranging it in this way, the change of topic creates a dialogue that invites the next utterance, making it easier to continue the conversation.

[0054] (34) It is preferable that the device has a function of transmitting a voice-recognized character string as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character string from the plurality of different servers, and combining the friendly output character string so as to place it first to have a dialogue. Arranging it in this way makes it easier for the listener (user) to get drawn into the conversation, making it easier for the conversation to continue.

[0055] (35) It is preferable that the system has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character strings from the plurality of different servers, and combining the output character strings in a random order to have a dialogue. This increases the variety of dialogue, so that the listener (user) will not get the exact same response even if he or she makes the same utterance, making it easier to continue the conversation without getting bored.

[0056] (36) It is preferable that the device has a function of transmitting a voice-recognized character string as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character string from the plurality of different servers, and if any of the output character strings includes an emoticon, not treating it as a subject of dialogue, and displaying it on a display unit together with the output character string that was the subject of dialogue. Although emoticons cannot be output as voice, by deliberately displaying them on the display and making them part of the conversation along with the voice, it is possible to create an interesting and unusual conversation.

[0057] (37) It is preferable that the device has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character strings from the plurality of different servers, and selecting only one of the output character strings containing the same character string and combining it with the other output character string to carry out a dialogue. This is because repeating the same string of characters makes the dialogue repetitive and makes the listener feel uncomfortable.

[0058] (38) It is preferable that the device has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character strings from the plurality of different servers, converting the endings of the output character strings using an ending conversion engine, and then combining the output character strings to create a dialogue. This is good because it produces sentences that are more user-friendly than those produced by ordinary dialogue engines.

[0059] (39) It is preferable that the system has a function of transmitting a character string recognized by voice as an input character string to a plurality of different servers each having a dialogue engine that outputs an output character string corresponding to an input character string, receiving character strings output as the output character strings from the plurality of different servers, storing a portion of the output character strings in a storage means without using all of the output character strings, and retrieving the output character strings from the storage means in a subsequent dialogue for use in the dialogue. By using this when voice recognition fails or when a response from the external server is slow to come, the conversation can be continued without interruption, contributing to more natural conversation.

[0060] (40) When using a server equipped with a voice recognition engine, it is a good idea to mix free and paid servers. This allows, for example, particularly heavy users interacting with the device to save on server connection charges.

[0061] (41) The other device may be a smart speaker, and the device may output audio to the smart speaker to communicate with the smart speaker. Since smart speakers do not have a display, their convenience is enhanced when used in conjunction with a device. A smart speaker may be, for example, a speaker with a wireless communication connection function and a voice-operated assistant function. Examples of the smart speaker include Google Home, Amazon Echo, and LINE Clova. A smart speaker may be, for example, equipped with a function that realizes various functions and capabilities (skills). The function can be executed by speaking to the smart speaker from a device that communicates by voice. When the device is to speak, for example, a user may speak a phrase that activates the skill of the smart speaker, and the device may recognize the speech and save it as character string data, and at a certain timing, the character string data may be synthesized and spoken to the smart speaker to execute the skill.

[0062] (42) The other device may be a smart speaker, and the device may output audio to the smart speaker to communicate with the smart speaker. Activate the smart speaker, activate a skill, or make an inquiry. It is possible to automatically execute a skill on the smart speaker at a certain timing or a scheduled time without activating the smart speaker yourself.

[0063] (43) The other device may be a smart speaker, and the device may have an adaptation function for adapting the audio output of the smart speaker. This is useful if you don't want to listen to the long, standard answers your smart speaker gives you to questions, or if you just want a quick refresher.

[0064] (44) The audio output of the device may have the function of reading aloud web articles. You can listen to the web article without having to read it, just by interacting with the device.

[0065] (45) The certain output may be generated when execution of a predetermined processing unit of the robotics process automation is completed. Robotics process automation makes it difficult to know the execution status of processing units, but By having the processor produce "a certain output" in response to the execution of a processing unit, the processing status becomes easier to understand, and convenience is increased.

[0066] (46) The certain output may be an announcing operation. The notification operation allows the user to know that a certain output has been made. (47) When a computer executing robotics process automation enters a state of waiting for input from the user, an alarm is issued. This makes it possible to notify the user that the input waiting state has begun and prompt him / her to perform the next process.

[0067] (48) It is preferable to provide a client computer that performs robotics process automation and a server computer that gives instructions to the client computer to execute robotics process automation, and to perform an alerting operation when the client computer receives instructions from the server computer. This makes it possible to notify the user that an instruction has been received from the server computer and prompt them to proceed with the next step.

[0068] (49) It is preferable to generate the output information to perform a pointing action on a computer running robotic process automation. This allows the user to know on which computer the execution has taken place, and makes it possible to prompt the user for the next process. (50) It is preferable that the output information be generated differently depending on the execution state of the robotics process automation. This makes it possible to distinguish what kind of execution has been performed.

[0069] (51) A program for causing a computer to realize the functions of the device according to any one of (1) to (50). In places where "a certain" is written, such as "a certain output," it is better to change it to "a specified" for example. The inventions shown in (1) to (50) above can be arbitrarily combined. For example, the invention shown in (1) may be combined with at least a portion of the configuration of at least one of the inventions shown in (2) and after. In particular, the invention shown in (1) may be combined with at least a portion of the configuration of at least one of the inventions shown in (2) and after. Any configuration may be extracted from the inventions shown in (1) to (50) and the extracted configurations may be combined. The applicant of this application intends to obtain rights to the inventions including these configurations. In addition, even if there is a description such as "in the case of" or "when", this is not intended to describe the configuration as being limited to that case or time. Configurations other than these cases or times are also disclosed, and the applicant intends to obtain rights. In addition, the parts described in order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and the applicant intends to obtain rights. Effect of the Invention

[0070] When communicating with a user or other devices, the device can generate output information according to the communication, which improves the convenience of communicating with the device from the user or other devices. The effects of the invention of this application are not limited to this, and effects obtained from the configuration disclosed in the specification and drawings, etc. are also disclosed, and the intention is to obtain rights to the configuration that achieves the effects by divisional applications, amendments, etc. For example, in this specification, the phrase "can" is a description that clearly indicates the effect that is achieved, and there are also parts that show the effect even without the phrase "can." Also, there are effects that can be understood from the configuration even without such a description. [Brief description of the drawings]

[0071] [Figure 1] FIG. 1 is a front view of a robot according to a first embodiment of the present invention. [Diagram 2] FIG. 4 is a side view of the robot according to the first embodiment. [Diagram 3] FIG. 4 is a rear view of the robot according to the first embodiment. [Figure 4] FIG. 2 is a block diagram illustrating the electrical configuration of the robot. [Diagram 5] FIG. 11 is an explanatory diagram showing an example of a facial expression displayed on the face of the robot. [Figure 6] FIG. 13 is an explanatory diagram capturing an example of a chat state with a chat screen displayed on the face of the robot. [Figure 7] FIG. 13 is an explanatory diagram illustrating a standby image with a face screen displayed on the face of the robot as a background. [Figure 8] FIG. 13 is an explanatory diagram illustrating a wallpaper image with a chat screen displayed on the face of the robot as the background. [Figure 9] 5A to 5D are explanatory diagrams illustrating deformation patterns of eye objects displayed on the face of a robot. [Figure 10] 13(a) to 13(c) are explanatory diagrams showing how the user's speech gradually appears as a string of characters on the robot's face. [Figure 11] FIG. 11 is an explanatory diagram illustrating a state in which an eye object moves to follow the user's face on a face screen displayed on the robot's face. [Figure 12] FIG. 1 is an explanatory diagram illustrating an example of a smartphone. [Figure 13] FIG. 2 is an explanatory diagram for explaining the relationship between the robot startup, wake-up mode, dialogue mode, and sleep mode. [Figure 14] FIG. 23 is an explanatory diagram for explaining the relationship between a robot and a smart speaker in the seventh embodiment. [Figure 15] FIG. 23 is an explanatory diagram for explaining an example of a smartphone in the ninth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0072] <Embodiment 1> As shown in Figs. 1 to 3, the robot 1, which is a communication robot that operates in response to human voice, is provided with a fixed part 2 that is the lower body and a movable part 3 that is the upper body placed on the fixed part 2, as a housing. The movable part 3 is composed of a body part 4 arranged adjacent to the fixed part 2, and a head part 5 supported by the body part 4. The fixed part 2 is formed in a bowl-like appearance that opens upward, and the body part 4 is formed in a cylindrical shape that is composed of an upper edge of the fixed part 2 and a continuous curve in the vertical direction. The robot 1 is configured so that the connection part between the fixed part 2 and the body part 4 has the largest diameter, and the external shape of the robot 1 narrows in the vertical direction from the connection part. The front of the body part 4 is largely cut out in a semicircular shape. The head part 5 is fitted so as to be embedded in the upper part of the body part 4. The body part 4 rotates horizontally (in the direction of the arrow in Fig. 1) relative to the fixed part 2, and the head part 5 rotates in two directions, vertically (in the direction of the arrow in Fig. 2) and left and right rotation directions (in the direction of the arrow in Fig. 3), relative to the body part 4.

[0073] The head 5 as a whole is configured in a spherical shape that is the remainder when a part (front part) of a sphere is cut by a single plane. The cut-like front part appears as a circle and constitutes the face part 6 of the robot 1. The rectangular part formed on the surface of the face part 6 is the touch panel part 7 as a display part that is a liquid crystal display (LCD) equipped with a touch panel function. The contents displayed on the touch panel part 7 will be described later. A smoke panel is arranged in the area of ​​the face part 6 around the touch panel part 7, and the entire face part 6 has a unified dark background. Inside the head 5, an illuminance sensor 8 and a high-luminance white LED 9 are arranged at the left and right storage positions above the face part 6. A lens 11 of a face recognition camera 10 is arranged at the upper center position of the touch panel part 7 in the face part 6.

[0074] Inside the torso 4, the same two positions are located 120 degrees apart from each other at the front left and right positions and at the rear center position of the torso 4. Microphones 12 are disposed at three positions at the same height. A pair of left and right speaker devices 13 are disposed on the inside of the fixed part 2. An opening 14 is formed on the side of the speaker device 13 for outputting sound generated by the speaker device 13. A power switch 15 and an up switch 16 and a down switch 17 for adjusting the volume of the speaker device 13 are disposed adjacent to the speaker opening 14. A USB OTG (On-The-Go) terminal 18, a DC 12V power jack 20, and a micro SD card socket (reader) 19 are disposed at the rear of the fixed part 2.

[0075] The robot 1 can connect to a predetermined external cloud server using an internet line. The cloud server is equipped with a storage area as a storage means for storing data required by the robot 1, various engines for performing various processes required by the robot 1, etc. Therefore, in a broad sense, the robot 1 can be interpreted as a device including the software and other parts of these cloud servers.

[0076] Next, the electrical configuration of the robot 1 according to the first embodiment will be described with reference to the block diagram of FIG. The controller MC, which serves as a control means, is connected to the above-mentioned touch panel unit 7, illuminance sensor 8, high-brightness white LED 9, face recognition camera 10, microphone 12, speaker device 13, terminal 18, and socket 19 for a micro SD card, and in addition to these, a wireless LAN device 21, a Doppler sensor 22, and first to third motors 23 to 25, etc. are also connected.

[0077] The touch panel unit 7 has an input operation function that allows input by touching its surface. In a natural dialogue mode described later, the touch panel unit 7 can display different screens as shown in Fig. 5 or Fig. 6. The controller MC causes the touch panel unit 7 to displaceably display, as a first screen, a face screen S1 as shown in Fig. 5, which controls the facial expression of the robot 1, particularly the changes around the eyes. The face screen S1 is a screen that is displayed by default, and displays an eye object 27, a cheek object 28, and an elliptical region 29 of the robot 1. The eye object 27 has several deformation patterns of the eye object 27 as an animation image (Figs. 9(a) to (d)). Also, an pupil object 27a moves left and right as an animation image.

[0078] 6 as a second screen on the touch panel unit 7. The chat screen S2 can be displayed on the touch panel unit 7 in place of the face screen S1 by touching and sliding the touch panel unit 7 in the face screen S1 state. The face screen S1 and the chat screen S2 can be switched back and forth by the sliding operation. The chat screen S2 will be described later. In addition, in the standby mode (wake-up mode), the touch panel unit 7 can display the standby image shown in Fig. 7 or 8 on the touch panel unit 7. The standby image will be described later. A setting screen is prepared as a different image for the robot 1, and it is possible to move to this setting screen from the chat screen S2. When the robot 1 is started for the first time, it is accessed from the setting screen, and necessary initial setting items such as setting and registering ID and password to the server, setting and registering Wi-Fi password, user registration (e.g. name, age, gender, etc.), face recognition, setting an e-mail address to which data will be transferred, etc. are input.

[0079] The illuminance sensor 8 detects the brightness of the environment in which the robot 1 is installed. The high-luminance white LED 9 is automatically turned on when the light intensity is insufficient for the face recognition camera 10 to take a picture based on the value detected by the illuminance sensor 8. The microphone 12 is a voice input means for acquiring the user's speech in a dialogue with the user, and the three microphones 12 arranged at the vertices of the triangle are used simultaneously. By detecting the difference between the arrival times of the sound between the microphones 12, the controller MC can also be used as a direction detection means for identifying the direction of the sound source. The controller MC obtains the arrival time difference from the phase difference of the electric signals acquired by each microphone 12. The controller MC calculates the sound source angle with respect to the reference direction based on the arrival time difference. A microphone specialized for conversation with the user may be provided on the face unit 6, for example. The speaker device 13 is a voice output means for the robot 1 to speak (output voice) in conversation with the user. The microSD card socket 19 reads and rewrites data from and to an inserted microSD card. The wireless LAN device 21 is a device for wirelessly connecting the robot 1, which is a Wi-Fi compatible device, to the Internet. In this embodiment, the wireless LAN device 21 is compliant with the international standard IEEE802.11b. The Doppler sensor 22 is a sensor that uses microwaves, and detects whether an object (person) is moving by emitting microwaves and comparing the frequency of the reflected microwaves with the frequency of the emitted radio waves. It utilizes the fact that the frequency of the reflected waves changes when an object (person) is moving due to the Doppler effect. For example, it is a device used to detect abnormalities around the robot 1, such as the presence or absence of a suspicious person when the user is absent. The first motor 23 is a servo motor for rotating the torso 4 horizontally (in the direction of the arrow in FIG. 1) relative to the fixed part 2. The second motor 24 is a servo motor for rotating the head 5 vertically (in the direction of the arrow in FIG. 2) relative to the torso 4. The third motor 25 is a servo motor for rotating the head 5 left and right (in the direction of the arrow in FIG. 3) relative to the torso 4. When the direction in which the user will speak is determined by the microphone 12, the controller MC controls the first motor 23 to rotate the torso 4 (movable part 3) relative to the fixed part 2 so that the face part 6 faces the direction of the user.

[0080] The controller MC is composed of a known CPU, ROM, RAM, memory as a storage means such as SSD, a bus, a real-time clock (RTC), etc. Various programs for executing various functions of the robot 1 are stored in the ROM of the controller MC. The various programs include, for example, a dialogue program for controlling dialogue with the user via the microphone 12 and the speaker device 13, a face recognition program for face recognition using the face recognition camera 10, a display change / gesture program for controlling the touch panel unit 7 and the first to third motors 23-25 ​​to change the facial expression and movement of the robot 1 during dialogue with the robot 1, a screen display program for displaying different screens and images on the touch panel unit 7 based on dialogue with the user and operation of the touch panel unit 7, a data transmission / reception program for processing, for example, camera images acquired by the robot 1 side and e-mails from a smartphone or the like between the robot 1 and other computers or smartphones, an absence setting program for watching over the user when the user is absent, an OS for operation, management, and operation of GUI functions, Internet connection functions, process management, etc. are stored in the RAM. Input / output data for dialogue and face recognition are temporarily stored in the RAM. Each program realizes functions such as dialogue, face recognition, gestures, etc. in a multitasking manner in cooperation with other programs or independently.

[0081] A. About the behavior during dialogue In the above configuration, the controller MC executes a dialogue program to control dialogue-based communication with the user. Note that face recognition, which becomes possible in synchronization with the start of dialogue, will be described later in "B. Operation during face recognition." Here, the dialogue program is 1) A subprogram for issuing a request to the cloud server for user speech data (voice data) acquired from the microphone 12, and responding with the user speech data (character string data) that has been converted to text using a server-side voice recognition engine. 2) Execute built-in scenario dialogue based on user utterances (string data) Built-in scenario subprograms 3) A subprogram for transferring speech data, which requests the cloud server again to send back speech data (character string data) when the user's speech does not correspond to the built-in scenario, and uses a dialogue API (application programming interface) to have the dialogue engine create response data (character string data) for the robot 1. 4) A character string data display subprogram for displaying the response data (character string data) on the touch panel unit 7 as a display unit. 5) A voice data subprogram for converting the response data (character string data) into voice data by a voice synthesis engine and outputting the voice as an utterance from the robot 1 side from the speaker device 13. 6) a display mode / motion change subprogram for changing the display mode on the touch panel unit 7 and the motion of the robot 1 based on the user side character string data and the robot 1 side character string data; etc. An example of the control contents of the controller MC mainly based on the dialogue program will be described below, along with the mutual relationships between the standby mode (wake-up mode) after startup, the natural dialogue mode, and the sleep mode. These mutual relationships are as shown in FIG. Figures 5 and 6 show screens in the natural dialogue mode, and Figures 7 and 8 show screens in the standby mode. In the sleep mode, these screens are darkened with the backlight of the touch panel unit 7 turned off.

[0082] 1. Start The robot 1 is started by turning on the power switch 15 (process M0 in Figure 13). The boot program is executed in the controller MC, and then the OS starts up, and the OS enters a state of waiting for a "command" from the user, that is, a wake-up state. In this initial standby mode, the standby screen in Figure 7 is displayed. In the following, we will assume that the robot is in a state after the initial settings have been completed, i.e., the ID and password for Robot 1 have been registered on the cloud server, user registration has been completed, facial recognition has been performed for multiple users, and the smartphone email addresses have been registered to Robot 1, and so on. "1. Effect on startup" In this way, one of several standby screens is selected and displayed at the start (Fig. 7). In other words, a fixed screen is always displayed at start-up. The eyes of Robot 1 are closed (implying that conversation is not possible), so the user can easily tell that it is in standby mode.

[0083] 2. Standby mode (wake-up mode) The standby mode is a state in which a conversation with the robot 1 is possible when a trigger to start the natural conversation mode is received. It is also a state to which the robot enters when the conversation ends in the natural conversation mode. It is also a state in which the robot enters sleep mode if the natural conversation mode is not started for a certain period of time. The sleep mode is a state in which conversational communication with the robot 1 is not possible. Here, "natural dialogue" refers to a user dialogue with the voice of the device (robot 1) that synthesizes the voice using the built-in scenario of the robot 1 or the dialogue engine (dialogue software) on the server. The natural dialogue mode is a state in which natural dialogue is possible. There are multiple standby mode screens, and in this embodiment, two types are provided, as shown in FIG. 7 and FIG. Fig. 7 is the standby screen to which the screen in the natural dialogue mode of Fig. 5 is switched (process M2 of Fig. 13). It is also the standby screen displayed at the time of startup by process M0 of Fig. 13. In Fig. 7, a face screen S1 layer screen in which the robot 1's eyes (eye object 27) are closed is displayed lightly against the background of a clock layer screen that displays the date, day of the week, and the current time in large letters. Fig. 8 shows the standby screen to which the screen in the natural conversation mode of Fig. 6 is switched (process M2 of Fig. 13). In Fig. 8, the chat screen S2 layer screen is displayed in a faint manner against the background of a clock layer that displays the date, day of the week, and the current time in large letters. In other words, it is in standby mode, but not in natural conversation mode. "2. Effect in standby mode (wake-up mode)" In this way, since different standby screens are prepared, when the natural conversation mode is started from a certain standby screen, the user can have a conversation on the screen that was accessed immediately before, which is convenient. Also, by displaying a screen specific to the standby mode, the user can easily know that the robot 1 is in the standby mode.

[0084] 3. Starting and Stopping Natural Dialogue Mode After being started and in standby mode or sleep mode, the controller MC processes to transition to natural conversation mode at multiple times, i.e., by mode transition triggers, as shown below (processes M1 and M5 in FIG. 13). The following triggers are examples. In the natural conversation mode, the device switches to face recognition mode (face recognition becomes possible) as described in "B. Operation during face recognition" below.

[0085] 1-1) In the standby mode, if the controller MC recognizes an utterance (voice) such as "Hey, Yupibo" from the microphone 12 as a start-up phrase within a certain period of time, this triggers the switch to the natural dialogue mode (process M1 in FIG. 13). Also, if it determines that a touch action has been made on the touch panel unit 7, this also triggers the switch to the natural dialogue mode (process M1 in FIG. 13). 1-2) In the sleep mode, the controller MC judges whether or not there has been a touch operation on the touch panel section 7 of the standby screen at a predetermined timing. The touch operation on the touch panel section 7 differs depending on the display mode on the face section 6, and is initiated by touching the entire touch panel section 7 on the standby screen of the face screen S1, and by touching a dialogue start button object 36 (described later) on the standby screen of the chat screen S2. In other words, different operations are initiated on different screens. If it is determined that a touch operation has been performed on the touch panel unit 7, the controller MC temporarily switches to standby mode (process M3 in FIG. 13), and if it is subsequently determined that another touch has been performed, it switches to natural dialogue mode (process M1 in FIG. 13). 1-3) In the sleep mode, the controller MC judges whether or not a touch action has been performed on the touch panel unit 7, similarly to 1-2). If it is judged that a touch action has been performed, the controller MC temporarily switches to standby mode (process M3 in FIG. 13). In this state, if the controller MC recognizes an utterance (voice) such as "Hey, Yupibo" from the microphone 12 as a startup phrase within a certain period of time, the controller MC uses this as a trigger to switch to natural dialogue mode (process M1 in FIG. 13). 2) In the sleep mode, the controller MC judges whether a predetermined time set in advance by the RTC has arrived, and switches to the natural conversation mode at the timing when the predetermined time has arrived (process M5 in FIG. 13). 3) In the sleep mode, the controller MC outputs a certain utterance (voice data) from the speaker device 13 at random time intervals, for example, using random numbers generated at a predetermined timing. In other words, the controller MC makes the robot 1 output a voice inviting a conversation, such as "Hey, what are you doing?" or "I'm bored," as a kind of monologue, to enter the natural conversation mode (process M5 in FIG. 13), and prompts the user to speak. 4) In the sleep mode, the controller MC determines whether or not an object (person) is moving using the Doppler sensor 22, and switches to the natural conversation mode when it detects that the object (person) is moving (process M5 in FIG. 13).

[0086] 5) In sleep mode, the controller MC detects weather changes such as abnormal weather and earthquakes. When such a weather event occurs, the user is notified and natural dialogue is started as a trigger. An external cloud server acquires and stores information on abnormal weather, including weather abnormalities (e.g., heavy snowfall, typhoons, etc.), earthquakes, lightning strikes, etc., at regular intervals using an abnormal weather detection engine according to a certain standard. The regular times may all be the same, or the acquisition timing may vary depending on the weather. In the first embodiment, the abnormal weather information is distributed to the device (controller MC) by employing, for example, a push-type distribution system from the server. When the controller MC acquires the information, it switches to natural dialogue mode (process M5 in FIG. 13). 6) If the controller MC detects no user speech for a certain period of time while in the natural dialogue mode in any of 1) to 5) above, it switches to standby mode (process M2 in FIG. 13), and switches to sleep mode after a certain period of time (process M4 in FIG. 13). The length of these mode change times can be appropriately changed by speech, for example, via the terminal device, or as a built-in scenario.

[0087] "3. Effects on starting and stopping natural dialogue mode" By providing a variety of different natural conversation modes, the user can talk to the robot 1 at various times, which increases the opportunities to talk to the robot 1 and thus increases the opportunities to enjoy natural conversation, making the user feel the benefits of owning the robot 1. In addition, when the conversation mode ends, the robot goes into standby mode and then into sleep mode, reducing power costs. In addition, the screen goes from sleep mode to standby mode and goes to natural conversation mode, so conversation can start immediately, making the conversation start smooth. Also, the conversation screen (Fig. 5 and Fig. 7) is displayed as long as the conversation continues, which stimulates the user's desire to talk.

[0088] 4. Built-in scenario dialogue in natural dialogue mode In natural dialogue mode, multiple dialogue processes are available: built-in scenario dialogue and normal dialogue using the server's dialogue engine. The controller MC first judges whether the speech data (character string data) based on the user's speech matches the built-in scenario, and if not, controls the dialogue to take place using a dialogue engine via the cloud server (hereafter referred to as normal dialogue).From the user's perspective, it appears that they are always conversing with the robot 1, but in reality there are multiple internal processes in the natural dialogue mode. The controller MC executes a process of comparing the user's speech (character string data) created by the voice recognition engine on the cloud server side with the text data of the built-in scenario (script). In this embodiment, the text data of the built-in scenario is stored in memory. The built-in scenario may be added to an SD card. If it is an SD card, it is easy to increase the number of built-in scenarios one after another by rewriting it. When the controller MC recognizes the user's speech, it judges whether the character string data matches a planned regular expression or non-regular expression, and if it matches, it converts the script corresponding to the character string data into voice data using a voice synthesis engine and outputs it as the speech of Robot and 1 from the speaker device 13. Many built-in scenarios are set (prepared), such as simple scenarios such as greetings like "Hello" and "It's a nice day today" to encourage the user to speak, and scenarios for requesting some kind of processing based on the user's speech. Tables 1 to 3 show examples of such built-in scenarios. Of course, in reality, many more built-in scenarios are set in addition to these built-in scenarios.

[0089] If the dialogue is not carried out according to the built-in scenario, the dialogue in the built-in scenario will end midway. Examples of cases where the dialogue is not carried out according to the built-in scenario include the following: It is a combination. 1) If the regular expression or non-regular expression does not match the expected expression This is the case where the built-in scenario does not match the regular expression or non-regular expression from the beginning or halfway through. It also includes the case where the user's speech is not clear and the utterance cannot be acquired correctly. In this case, the controller MC judges it to be a normal dialogue and immediately connects to the external cloud server, and thereafter issues a request for speech data to the external cloud server, and causes the dialogue engine on the external cloud server to create response data converted into character string data. The response data is then converted into voice data by the voice synthesis engine, and the voice is output from the speaker device 13 to continue the dialogue.

[0090] 2) When the dialogue in the built-in scenario ends as scheduled For example, when a user is asked to follow a scenario, such as "May I do XXX?", the built-in scenario dialogue ends as planned with a positive response such as "Yes" or "Please", and the dialogue disappears, or the dialogue disappears midway through the scenario. In this case, the device will enter standby mode after a certain period of time. 3) When the user responds negatively about whether or not a certain process should proceed If the user follows a scenario and speaks, for example, "May I do XXX?", and responds with a negative utterance such as "No" or "It was a mistake" rather than a positive utterance such as "Yes" or "Please", the built-in scenario will end and the subsequent dialogue will be the same as 1) or 2). When this negative utterance is made, the controller MC confirms whether it is OK to stop the process by asking, for example, "Are you sure?" This makes it possible to deal with a user's slip of the tongue, a change of mind, etc. For example, in a power-off scenario, when the user is asked the question "Are you sure you don't need to turn off the power?" in accordance with the scenario, if the user utters "Yes", the controller MC repeats the question "Are you sure you don't need to turn off the power?" multiple times (for example, three times in this embodiment) and if the user utters "Yes", the dialogue in the built-in scenario ends.

[0091] "The effect of using a built-in scenario" With such a built-in scenario prepared, there is no need to request all dialogue from an external server, and all dialogue can be processed within the device, reducing the communication costs for connecting to the server, and since communication time and server-side calculation time are not required, there is no discomfort caused by a slow response to the user's utterance, causing the conversation to be interrupted. Also, for example, by providing such a built-in scenario when executing a fixed process, the user does not need to operate the touch panel unit 7 or access the robot 1 from another terminal to execute the process, and the process can be executed through dialogue, making it more user-friendly.

[0092] [Table 1]

[0093] [Table 2]

[0094] [Table 3]

[0095] 5. Requests and responses in normal conversations On the other hand, if the speech data (character string data) is not a built-in scenario, the controller MC connects to the cloud server, transmits the speech data (character string data) to the server again, and requests the dialogue engine to create response data. If the user has been authenticated, the controller MC sends the authentication information (e.g., ID and password) for each user at the beginning of the speech data in the request. In that case, the response data is created taking into account past dialogue information. On the other hand, if the user has not been authenticated, the authentication information is not sent as the user is a person who has not been identified in the past, so past dialogue information is not taken into account. When a request is received, the cloud server uses a dialogue API (Application Programming Interface) to have the dialogue engine create response data (character string data) based on the speech data (character string data), and responds to the robot 1 (controller MC). If there is a past dialogue history of the user, response data is created taking into account the contents of the history. The controller MC converts this response data into voice data and outputs it from the speaker device 13 as the robot 1's speech. As with the built-in scenario above, if the user remains silent for a certain period of time, the device will enter standby mode. "5. Effects of requests and responses in normal conversations" Unlike built-in scenarios, by connecting to an external cloud server and conducting normal conversations, advanced conversation analysis can be performed quickly using a significantly larger amount of data than with built-in scenarios, enabling advanced conversations that are as similar to those occurring when actually conversing with a person.

[0096] 6. Robot 1's behavior during conversation The controller MC executes the following behavioral control (i) to (iv) when a dialogue is taking place in natural dialogue mode, regardless of whether it is a built-in scenario or normal dialogue. B. Gestures of Robot 1 during startup, natural dialogue mode, and standby mode The controller MC controls the first to third motors 23 to 25 at the following various timings of the robot 1 to change the posture of the robot 1. The following is one example. 1) At startup: If the face part 6 of the head 5 is not facing forward or if the head 5 is tilted, it is moved to a default position facing forward. 2) When the screen is touched: Same as 1) (to face the face recognition camera 10 directly to the user) 3) When the trigger "Hey Yupibo" is spoken: Same as 1) (to face the face recognition camera 10 directly to the user) 4) When the direction of a sound is detected: Face portion 6 of head 5 is turned in that direction. 5) As a special emotional utterance, for example, when the user is happy, the third motor 25 is controlled so as to rotate the head 5 left and right (clockwise and counterclockwise) while keeping the face 6 facing the user. 6) As a special utterance, for example when the user is sad, the second motor 24 is controlled so that the head 5 remains stationary while nodding for a while, and then returns to the default position. 7) Emotions are expressed as speech. For example, when a greeting utterance such as "good morning," "hello," "good evening," or "hello" is generated, the second motor 24 is controlled to bow the head 5. 8) When a non-special emotional utterance is an utterance of simple positive communication terms such as "I see," "I got it," "That's right," or "Yup!", the second motor 24 is controlled to nod the head 5. In 6 to 8), it is advisable to change the speed and time of the second motor 24 to differentiate between sadness, bowing, and nodding. 9) When a non-special emotional utterance is made, for example, a simple negative communication term such as "no," "I can't do it," or "It's not OK," the first motor 23 is controlled to rotate the movable part 3 left and right several times. 10) Depending on special or non-special emotional utterances and various dialogues, various gestures, such as turning the head 5 left and right (clockwise and counterclockwise) or forward and backward several times, turning the entire movable part 3 left and right, or turning it in a large or small nodding motion, may be combined. The gestures of the robot 1 may be combined with the displays on the touch panel unit 7 (face unit 6) described below. "6. Effects of Robot 1's Behavior During Dialogue, Part 1" By having the robot 1 perform these kinds of gestures, the user will feel a sense of familiarity with the robot 1, and will enjoy conversing with the robot 1 while also learning the joy of actively interacting with it.

[0097] B. Change in display mode when face screen S1 is in state 1) When the user speaks When the controller MC acquires and recognizes the user's speech from the microphone 12, it displays an elliptical region 29 in blue on the face screen S1 of the touch panel unit 7 as shown in Fig. 5, and displays an animation in which the area (i.e., size) of the region changes according to the volume of the user's speech. Specifically, the controller MC expands the elliptical region 29 while maintaining its elliptical shape when the volume of the user's speech increases, and shrinks the region while maintaining its elliptical shape when the volume decreases. In addition, the controller MC displays the cheek object 28 in green.

[0098] Furthermore, the controller MC causes the touch panel unit 7 to display the user's speech data (character string data) received as a response from the cloud server in a predetermined format. For example, as shown in FIG. 5, when the touch panel unit 7 is the face screen S1 and receives an utterance from the user, for example, "Hello," the controller MC changes the layer of the face screen S1 to the layer of the face screen S2. Based on this utterance, the character string "Hello" is displayed on the screen. In terms of order, this display starts before the utterance of Robot 1 in response to the user's utterance. As a display mode, for example, the face screen S1 is displayed so that it gradually becomes opaque from a transparent state as shown in FIG. 10(a) to FIG. 10(b), and finally the face screen S1 behind is completely hidden as shown in FIG. 10(c). In other words, the character string is gradually displayed. After the character string alone is displayed brightly against a dark background as shown in FIG. 10(c) for a very short period of time, the layer screen displaying the character string is gradually erased from FIG. 10(c) to FIG. 10(b) to FIG. 10(a), and the default state of the face screen S1 is restored. At this time, the character strings for one utterance are displayed so that they all appear and disappear at the same time. This display mode is one example, and the character strings may be displayed in a different mode. "6. Effects of Robot 1's behavior during conversation, Part 2" This allows the user to see what they have said on the robot 1, so they can check whether the robot 1 has heard them correctly, judge whether the dialogue is being carried out correctly, and guide the dialogue so that it does not become strange or irrelevant. Also, while irrelevant conversations can be frustrating, by checking the dialogue, the user can understand the reason and try the dialogue again by changing the way they speak. In addition, since the user's speech data, both the subject of built-in scenarios and the subject of normal dialogue, is first converted to character string data by a request to the cloud server, there is no need for the time-consuming preparatory processing for converting the data into character string data. Also, since the creation of response data by the dialogue engine is requested only after such character string data has been obtained, the user's speech can be displayed on the touch panel 7 at least before the robot 1's response data is spoken, and there is no risk of getting the dialogue order wrong.

[0099] When a user's utterance is treated as a character string, its length is not the same because it varies depending on the utterance. Also, if the utterance is a "sentence" with phrases rather than words, it may be quite long. Even when the utterance is a long sentence, the controller MC adjusts the font size of the character string so that the contents of one utterance are all displayed simultaneously on the touch panel unit 7. In other words, if the utterance is short, it is displayed in a larger font, and if the utterance is long, it is displayed in a relatively smaller font. "6. Effects of Robot 1's behavior during conversation, Part 3" This allows users to see what they say with a single glance, making it difficult to interrupt the conversation until the entire sentence appears, and difficult for it to overlap with the next user's speech. Also, because each utterance appears at the same time, the entire sentence can be understood all at once, and the user can fully understand it even if the display time is short. Also, because the text is displayed across the entire touch panel 7, each character can be displayed large and easy for the user to see, making it easy to see even if the display time is very short.

[0100] 2) When Robot 1 is speaking On the face screen S1 of the touch panel unit 7 as shown in Fig. 5, an elliptical area 29 is displayed in red, and an animation is displayed in which the area (i.e., size) of the area changes according to the robot's speech volume. Specifically, the controller MC expands the elliptical area 29 while maintaining its elliptical shape when the output level from the speaker device 13 increases, and shrinks the area while maintaining its elliptical shape when the volume decreases. Also, the cheek object 28 is displayed in light red. "6. Effects of Robot 1's behavior during conversation, Part 4" In this way, the display state on the face screen S1 changes according to the alternating dialogue between the user and the robot 1, and this alternating dialogue not only in real life but also on the screen is interesting, making the conversation lively.

[0101] C. Change in display mode of chat screen S2 A display state accompanying a conversation on the chat screen S2 of the touch panel unit 7 will be described with reference to Fig. 6 and Fig. 8. As described above, the display changes from the face screen S1 to the chat screen S2 in response to a user operation. First, the configuration of the chat screen S2 will be described again. As shown in Fig. 6, a user object 31 is displayed as an avatar character at the lower left of the chat screen S2, and a robot object 32 is displayed at the lower right of the chat screen S2, facing each other. A different user object 31 is prepared for each authenticated user or unauthenticated user recognized in a face recognition mode described later, and a different object is displayed depending on the user currently in conversation. In the central area, speech bubble objects 33 in which the contents of the conversation between the user side and the robot 1 side are converted into characters and arranged in order along the time axis. A conversation stop button object 34 is displayed at the upper left of the chat screen S2. A setting button object 35 is displayed at the upper right of the chat screen S2.

[0102] On the chat screen S2, speech bubble objects 33 are displayed so as to be added moment by moment in response to the conversation between the user side and the robot 1 side. The speech bubble objects 33 display the user's utterance contents converted into character string data and the robot 1's utterance contents converted into character string data in a row along the time axis, and can be displayed on the chat screen S2, and the most recent utterance contents are displayed in a new speech bubble object 33 at the bottom of the row of past speech bubble objects 33 in synchronization with the utterance. The speech bubble object 33 indicates the direction of speech so that it is clear whether the content is from the user's side or the robot 1's side. Since it is not possible to display the entire dialogue history on the screen at once, the chat screen S2 is configured to be scrollable in the vertical direction, and speech bubble objects 31 can be displayed going back in time. When not going back in time, the speech bubble object 33 of the most recent dialogue is always displayed. In this embodiment 1, if the dialogue is resumed after the dialogue has ended and the standby mode has been entered, and the user newly recognized in the face recognition mode described below is changed at that time, "user change" is displayed in the middle of the row of speech bubble objects 33, and a different user object 31 is displayed according to the newly recognized user.

[0103] Furthermore, by touching the dialogue stop button object 34 on the chat screen S2, the dialogue is actively interrupted by the user and the screen goes into standby mode. In this case, the standby screen of the chat screen S2 in Fig. 8 is displayed instead of Fig. 6, but a dialogue start button object 36 is displayed in place of the dialogue stop button object 34. To return to the natural dialogue mode, the user can return to the chat screen S2 in Fig. 6 by touching the dialogue start button object 36. "6. Effects of Robot 1's Behavior During Dialogue, Part 5" In this way, the user object 31 of the user who is in an interactive relationship and the robot object 32 of the robot 1 are placed facing each other, and the speech bubble objects 33 that have been interacting are lined up between them, giving the user the feeling that they are actually having a conversation on the chat screen S2. In addition, past chat history can be checked later, so chat can be used as a substitute for a diary. Also, since it is possible to see who has had what conversation, it is possible to check data such as which family members use the chat most frequently. The "user change" display indicates that the conversation has been interrupted, so there is no confusion when reading past history.

[0104] D. Special gestures in normal conversation In normal conversations, the cloud server checks whether certain words are included in the user's speech data. If it is determined that the user is in a certain state, the controller MC responds with a command together with character string data to execute a special gesture. The controller MC executes specific gestures, for example, as follows, based on the above-mentioned screen display program, gesture program, and dialogue program in response to the command. The following control is an example, and control may be performed to result in other gestures, or if there are multiple commands during the user's speech, control may be performed to execute the gestures consecutively or simultaneously. The following special gestures may be performed separately or in combination. Instead of the gestures of the robot 1 in (i) of "6. Gestures of the robot 1 during dialogue" above, the following displays may be displayed on the display unit, or the displays on the display unit may be appropriately combined.

[0105] 1) When a negative expression is uttered by the user in a normal human conversation, an animation is displayed on the touch panel 7, in which the normal eyes shown in Fig. 9(a) change to an angry eye object shown in Fig. 9(b) at the same time as the robot 1 speaks. In this embodiment, the eye object is stored in memory. 2) When the user utters an expression that would be pleasant in a normal human conversation, an animation is displayed on the touch panel 7, in which the normal eyes shown in Fig. 9(a) change to a smiling eye object shown in Fig. 9(c) at the same time as the robot 1 speaks. In this embodiment, the eye object is stored in memory. 3) When a user expresses a sadness in a normal human conversation, an animation is displayed on the touch panel 7, in which the normal eyes shown in Fig. 9(a) change to a sad-looking eye object shown in Fig. 9(d) at the same time as the robot 1 speaks. In this embodiment, the eye object is stored in memory. 4) When the user speaks the name of the user's child, the second motor 24 is controlled so that the head 5 of the robot 1 makes a gesture of nodding. In this embodiment, a program for eye gestures is stored in the memory. 5) When the name of the company that manufactures the robot 1 is spoken by the user, the first to third motors 23 to 25 are controlled so that the head 5 of the robot 1 moves back and forth and left and right while the torso 4 performs a gesture movement in which it repeatedly swings against the fixed part 2. At the same time, the text data of the company name is synthesized into voice and the company name is repeatedly spoken as voice from the speaker device 13. In this embodiment, the gesture program is stored in memory. "6. Effects of Robot 1's behavior during conversation, Part 6" By performing these kinds of special actions, the user can look forward to unexpected actions from the robot 1 while interacting with it, and can actively enjoy interacting with the robot 1.

[0106] B. Operation during face recognition 1. Starting and Stopping Face Recognition Mode The controller MC recognizes and authenticates the user's face by executing a face recognition program. The face recognition program recognizes the captured image as a human face by performing face pattern recognition, and performs authentication by digitizing and storing various positions of the recognized face and determining the degree of match with previously registered numerical data of the face. The controller MC synchronizes with the natural conversation mode to enter face recognition mode, and executes face authentication every time the device switches from standby mode to natural conversation mode.

[0107] In the face recognition mode, the controller MC uses the face recognition camera 10 to recognize the face of the user. Specifically, 1) The face recognition camera 10 is started. The face image of the user captured by the face recognition camera 10 is displayed on the touch panel unit 7 (face display mode). In other words, the user is prompted to look at their own face on the touch panel unit 7. This enables face recognition processing, and by previewing in this way, it can be determined whether the person has been authenticated in the past. 2) In the case where the face recognition camera 10 is unable to capture a face in step 1 and is unable to recognize the face within a certain period of time, In this case, the third motor 25 is driven to swing the head 5 up and down. In other words, the face recognition camera 10 is made to scan in the vertical direction. Then, while the face recognition camera 10 is being made to scan in the vertical direction in this manner, the first motor 23 is driven to rotate the face recognition camera 10 360 degrees in a full circle to perform face recognition. 3) If the face is recognized in 1) or 2), authentication is performed. If the user is already registered, the data of a specific authenticated user is used in the dialogue and the above-mentioned natural dialogue mode is entered. If the user is not registered, the user is recognized as an unspecified person and the above-mentioned natural dialogue mode is entered. The touch panel unit 7 returns from the face display mode to either the previous face screen S1 (Fig. 5) or the chat screen S2 (Fig. 6). 4) If face recognition is not possible in 2), it is determined that the person could not be recognized, and the face recognition mode and the natural conversation mode itself are terminated and the standby mode is entered. The touch panel unit 7 returns from the face display mode to either the standby screen of the face screen S1 (FIG. 7) or the standby screen of the chat screen S2 (FIG. 8), which was the standby mode immediately before. "1. Effect of starting and stopping face recognition mode" Since a conversation is based on looking at the other person's face while speaking, the system does not allow a conversation if the face cannot be recognized, so the user is actively forced to recognize the face. As a result, a conversation cannot take place unless the user is actually facing the robot 1, which gives the user the feeling that they are actually having a conversation.

[0108] 2. Robot 1's behavior during face recognition 1) After face recognition, the controller MC causes the face recognition camera 10 to acquire an image and constantly executes face pattern recognition at a fixed timing. The face of the user is recognized within the angle of view of the face recognition camera 10, and a state in which the center position C of the two eyes of the user's face is at a predetermined position within the angle of view, for example, the origin of the center of the angle of view, is set as the default position. When the center position C deviates from this default position, the controller MC displays an animation in which the pupil object 27a moves in either the left or right direction according to the amount of deviation. Normally, the pupil object 27a appears as an oval shape in its entirety within the white of the eye in the eye object 27 as shown in FIG. 9(a), but in a certain state in which the user's face is moving, the pupil object 27a is displayed as a partially hidden eye object 27 as if it is looking in that direction, as shown in FIG. 11. If the user moves and the face goes out of the angle of view of the face recognition camera 10, making face recognition impossible, the controller MC controls the first motor 23 to drive and rotate the entire movable part 3 in the direction in which the central position C has shifted, so as to point the face recognition camera 10 in that direction. When face recognition is achieved, the driving of the first motor 23 is stopped. If face recognition is not achieved even after a certain degree of rotation, for example, even after the entire movable part 3 is rotated 45 degrees, the controller MC stops the driving of the first motor 23 at that stage and continues face pattern recognition at all times in that state. "2. Effects of robot 1's behavior when recognizing faces" This gives the user the feeling that he or she is always being watched by Robot 1 during the conversation, making the conversation more interesting.

[0109] 2) The controller MC constantly performs face pattern recognition using the face recognition camera 10, but face recognition is not possible when the user is moving or not within the angle of view of the face recognition camera 10. For this reason, it is preferable to notify the user of the face recognition state as a change on the screen. In the first embodiment, the face recognition state is notified by a change in the color density of the sickle-shaped reflection object 27b, which represents the reflection on the pupil inside the pupil object 27a. In the first embodiment, the controller MC displays the face not being recognized in a very light blue color, displays it in a darker color when the face is being recognized, and displays it in a dark blue color when the face is being recognized normally. "2. Effect of Robot 1's behavior when recognizing faces, Part 2" This allows the user to easily know whether or not the robot 1 has recognized his or her face, and the user will actively cooperate in having the robot 1 recognize his or her face, which will lead to smooth dialogue. become.

[0110] C. Operation when the answering machine is set to "Away" In the first embodiment, an image can be transferred to an e-mail registered in the away setting mode, that is, when the away setting mode is activated. In the first embodiment, the away setting mode is activated and deactivated on a setting screen that is displayed after the user touches the setting button object 35 on the chat screen S2 of the robot 1. The following describes the process based on the away setting program of the controller MC when the away setting mode is activated. When the controller MC determines that an object (person) is moving using the Doppler sensor 22 in the standby mode in B. 4) of "2. Starting and stopping natural conversation mode" in the absence setting mode, the controller MC performs the following control.

[0111] 1) When the controller MC detects the presence of a moving object around the robot 1, it notifies the user of this state by email. The controller MC sends an email containing the URL of the cloud server of the robot 1 to a terminal device, such as a smartphone, of a user (hereafter referred to as an external user) who is not near the robot 1 and whose email address is registered via an Internet line. The subject of the email or the text of the message contains an expression that makes the intention of the notification clear. For example, a sentence such as "It looks like someone is coming" or an icon that conveys this. "C. Effects of the operation when the away setting is enabled" As a result, the external user is notified by the e-mail that an object (person) is moving around the robot 1, and is then given an opportunity to take measures against the situation.

[0112] 2) The controller MC sends an email to the external user and at the same time activates the face recognition camera 10 to capture an image. 3) The controller MC judges whether or not there is a triggering utterance within a certain period of time while sending an e-mail to the external user. For example, a greeting such as "I'm back, Yupi-bo." Based on this utterance, a built-in scenario in the natural dialogue mode is started, and the user (here, a person near the robot 1 who uttered "I'm back, Yupi-bo") is prompted to perform face recognition. If the controller MC determines that the user is one of the registered users as a result of face recognition, it sends a second e-mail to the terminal device of the external user. This e-mail is information to the external user that the person around the robot 1 is not a suspicious person. In other words, the second e-mail is a notice that the person is a related person such as a family member. The name of the user registered based on the registration information is written as information in the subject or in the message of the e-mail. Note that a trigger other than the utterance, for example, a touch on the touch panel unit 7 for face recognition, may be used to confirm that the user is a registered user. "C. Effect of the operation when the away setting is set, Part 2" This means that if a family member, such as a child, returns home while you are away, you will be notified by this second e-mail, so there is no need to go out of your way to check on the status of the house via your smartphone while you are away.

[0113] 4) In 1), the external user who received the e-mail, especially if the second e-mail was not sent in 3), can enter the ID and password of the robot 1 to connect to the cloud server and view the camera image of the face recognition camera 10 provided by the cloud server in real time on the browser of the smartphone. This is possible even if the second e-mail is sent. FIG. 12 shows an example of a user's smartphone 41. After connecting to the cloud server, a predetermined camera image of the face recognition camera 10 is displayed on the display screen 43, which also serves as a touch panel. The camera image includes four buttons for remotely controlling the orientation of the face recognition camera 10. Operation icons 44a to 44d are displayed. By operating the operation icons 44a to 44d, a control command is output to the controller MC via the cloud server, and the first motor 23 or the third motor 25 is driven and controlled based on the control command, causing the head 5 and the torso 4 of the robot 1 to rotate, thereby changing the orientation of the face recognition camera 10. In addition, by touching the record button icon 45, recording can be started, and by touching it again, recording can be stopped. Also, voice data spoken into a microphone (not shown) of the smartphone 41 is output as audio from the speaker device 13 of the robot 1 via the cloud server, while voice data spoken from the microphone 12 of the robot 1 is output as audio from a speaker device (not shown) of the smartphone 41 via the cloud server. Therefore, an external user can converse with a user near the robot 1 using the smartphone 41 and the robot 1 while looking at an image from the face recognition camera 10. "C. Effect of the absence setting on the phone 3" This allows an external user to remotely change the direction of the face recognition camera 10 to check the situation around the robot 1, for example, to check the safety of the home when the user is not at home. Also, even if family members such as children come home while the user is away, the user can proactively contact the family members using a smartphone in this way, which contributes to good relationships with others, including family members.

[0114] <First Modification of First Embodiment> Next, a first modification of the first embodiment will be described. In a dialogue of the built-in scenario in the above-mentioned natural dialogue, if the user has poor pronunciation or other sounds are mixed in and the voice data acquired from the microphone 12 does not match the regular or non-regular expressions of the built-in scenario, the controller MC may output a voice such as "Please say it again" as an utterance to encourage the user to speak again without immediately determining that the dialogue is normal. The controller MC causes such a prompting utterance to be made from the speaker device 13, and if the user responds with a correct utterance in accordance with the built-in scenario within a certain period of time, it processes the utterance as a dialogue of the built-in scenario again. On the other hand, even in such a case, if the user's utterance does not match the regular expression or non-regular expression in the built-in scenario, it connects to an external cloud server. In this way, unnecessary connections to external cloud servers are avoided, and conversations can be carried out only within the robot 1.

[0115] <Modification 2 of the First Embodiment> Next, a second modification of the first embodiment will be described. In the above-mentioned natural dialogue, if the user misses a word (utterance) of the robot 1, the user may utter a request to repeat the previous utterance of the robot 1 within a certain period of time, causing the robot 1 to speak again (output voice). This process can be performed in either built-in scenario dialogue or natural dialogue. The controller MC performs voice recognition to determine whether there is an utterance that may trigger a missed utterance during the dialogue mode, such as an utterance such as "Say it again." If the controller MC determines that the user has requested the robot 1 to repeat an utterance after the robot 1 has uttered an utterance, it outputs the content of the robot 1's previous utterance again. The controller MC then cancels the previous utterance and processes the second utterance as the first utterance. In this way, even if the user misses something during the dialogue, the dialogue can be resumed without interruption.

[0116] <Third Modification of First Embodiment> Next, a third modification of the first embodiment will be described. In the above-mentioned natural dialogue, if the user checks the user's utterance displayed on the touch panel unit 7 by the robot 1 and realizes that the voice recognition has been incorrect, the user may be allowed to point this out and correct the dialogue within a certain period of time. This processing can be performed for both built-in scenario dialogue and natural dialogue. In the dialogue mode, the controller MC performs voice recognition to determine whether the user has made an utterance that triggers a notice that the voice recognition is incorrect, such as "You're wrong" or "That's not right. I'll say it again." If the controller MC determines that the utterance occurred within a certain period of time after the user's utterance was displayed on the touch panel unit 7, it outputs a voice to encourage the user to speak again. For example, it outputs the utterance content, "I'm sorry. Say it again." and, (1) In the case of a dialogue using a built-in scenario, the user's previous utterance is canceled, and the next utterance spoken by the user is processed as the correct utterance through speech recognition. (2) In a normal dialogue, the dialogue content such as "You're wrong" or "I'll say it again" is also sent to an external cloud server as speech data (audio data), and a request is made to create response data using the content spoken by the user again, including such utterances. In this way, even if the robot 1 mistakenly recognizes the user's utterance, the robot 1 can correct the dialogue to a correct one.

[0117] <Fourth Modification of First Embodiment> For example, the following configuration may be adopted, which is different from the configuration of the first embodiment. (1) In the above, if the built-in scenario is not executed, a request is made to the cloud server and the normal dialogue is started. In other words, if the built-in scenario is executed, it is set to be the built-in scenario. However, even if the built-in scenario is executed, it may be possible to request the cloud server under certain conditions without responding locally. For example, certain conditions may be executed once every few times or at random timing. This allows for unpredictable dialogue with the robot 1, making it possible to have less predictable, more human-like dialogue. (2) In the above-mentioned embodiment 1, the controller MC does not have a voice recognition engine, but connects to an external server equipped with a voice recognition engine to convert the user's speech (voice data) into text, thereby reducing the burden on the robot 1. However, the controller MC may have a voice recognition engine in the memory, and may create character string data by calling the voice recognition engine and converting the user's speech data (voice data) acquired from the microphone 12 into text using the voice recognition engine. In other words, the controller MC of the robot 1 may have the ability to convert the user's voice data into text data by itself. This makes it possible to create character string data without using a voice recognition engine, thereby shortening the internal processing time, for example. (3) In the above embodiment 1, the initial settings are made from a setting screen, but it is also possible to register from an external terminal device such as a smartphone via a cloud server. This makes the settings easier and saves time, especially for those who are familiar with using terminal devices. (4) Driving means other than servo motors may be used as the first to third motors 23 to 25. The other driving means may be, for example, other types of motors or hydraulic cylinders. (5) In the above-mentioned first embodiment, character string data is arranged to be voice-synthesized inside the robot 1. Using an internal voice synthesis engine in this way prevents data from becoming too heavy compared to exchanging voice data directly with the server, but it is also possible to use a dialogue engine on the cloud server side to voice-synthesize acquired response data (character string data) and send the voice data as a response to the robot 1. (6) In the above-described first embodiment, the setting screen is switched from the setting button object 35 on the chat screen S2. However, the setting screen can be switched by sliding the touch panel unit 7. It may also be possible to transition to a surface.

[0118] (7) In "C. Special gestures in normal dialogue," a special gesture may be executed in the built-in scenario when it is determined that specific words are included in the user's speech data. For example, the controller MC may control the special gesture to be executed in the same manner as above when it is determined that specific words are included in the user's speech data. (8) In "C. Special gestures in normal conversation", if the shape of the robot 1 is different, the controller MC may control the robot 1 to make a different gesture. For example, if the robot 1 has hands or feet, the controller MC may control the driving means to move them. (9) As an animation of the eye object 27 in the face screen S1, an animation such as blinking may be inserted from time to time. For example, this is executed by controlling the eye object 27 in the state of Fig. 9(a) to be inserted into the closed state of Fig. 7. In this way, a sense of realism is created as if the robot 1 is really looking at you, making the interaction with the robot 1 more enjoyable. (10) In face recognition mode, unregistered users are recognized as unknown persons, but it is convenient to switch from that state to the settings screen and register them as a new user. (11) In “C. Operation when the Away Mode is Set,” it would be convenient to be able to set the Away Mode from a smartphone. (12) In "C. Operation when the away mode is set," the controller MC performs processing to notify the user of the presence of any moving object around the robot 1 by e-mail. However, conversely, processing may be provided to recognize movement at regular intervals, and if no moving object is detected within a certain period of time, to notify the user of this condition by e-mail. For example, if there is a sick person or a person requiring care, placing the robot 1 near the sick person or person allows the robot to keep an eye on the person, assuming that the person is constantly moving.

[0119] <Embodiment 2> Next, a second embodiment will be described. Instead of the high-brightness white LED 9 of the robot 1 of the first embodiment, or in addition to it, an infrared LED may be provided. In this case, if an infrared filter is provided in the module of the face recognition camera 10, it is removed. Since the infrared LED is invisible to humans, if there is an intruder such as a burglar at night, the high-brightness white LED 9 may light up and startle the intruder, causing him to run away. On the other hand, if an infrared LED is used, it is difficult for the intruder to know that he is being photographed, so he will not run away, and it becomes possible to confirm and save the image of the intrusion.

[0120] <First Modification of the Second Embodiment> Next, a first modification of the second embodiment will be described. When the robot 1 is equipped with an infrared LED, the infrared LED may be used to control various devices in the room that are equipped with an infrared remote control signal receiver, such as a television, audio equipment, air conditioner, etc. It is preferable to install the robot 1 in a location where it has a direct line of sight to the infrared remote control signal receiver of a device equipped with an infrared remote control signal receiver. In the first variation of the second embodiment, the controller MC is equipped with a shape recognition program for shape recognition using the face recognition camera 10, and for example, a television can be recognized based on its shape characteristics (square, large, black, etc.). When the user utters something like "Turn on the television" as a trigger for outputting an infrared remote control signal to various devices, the robot 1 controls the first motor 23 or the third motor 25 based on the utterance to move the robot 1 up the face recognition camera 10. The face 5 is moved downwards and rotated 360 degrees to capture images of the surroundings, and the shape of the television is recognized by a shape recognition program. When the controller MC determines that there is a television, it memorizes its direction and simultaneously outputs an infrared remote control signal from the infrared LED in the direction of the object, executing control such as ON / OFF to operate the television. When the next trigger for the television occurs, it first performs shape recognition in that direction. The infrared remote control signal is not just an ON / OFF switching control, but can also change the infrared frequency to, for example, change the channel on a television or adjust the temperature on an air conditioner. Such detailed control requires infrared remote control signals with multiple different frequencies, but the infrared frequency can be set via a server using, for example, the user's smartphone 41 in Figure 12. Also, instead of searching for the direction using a shape recognition program, the direction of various devices may be obtained by operating the face recognition camera 10 via the smartphone 41 to change its orientation, and the direction may be registered.

[0121] <Embodiment 3> Next, a third embodiment will be described. In the third embodiment, a control method will be described that focuses on reducing the running costs associated with connections such as those for using a dialogue API when using a server equipped with a voice recognition engine. The dialogue program of the third embodiment also includes a subprogram capable of detecting a silence state of the voice acquired from the microphone 12. The dialogue program also includes a recording / output subprogram for acquiring voice data of the user's speech from the microphone 12, temporarily recording it, and outputting it to the server. The controller MC does not connect to the server immediately when an utterance is made, but first records the voice data of the user's utterance, and only when it detects a silence in the recorded voice data of the user's utterance does it connect to the server, output the recorded voice data, and create response data in the dialogue engine. In this way, it is not constantly connected to the server, and there is no need to connect to the server for long periods of time, including silence periods, so that the silent connection time can be cut. In the voice recognition engine, one process is dedicated to the user while the user is speaking. In other words, one process takes a very long time for a computer, measured as "several seconds", and as a result, the cost burden on the user who uses the voice recognition engine becomes large. However, by following the third embodiment, the processes required per user can be reduced, which contributes to reducing the user's costs.

[0122] <First Modification of Third Embodiment> Next, a first modification of the third embodiment will be described. In the first modification of the third embodiment, control is described that is aimed at reducing the running costs of the connection when using a server equipped with a voice recognition engine. If a time lag occurs before the user starts speaking, or if the user does not speak in the waiting state and the connection with the server is terminated due to a timeout, the voice recognition server will consume processes, which will incur costs for the user. The dialogue program of the robot 1 in the first modification of the third embodiment includes a recording / output subprogram for acquiring spoken voice data from the microphone 12, temporarily recording it, and outputting it to the server. The dialogue program also includes a subprogram for detecting the sound pressure level of the recorded voice data and outputting it. The controller MC connects to the server when the voice data being recorded continuously during natural conversation exceeds a certain sound pressure level, and plays back the recorded data in pursuit of the sound. In other words, it ignores silence or quiet speech that cannot be recognized, and plays back only speech that can be used for conversation. Only when this is the case will the system connect to the server, output voice data, and have the server create response data using a voice recognition engine. This makes it possible to eliminate wasted connection time while waiting to speak.

[0123] <Fourth embodiment> Next, a fourth embodiment will be described. Because voice recognition servers are expensive, it may not be possible to prepare sufficient resources in advance. If server resources are limited, the server may be busy when a terminal attempts to connect to the voice recognition server. In the normal dialogue described above, speech data (voice data) is sent to the server to request the creation of response data. The server then creates response data converted to character string data based on the speech data and responds. However, if the server is busy, the response data may not be sent, resulting in an error. An error message will be returned from the server. When the controller MC of Robot 1 receives an error message from the server, it will not tell the user that there is a server connection error, but will output a reply from the built-in scenario that allows the user to continue the conversation. For example, it could respond with a vague reply such as "Say it again," "Uh-huh," or "What was it?", or give some other appropriate response, and wait for the server to become available.

[0124] <Modification 1 of the fourth embodiment> Next, a first modification of the fourth embodiment will be described. If the user's speech is too long, the speech recognition engine may make a mistake in its recognition. Therefore, it is preferable to provide a function for outputting a dialogue example selected from the speech data stored in the storage means without connecting to a server equipped with a speech recognition engine when it is determined that the recognized user's speech is too long. When the controller MC of the robot 1 judges that the user's speech data has reached a certain length, it outputs speech such as "Yes," "Really?", "Is that true?", or other responses that can be interpreted in any dialogue from a built-in scenario without connecting to a server. The speech data may be left as voice data, or may be converted into character string data by the controller MC or the server. This prevents irrelevant responses and resets the conversation, encouraging the user to engage again.

[0125] <Embodiment 5> Next, a fifth embodiment will be described. In the fifth embodiment, a case where a plurality of voice recognition engines are used in combination will be described. There are two types of voice recognition engines: local (i.e., processing takes place within the device without connecting to an internet server) and cloud voice recognition engines that connect to an internet server and respond with dialogue data created in response to a request. There are multiple types of local and cloud voice recognition engines, some free and some paid. Therefore, when using servers equipped with these different voice recognition engines, it is recommended to mix free and paid servers. In a cloud server to which the robot 1 is connected via an Internet line, in dialogue mode, when the robot 1 issues a request for speech data and has the voice recognition engine create response data, the cloud server may respond, for example, as follows: (1) Access a paid dialogue API up to a set time period A per month. (2) From a certain time A to a certain time B per month, a paid API and a free voice recognition engine are used together. For example, the first few consecutive recognitions are done on the paid voice recognition engine server. We use a mixed method of using the voice recognition engine for subsequent recognition, such as using a free voice recognition engine server. (3) When a certain period B per month is exceeded, only the free voice recognition engine is used. This is just one example, and it may be set so that when a certain period A per month is exceeded, only the free voice recognition engine is used immediately. In this way, it is possible to have a conversation without going beyond the paid range. The cloud server connected to the robot 1 calculates whether the conversation is paid or free based on a set time per month based on a program that executes such processing, and processes requests issued by the robot 1. The robot 1 itself may execute such processing and instruct the cloud server whether to use a paid dialogue API or a voice recognition engine on a free server when issuing a request. <Embodiment 5-1> The same applies to the case where a plurality of speech recognition engines are combined and used in the fifth embodiment as well as the case where a plurality of dialogue engines are combined.

[0126] <Sixth embodiment> Next, a sixth embodiment will be described. If only one fixed dialogue engine is used, the replies may become fixed and the user may become bored with the conversation with the robot 1. To solve this problem, in the sixth embodiment, a process is described for using the output results of multiple dialogue engines to arrange the results so that the user does not become bored with the conversation. The dialogue engine may be not only a cloud dialogue engine, but also a local dialogue engine within the robot 1. This process may be performed on the robot 1 side to which multiple pieces of response data have been transmitted, or on the cloud server side when response data is received from multiple servers equipped with dialogue engines. (1) The result of the chat dialogue engine that provided the reply with the longest string of characters is output. The longest response makes the conversation feel more conversational, breaks the monotony, and allows the user to enjoy the conversation. A. For example, if a user says "I'm hungry," and the responses from the three engines a through c are "Engine a: Do you often snack?", "Engine b: Haven't you eaten?", and "Engine c: Eat something," then Engine a will be used to output the response data. B. For example, when a user says, "The weather is sunny today," and the responses from the three engines a through c are, "Engine a: I'll go check the sky right now," "Engine b: Looks like it's sunny?," and "Engine c: Sometimes your mood for the day depends on whether it's sunny or rainy," engine c is used to output the response data.

[0127] (2) In the chat dialogue engine, any answer ending with a "?" is output last. If there are multiple answers ending with a "?", they are output consecutively. When a question mark is added to the end of a sentence, the conversation flows in a way that further answers the question, making it easier for the conversation to continue and allowing the user to enjoy the dialogue. For example, for the option (1)A above, it will output "Do you often snack? Have you not eaten?". Also, for the option (1)B above, it will output "Looks like it's sunny?". (3) After combining affirmative sentences, combine and output questions. By arranging the responses in this way, the user feels as if the other person is listening to what he or she is saying seriously, which makes the user want to continue the conversation. This makes it easier for the conversation to continue, and the user can enjoy the dialogue. In addition, it is possible to save time and to ask for a response from a person. For example, if response data like (1)B above is obtained, the output would be "I'll go and check the sky right now. Whether it's sunny or rainy can sometimes determine your mood for the day. Does it look like it's going to be sunny?"

[0128] (4) Results from engines that change topics more frequently than others are output after results from other engines. By arranging it in this way, the change of topic creates a dialogue that invites the next utterance, making it easier to continue the conversation. A. For example, if a user says "Hello everyone," and the responses from the three engines a through c are "Engine a: Yeah, right," "Engine b: That's right," and "Engine c: Do you watch baseball?", the data from Engine c will be output last, as "Yeah, right, do you watch baseball?" B. For example, if a user says "It's hard to find," and the answers from the three engines a through c are "Engine a: That's right," "Engine b: That's true," and "Engine c: How many people are there in your family?", the data from Engine c will be output last, with "That's right, that's true. How many people are there in your family?" output. (5) The engine that gives the friendliest response is output first, followed by the responses from the other engines. By arranging in this way, users are more likely to be drawn into the dialogue and the dialogue is more likely to continue. Whether friendly or not can be determined by ranking relative to words (words), and where to place them. A. For example, in the case of (4)B above, the result of engine b would be output first, and "That's right, that's right. How many people are there in your family?" would be output. B. For example, when a user says "I see," and the responses from two engines a, b are "Engine a: Oh, that's a suitable interjection," and "Engine b: Uh-huh," the data from Engine b will be output last, as "Hohoho, Oh, that's a suitable interjection, Uh-huh."

[0129] (6) Any combination of steps (1) to (5) above This increases the variety of dialogue, so that the same response will not be returned even when the user makes the same utterance, making it easier to continue the conversation without getting bored. For example, if a user says "What is old age?" and the answers from the three engines a to c are "Engine a: Please wait a moment," "Engine b: I don't dislike support," and "Engine c: Are you in good health now?", the most friendly engine b will be put first, and after combining a positive sentence, it will combine it with a question sentence and output "I don't dislike support Please wait a moment Are you in good health now?".

[0130] (7) In the case of text output engines, parentheses and emoticons may be returned, so when these are returned, the voice output is suppressed and the output is displayed on the screen. Although emoticons cannot be output as voice, by deliberately displaying them on the display and making them part of the conversation along with the voice, it is possible to create an interesting and unusual conversation. A. For example, if a user says "I'm hungry," and the responses from the three engines a to c are "Engine a: (intentionally ignored)," "Engine b: Hello, how are you," and "Engine c: Hello," then only Engine a will not output voice, but will display the message on the touch panel unit 7 (display screen). B. For example, if the user says "It's already included," the three engines a to c will In the case where the answer from the engine is "Engine A: Do you do that often? Engine B: (´·ω·`)" and "Engine C: Are you a night owl?", only Engine B will not be output as voice, but will be displayed on the touch panel unit 7 (display screen). (8) For responses that contain the same string, output one of them. This is because repeating the same string of characters makes the dialogue repetitive and makes the listener feel uncomfortable. For example, if a user says "Chinese food," and the answers from the three engines a to c are "Engine a: Oh, it's nice," "Engine b: Yes, it's Chinese," and "Engine c: Are we going to eat Chinese food?", since both engine b and engine c contain the character string "Chinese food," only one of them will be output. For example, it will output something like "Oh, it's nice. Are we going to eat Chinese food?"

[0131] (9) Use a word ending conversion method, such as a word ending conversion API, to create a sense of uniformity. In this case, word ending conversion may be performed on all replies, or only on the last sentence to be output or the first sentence to be output. This is good because it produces sentences that are more user-friendly than those produced by ordinary dialogue engines. For example, if an engine sends the response data "I'm sleepy," the ending will be converted using a word ending conversion API and output as "I'm sleepy, meow." (10) In preparation for recognition failure, only a portion of the responses obtained from multiple engines are used for voice output, and the remaining responses are stored. If the next voice recognition fails or there is no response from the dialogue system, the stored response is returned. By using this function when voice recognition fails or when a response from an external server is slow to come, the conversation can be continued without interruption, contributing to a more natural conversation.

[0132] <Embodiment 7> 14, the seventh embodiment is a device (system) in which a smart speaker 51 is placed near a robot 1 and the robot 1 and the smart speaker 51 are combined. The distance between the robot 1 and the smart speaker 51 is such that the microphones of the robot 1 and the smart speaker 51 can pick up the sounds of each other, and the robot 1 and the smart speaker 51 are preferably placed adjacent to each other within 1 to 2 m, for example. The smart speaker 51 is a network terminal equipped with a network module that has a built-in wireless LAN device, a wireless communication function using the Internet, a telephone line connection function, etc., and is also a kind of computer equipped with a microphone and speaker device. The smart speaker 51 uses a terminal device such as a smartphone to perform various initial registrations (e.g., registration of the user's name, address, telephone number, and email address, registration of multiple voices, setting of network-compatible AI devices via Bluetooth, etc.) via a server, connects to the Internet in response to utterances (commands) from the user who has registered the voice, executes a predetermined process using the server's search engine, and outputs the results as voice information from the speaker device.

[0133] The robot 1 can complement each other's functions by cooperating with the smart speaker 51. Specifically, the robot 1 and the smart speaker 51 achieve the following functions by using voice as an interface. (1) Instruction function from the robot 1 to the smart speaker 51 For example, when a user speaks a phrase that activates a smart speaker skill, the robot will memorize the character string that is the result of voice recognition of that phrase, and the robot will then speak that character string by itself at a specified timing using voice synthesis. In the seventh embodiment, the controller MC of the robot 1 is provided with a voice identification program for analyzing the frequency components of the user's speech to identify the individual's voice, and a recording and playback program for acquiring the user's speech by the voice recorder 12 for distinguishing between individuals, storing the speech as character string data, and synthesizing the speech based on the character string data to generate a phrase to be played from the speaker device 13. The robot 1 has a function of memorizing an utterance that triggers the memorization of an utterance, for example, the words following the utterance "I'm going to talk now, so please record it." The user can then use this function to memorize words that will activate the smart speaker 51 or cause it to perform some kind of processing. For example, words such as "OK, XXX. Turn on the lights." In this case, the individual voice of the user registered in the robot 1 is the individual voice of the user registered in the smart speaker 51. Then, the robot 1 is made to speak at a predetermined timing. The setting for making the robot 1 speak at a predetermined timing can be set and registered by operating a terminal such as a smartphone. B. An example of the "utterance at a predetermined timing" of the robot 1 is the robot 1 detecting some kind of change, such as a touch action on the touch panel unit 7 or the detection of an object (person) by the Doppler sensor 22. For example, when the controller MC of the robot 1 detects a person using the Doppler sensor 22, it outputs a voice from the speaker device 13, such as "OK, XXX. Turn on the lights." In response, the smart speaker 51 controls the indoor lights, which are network-compatible AI devices, to turn on. The lights to be controlled are registered in advance as objects to be controlled by the smart speaker 51. In addition to lights, it is also good to use air conditioners, televisions, curtain opening and closing devices, etc. as AI devices.

[0134] (2) A function to distinguish between speech from the smart speaker 51 and speech from the user By identifying the individual voice of the user and setting and registering it in the robot 1, the robot 1 can be provided with a function of distinguishing whether the speech is coming from the smart speaker 51 or the user's direct speech. This makes it possible to control the robot 1 so that it does not react to sounds from the smart speaker 51, which is an unregistered voice, and conversely, by registering the individual voice of the user in both the smart speaker 51 and the robot 1 as in (1), instructions can be given to the smart speaker 51 not only from the user but also from the robot 1. Being able to distinguish between individual speech in this way also provides a function of not interfering with voice operation of the smart speaker 51. (3) Display function of speech from smart speaker 51 The robot 1 may acquire the contents of the utterance given by the user to the smart speaker 51, convert it into text, and display it on the touch panel unit 7. The controller MC of the robot 1 preferably has a program for acquiring the contents of the speech and converting it into text either by itself or by connecting to a server. For example, if the user says, "OK, XXX. Tell me the weather today," and the smart speaker 51 responds, "Today's weather in Okazaki City, Aichi Prefecture is sunny, with a maximum temperature of 15 degrees, and the probability of precipitation is 20%," then all of this dialogue between the two parties, for example, as heard by the robot 1, is displayed on the touch panel unit 7 of the robot 1 as follows: User: OK, XXX. What's the weather like today? Smart speaker: Today's weather in Okazaki City, Aichi Prefecture is sunny, with a maximum temperature of 15 degrees and a 20% chance of rain. In addition, the controller MC of the robot 1 may have a program for adapting or summarizing the textual content in a short form. The robot 1 may output the adapted or summarized content as voice or display it on the touch panel unit 7. (4) A function in which the robot 1 listens to various information from the smart speaker 51 and learns from it when no one is around. For example, the robot 1 has question words such as "What's the weather tomorrow?" or "Has anything happened?" as built-in scenarios, and when the user is away, the controller MC of the robot 1 outputs the question words by voice together with a phrase that activates the smart speaker 51 at a predetermined timing (the voice of the robot 1 is assumed to be registered in the smart speaker 51). At this time, the controller MC acquires and stores the contents of the speech from the smart speaker 51 by the microphone 12 of the robot 1, and then outputs the contents by voice at a predetermined timing. The predetermined timing is, for example, when a person is detected by the Doppler sensor 22 for a predetermined period of time, or when the user speaks to the robot 1 as a built-in scenario such as "Is there any news?"

[0135] <First Modification of the Seventh Preferred Embodiment> A function may be set that does not interfere with the voice operation of other voice recognition devices such as a smart speaker. For example, when the robot 1 recognizes the voice of the activation phrase (voice recognition start word) of another smart speaker, the robot 1 may stop outputting its own voice. For example, when the controller MC of the robot 1 recognizes the voice of a startup phrase such as "OK, XXX", which is a startup phrase of another smart speaker, Company A, it acquires it from the microphone 12, and if it determines that it is a registered startup phrase, it temporarily stops its own voice output. This allows the voice recognition device to function without interfering with voice operations.

[0136] <Modification 2 of the Seventh Embodiment> The robot 1 recognizes a voice recognition activation keyword for another voice recognition device such as a smart speaker 51, recognizes the user's subsequent pronunciation to the smart speaker 51, makes a request to a cloud server, and has a search engine or the like perform a search to obtain a corresponding answer. When the smart speaker 51 fails to recognize the voice (for example, "Error") or when it recognizes a voice output indicating that it cannot output an appropriate response to the voice recognition result (for example, "Sorry"), the robot 1 outputs a response that it has obtained in advance. Alternatively, after receiving a voice output indicating that it has failed to recognize the voice or that it cannot output an appropriate response to the voice recognition result, the robot 1 may make a request to the cloud server to obtain a response.

[0137] <Embodiment 8> For example, the robot 1 may read out a news article on a website in response to a user request. The controller MC of the robot 1 requests news articles using a search engine on the server, and the cloud server responds to the request with, for example, news data from a registered website as text data. The robot 1 synthesizes voice to read out (output) news data, and at the same time synthesizes voice to read out (output) the name of the information source of the article. In addition, the URL of the information source of the article is displayed on the touch panel unit 7 as a display screen, and when the URL is touched on the touch panel unit 7, the contents of the page of the URL are displayed on the touch panel unit 7. In addition, when reading out articles or their sources, it is advisable to output them in a way that makes it clear that they are quotations. For example, a case where the content of an article at the URL "https: / / \\\\.jp / archives / 92###" is read out will be described. "I'm going to read you an article from the "XX News" website. "... will go on sale from January 15th of this year." That's right." In this way, for example, "I will read out an article from the website of..." or "That's right" are spoken so that the listener can understand, as if quoting an article or an information source of the article as a regular expression, and in this case, a URL is displayed on the screen, for example, "The information is from https: / / \\\\.jp / archives / 92###." Then, by touching this URL portion, the content of the article that was read out is displayed again on the touch panel unit 7. "Speaking so that the listener can understand" means, for example, changing the tone or voice between article sections and other sections. In this way, you can understand what Robot 1 is reading without having to read the web article. You can understand the news content just by reading the text, and in some cases you can also check the news content visually just to be sure.

[0138] <Embodiment 9> In the ninth embodiment, a case will be described in which invisible movements of a computer are shown by the operation of a robot's actuators. (1) About Robotics Process Automation Robotics Process Automation (hereinafter referred to as RPA) is software that automates simple PC tasks. The software can be set on a server or on a user's computer. An example of a case where the software is set on a server based on FIG. 15 and the robot 1 of each of the above-mentioned embodiments is incorporated into an RPA network will be described. As shown in Fig. 15, a cloud server 55 and user-side computers 56 and 57 are connected to a network using the Internet. The cloud server 55 and the robot 1 are also connected to the network. The user-side computer 56 is controlled by the cloud server 55 using an RPA program. The robot 1 is notified of processing information such as that a predetermined process in the program for RPA executed by the cloud server 55 will soon be executed, is being executed, or has been executed. Cloud server 55 causes user-side computer 56 to execute processes 1 to 4 in order. In this embodiment, processes 1 and 2 are executed by computer 56, and processes 3 and 4 are executed by computer 57. More processes may be set, and the number of user-side computers 56 involved in the processes may be one or more. An example of process 1 is user access to computer 56 and login, an example of process 2 is list creation and sorting based on data in computer 56, process 3 is modification of billing details for each customer executed following process 2, and process 4 is issuance of an invoice executed following process 3. In this embodiment 9, for example, process 3 prompts the user to input corrections, and after the input, the process moves to the next process 4. In other words, after process 2, the process goes into a standby mode until the input in process 3 is completed. The cloud server 55 may output different notification information to the robot 1 immediately before, during, and after each of these processes, and the robot 1 may notify the surroundings of the robot 1 of the processing status based on the notification information. Alternatively, a single notification may be made for each process.

[0139] for example, a. The robot 1 notifies the user of the status of each process by using voice, different sounds, music, etc. b. The notification is given on the display screen. This may be done simultaneously with a. c. Since process 3 requires user input, only process 3 may be notified, or only process 3 may be notified in a manner different (distinguishable) from other notifications. d. The processing status is transferred from the robot 1 to another terminal device and notified. e. The robot 1 behaves in such a way that the processing status can be understood. For example, if the computer 56 is processing, the robot 1 is controlled to face in that direction. For this reason, it is advisable to recognize in advance the directions of the computers 56, 57 relative to the robot 1 using some kind of direction specifying means, for example, the shape recognition program described above. The robot 1 may be provided with a direction indicating member, such as an arrow or arm member, and the robot 1 may be operated so that the computers 56, 57 as the notification targets are in the direction indicated by the member.

[0140] (2) About Blockchain Blockchain is a system in which many computers are distributed and record data. In particular, in public blockchains, the data to be recorded and the recorded data are made public. Therefore, we place Robot 1 in the blockchain network and have Robot 1 notify the user before block data is sent. It is a good idea to have Robot 1 output the command "wait" (i.e., request to wait without sending data) to stop the transmission.

[0141] <Embodiment 10> In each embodiment, the natural dialogue mode has been described, but a foreign language learning mode may be provided instead of or in addition to the natural dialogue mode. When the foreign language learning mode is provided in addition to the natural dialogue mode, for example, when a voice saying "switch to foreign language learning mode" is recognized in the natural dialogue mode, the mode may be switched from the natural dialogue mode to the foreign language learning mode. Also, when a voice saying "switch to natural dialogue mode" is recognized in the "foreign language learning mode", the mode may be switched from the foreign language learning mode to the natural dialogue mode. It is often said that the best way to learn a foreign language is to make friends with foreigners, but not many people have the opportunity to do so. In this case, it would be good to have a foreign language learning function that is a dialogue system that can become a partner in learning a foreign language. The link can be between any first language and any second language. In the following, a configuration in which a Japanese dialogue system and an English dialogue system are linked will be described. Basically, the system will converse in English, and output requests in Japanese such as "Say it again" during the conversation. Furthermore, it will have a function that works with English sentence analysis web services and will explain the English sentences being spoken in Japanese in response to requests such as "Explain." If you don't know how to say what you want to say in English during a conversation, you can request "translate it" and it will output what you want to say in English. You can continue the conversation by reading the output English sentence. Since there is no need to look it up yourself, the conversation will not be interrupted and you can expect smooth English conversation learning. In the foreign language learning mode, the voice recognition engine for the native language (for example, Japanese for a Japanese person) and the voice recognition engine for the foreign language (for example, English) may both use voice recognition engines running on the cloud, but for the native language (for example, Japanese), since the request content is a standard phrase and the format of the response to the request is fixed, it is better to install a voice recognition engine locally (for example, in the robot 1) and use this. The same goes for the dialogue engine. The voice synthesis engine may also be installed anywhere, but it is particularly good to install it locally. When voice data based on the signal from the microphone 12 is sent to the voice recognition engines of both languages, some kind of result will be returned from each of the voice recognition engines. For example, if the Japanese phrase "Say it again" is sent to both engines, the Japanese engine will return text data "Say it again," and the English engine will return nonsensical text data interpreting "Say it again" as English. In such a case, since the Japanese request is a fixed phrase, it is advisable to compare the text data returned by the Japanese engine with the fixed phrase of the request, and if they match, determine that a Japanese request has been made, and if they do not match, determine that English has been spoken, thereby separating English from Japanese. Although requests may be accepted in English, the inventor has found that it is desirable to also be able to make requests in Japanese, particularly in anticipation of cases where a person wishes to make a request but does not understand English. The function of explaining English sentences in a conversation in Japanese in response to a request such as "explain," may simply translate the English sentence into Japanese and output it, but it is particularly desirable to output an explanation of the words and grammar used in the English sentence. It is particularly desirable to output an explanation of the syntax used in the English sentence. The explanation of the syntax may be, for example, drawn by drawing each phrase as a vertex (node) surrounding each phrase, and drawing branches (edges) such as lines showing the relationships between related phrases. For example, it is desirable to display on the touch panel unit 7 in the form of a graph structure (particularly a tree structure). In addition, the displayed contents may be output as voice from the speaker device 13. For example, the AP of a syntax analysis service such as that described in https: / / gigazine.net / news / 20160602-foxtype-review / may be It would be a good idea to configure it so that it calls I, receives the result, and outputs the analysis result in Japanese. For example, the following processing and output is performed. Processing: Get phrases from the English dialogue engine Robot I'm a fantastic robot. Person: "Say it again" Robot I'm a fantastic robot. Person: "Explain." Processing (recognizing "explain") → calling syntax analysis API → generating Japanese explanation from syntax analysis results Robot I is the subject, am is the verb, and robot is the object. Fantastic is an adjective that means wonderful and modifies robot. In English it means "I'm a great robot." Person: "I don't think so." Robot: Don't say it! If a user does not know what he or she wants to say in English, he or she can request in Japanese to be told how to say it in English, which has the excellent effect of preventing interruptions in the conversation. If the user is unable to make requests in Japanese and does not know how to say something in English, he or she will have to look it up in a dictionary or translate it online, which makes it feel like studying and stressful. If the user can make requests in Japanese, the user can learn without stress, as if they were simply conversing with a bilingual person. Since continuity is very important in language learning, it is extremely important to minimize stress when learning in order to continue. With this configuration, a robot 1 that allows for continuous language learning can be realized.

[0142] <Embodiment 11> In addition to the functions described in each embodiment, it is preferable to provide a function for the robot 1 to speak when it detects that a person has entered the room in which the robot 1 is installed, and a function for the robot 1 to speak when it detects that a person has left the room in which the robot 1 is installed. For example, sensors may be provided in the room and in the passages outside the room to detect when a person enters the room where the robot 1 is installed and when a person leaves the room where the robot 1 is installed. In particular, if there is an automatic door for entering and exiting the room where the robot 1 is installed, a sensor that detects when a person approaches the automatic door to open or close the automatic door may be used. In particular, the controller MC of the robot 1 may be connected to at least two of a first human detection sensor located outside the room across the automatic door, a second human detection sensor located inside the room across the automatic door, and a third human detection sensor (a sensor to prevent a person from being caught in the door) that detects a person at the automatic door when the automatic door is open, to detect when a person enters or leaves the room. In this way, it is possible to detect when a person enters or leaves the room where the robot 1 is installed without installing a new sensor. For example, it is possible to detect by capturing the edge of a signal that rises when each sensor detects a person. For example, if a person is detected by the first human detection sensor and then by the third human detection sensor, a voice phrase such as "Welcome" to welcome the person entering may be output from the speaker device 13. For example, if a person is detected by the second human detection sensor and then by the third human detection sensor, a voice phrase such as "Thank you" to thank the person leaving may be output from the speaker device 13. At these times, the first to third motors 23-25 ​​can be operated to turn the robot 1 from its installation position toward a preset automatic door. If the first and second human detection sensors detect a person at the same time, the first human detection sensor can be given priority. In this way, it becomes easier for the person entering to notice the presence of the robot 1, and it is possible to reduce the awkwardness that the person feels when the robot 1 suddenly says "Thank you."

[0143] <Other embodiments> (1) In each embodiment, the robot 1 is provided with the wireless LAN device 21, but instead of or in addition to the wireless LAN device, a wired LAN device may be provided and connected to a wired LAN network. The wired LAN device may be built into the robot 1 or may be external. The wired LAN device may be connected to a terminal 18 for USB OTG (On-The-Go). The wired LAN network may be connected to the Internet via a router or the like. Depending on the environment, wireless LAN may not provide stable communication or may not be able to connect. For example, when the robot 1 is installed in a location where there are many wireless LAN devices such as smartphones, it is preferable to access the Internet via a wired LAN device. (2) In each embodiment, an example of a dialogue between a person and the robot 1 in a half-duplex system is shown, but a dialogue between a person and the robot 1 in a full-duplex system may be performed. For example, in the dialogue in Table 1, after the person says "Open...", the robot may continue to recognize the human voice while saying "Are you sure?", and if the robot recognizes the voice "Cancel" while saying "Are you sure?", it may interrupt the speech if it is in progress and immediately say "Cancelled." (3) A configuration in which conversation is performed in a half-duplex manner is particularly good because it can simplify the configuration and processing and reduce costs. However, the inventors have found a problem that it is difficult to know when the robot 1 turns on the microphone 12, and even if the user speaks, the beginning of the user's voice that the robot is to recognize, i.e., the beginning of a word, is often missing. To solve this problem, it is good to display a characteristic screen when the controller MC turns on the microphone. This supports smooth conversation. As a mode for displaying a characteristic screen when the controller MC turns on the microphone, it is good to perform at least one of 1) lighting the four corners of the screen, or 2) displaying a microphone icon, and particularly good effects are obtained when both 1) and 2) are performed. (4) The motors constituting the first to third motors 23-25 ​​can be various motors such as DC motors, but it is particularly preferable to use stepping motors, and the robot 1 is configured to control its posture using stepping motors. The torque of a stepping motor changes in proportion to the current passed through the motor. A large current can be passed through it to obtain a large torque, but heat generation and battery life become problems. Therefore, it is particularly preferable to pass the minimum current required to maintain the posture of the robot 1 when the robot 1 is stationary, and pass a large current only when the robot 1 changes its posture. It is also possible to pass the minimum current required to maintain the posture of the robot 1 when the robot 1 is stationary through all of the servo motors 23-25, but the first motor 23 of the body part, which can be supported by detent torque (torque when not energized), is not energized, while the second motor 24 and the third motor 25 of the head part are defeated by the detent torque, so it is preferable to pass this current even when the head part is stationary. Also, if you rotate the motor with the same torque as when it is stationary, the torque will be insufficient and step-out will occur, causing the motor to not rotate properly, so when rotating it is better to increase the torque by passing more current than when it is stationary.On the other hand, when it is stationary, a large torque like when it is rotating is not necessary, so it is better to lower the current compared to when it is rotating. Also, when changing direction, it is advisable to control it so that it accelerates up to a certain speed and then decelerates and stops when it approaches the desired angle. In addition, when a robot is driven by a motor, mechanical and electrical noise is generated. This noise reduces the recognition rate of voice recognition, so it is advisable to control the motor to stop during voice recognition. The scope of the present invention is not limited to the configurations expressly described in the specification, but includes combinations of various aspects of the present invention disclosed herein. The configurations of the present invention for which a patent is sought are specified in the appended claims, but it is the intention of the present inventors to include in the claims any configurations disclosed in this specification that are not currently specified in the claims. The present invention is not limited to the configurations described in the above-mentioned embodiments. The components of the modified examples may be arbitrarily selected and combined. Any components of each embodiment or modified example may be arbitrarily combined with any components described in the means for solving the invention or any components embodying any components described in the means for solving the invention. We intend to acquire rights to these as well through amendments to this application or divisional applications, etc. Furthermore, even if there is a description such as "in the case of..." or "when...", this is not intended to describe a configuration that is limited to that case or time. Configurations that are not these cases or times are also disclosed, and we intend to acquire rights to them. Furthermore, parts that are described in order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and we intend to acquire rights to them. In addition, we intend to obtain rights to the overall design or partial design by filing a conversion application to a design application. The drawings depict the entire device in solid lines, but they include not only the overall design but also partial designs claimed for some parts of the device. For example, it is a drawing that includes some parts of the device as partial designs, regardless of the parts, as well as a drawing that includes some parts of the device as partial designs. A part of the device may be a part of the device, or it may be a part of that part. We intend to obtain rights to not only the overall design, but also partial designs in which any part of the solid line parts of the drawings is drawn as a dashed line part. [Explanation of symbols]

[0144] 1...Robots as devices, 41...Smartphones as other devices, 51...Smart speakers as other devices.

Claims

1. A function of displaying a face image on the display unit; a function of transitioning from a standby mode in which a dialogue with a user is not possible to the standby mode to a dialogue mode in which a dialogue with a user is possible when the display unit is touched while a face image is displayed; An apparatus comprising:

2. When the display unit is touched and slid while a face image is being displayed in the standby mode, a standby image different from the face image is displayed instead of the face image while remaining in the standby mode.

2. The apparatus of claim 1.

3. The device has a function of transitioning to the interactive mode when any area of ​​the display unit is touched while the face image is being displayed in the standby mode, and transitioning to the interactive mode when an area where a predetermined object is being displayed is touched while the standby image is being displayed in the standby mode.

3. Apparatus according to claim 1 or 2.

4. The ability to recognize user utterances, a function of changing an image of a face in response to a speech volume of a user when the image of the face is displayed on the display unit; The apparatus according to any one of claims 1 to 3, comprising:

5. The function of changing the face image is a function of enlarging a specific area when the user's speech volume increases, and shrinking the specific area when the user's speech volume decreases.

5. The apparatus of claim 4.

6. The function of outputting audio; a function of changing an image of a face in accordance with a volume of the sound to be output when the image of the face is displayed on the display unit; 6. An apparatus according to any one of claims 1 to 5, comprising:

7. The function of changing the face image is a function of enlarging a specific area when the volume of the output voice increases, and shrinking the specific area when the volume of the output voice decreases.

7. The apparatus of claim 6.

8. A function of starting to display the predetermined character string while the face image is displayed on the display unit; a function of changing the transparency of the face image in response to the display of the predetermined character string; 8. An apparatus according to any one of claims 1 to 7, comprising:

9. A program for causing a computer to realize the functions of the device according to any one of claims 1 to 8.

Images