Action control system and information processing system

Abstract

The present invention causes an avatar to perform an appropriate action in response to a user's action.　In this action control system, the actions of an avatar include dreaming, and when an action determination unit determines dreaming as the avatar's action, the action determination unit creates an original event by combining a plurality of sets of event data included in history data.

Description

Behavior control system and information processing system

[0001] The present disclosure relates to a behavior control system and an information processing system.

[0002] Patent Literature 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Literature 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

[0003] Patent document 2 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

[0004] Japanese Patent No. 6053847 Japanese Patent Application Laid-Open No. 2022-180282

[0005] However, the conventional technology has room for improvement in terms of making the robot perform appropriate actions in response to the user's actions.

[0006] Furthermore, when an earthquake early warning is issued, the only information available in television studios is the seismic intensity, magnitude, and depth of the epicenter. As a result, announcers have no choice but to announce predetermined messages to viewers, such as "Just in case, be careful of tsunamis. Do not go near cliffs. I repeat," making it difficult for viewers to take measures against earthquakes.

[0007] According to a first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes dreaming, and when the behavior determination unit determines that the behavior of the avatar is dreaming, it creates an original event combining a plurality of event data from the history data.

[0008] According to a second aspect of the present disclosure, the behavior determination model is a data generation model capable of generating data according to input data, and the behavior determination unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0009] According to a third aspect of the present disclosure, when the behavior decision unit determines that the avatar's behavior is to dream, it causes the behavior control unit to control the avatar to generate the original event.

[0010] According to a fourth aspect of the present disclosure, the electronic device is a headset type terminal.

[0011] According to a fifth aspect of the present disclosure, the electronic device is a glasses-type terminal.

[0012] According to a sixth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting an activity, and when the behavior determination unit determines to suggest the activity as the behavior of the avatar, determines the user behavior to suggest based on the event data.

[0013] According to a seventh aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes comforting the user, and when the behavior determination unit determines comforting the user as the behavior of the avatar, it determines utterance content corresponding to the user state and the user emotion. Here, the electronic device may be a robot, and a robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.

[0014] According to an eighth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question to the user, and when the behavior determination unit determines to ask a question to the user as the behavior of the avatar, creates a question to ask the user.

[0015] According to a ninth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes teaching music, and when the behavior determination unit determines that the behavior of the avatar is to teach music, it evaluates a sound produced by the user.

[0016] According to a tenth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question to the user, and when the behavior determination unit determines to ask the user a question as the avatar behavior, the behavior control unit asks the user a question suited to the user based on the content of a text used by the user and a target deviation score of the user.

[0017] According to an eleventh aspect of the present disclosure, there is provided a behavior control system, wherein the behavior decision model of the behavior control system is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on an output of the data generation model.

[0018] According to a twelfth aspect of the present disclosure, there is provided a behavior control system, wherein the behavior decision unit of the behavior control system presents a question suited to the user when determining, as the user's emotion, that the user is in a state of being bored or that the user has been scolded by their parent / guardian to study.

[0019] According to a thirteenth aspect of the present disclosure, there is provided a behavior control system, wherein the behavior decision unit of the behavior control system presents a question with a higher difficulty level to the user when the user can answer the question that has been presented to the user.

[0020] According to a fourteenth aspect of the present disclosure, there is provided a behavior control system, wherein the electronic device of the behavior control system is a headset terminal.

[0021] According to a fifteenth aspect of the present disclosure, there is provided a behavior control system, wherein the electronic device of the behavior control system is a glasses-type terminal.

[0022] Here, the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.

[0023] According to a sixteenth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes a state recognition unit that recognizes a user state, including the user's behavior, and the state of an electronic device; an emotion determination unit that determines the user's emotions or the emotions of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including inaction, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotions, and the avatar's emotions, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes giving advice about a specific competition to the user participating in the specific competition. The behavior determination unit includes an image acquisition unit that can capture an image of the competition space in which the specific competition in which the user is participating is being held, and a feature identification unit that identifies features of a plurality of athletes participating in the specific competition in the competition space imaged by the image acquisition unit. When it is determined that the avatar's behavior is to give advice about the specific competition to the user participating in the specific competition, the advice is given to the user based on the identification result of the feature identification unit.

[0024] According to a seventeenth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes a state recognition unit that recognizes a user state, including user behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes setting a first behavior content that corrects the user's behavior, and the behavior determination unit detects the user's behavior autonomously or periodically, and when it determines, based on the detected user behavior and pre-stored specific information, to correct the user's behavior as the avatar's behavior, causes the behavior control unit to display the avatar in the image display area so as to execute the first behavior content.

[0025] According to an eighteenth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes: a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes providing advice to the user regarding a social networking service, and when the behavior determination unit determines that the avatar's behavior is to provide advice to the user regarding the social networking service, the advice is provided to the user. Here, a robot includes a device that performs physical actions, a device that outputs video and audio without performing physical actions, and an agent that operates on software.

[0026] According to a nineteenth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as an avatar behavior using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays an avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes providing care-related advice to the user, and when the behavior determination unit determines that the avatar behavior is to provide care-related advice to the user, the behavior control unit collects information regarding care-related care for the user and provides the care-related advice to the user based on the collected information.

[0027] Here, the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.

[0028] According to a twentieth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes providing advice to the user regarding an impending risk, and when the behavior determination unit determines that the behavior of the avatar is to provide advice to the user regarding the impending risk, the behavior control unit provides the advice to the user regarding the impending risk.

[0029] According to a twenty-first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving health advice to the user, and when the behavior determination unit determines that the behavior of the avatar is to give health advice to the user, the health advice is given to the user.

[0030] According to a twenty-second aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes autonomously converting a user's utterance into a question, and when the behavior determination unit determines that the avatar's behavior is to convert the user's utterance into a question and answer the question, it converts the user's utterance into a question and answers the question using a sentence generation model based on the event data.

[0031] According to a twenty-third aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes: a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of multiple avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes increasing vocabulary and speaking using the increased vocabulary, and when the behavior determination unit determines to increase vocabulary and speak using the increased vocabulary as the avatar's behavior, the avatar increases vocabulary and speaks using the increased vocabulary. Here, a robot includes a device that performs physical actions, a device that outputs video and audio without performing physical actions, and an agent that operates on software.

[0032] a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes learning a speech method and changing a setting of the speech method, and wherein, when the behavior determination unit determines that the behavior of the avatar is to learn the speech method, the behavior control unit collects utterances of speakers in a predetermined information source, and when the behavior determination unit determines that the behavior of the avatar is to change the setting of the speech method, the behavior control unit changes the voice uttered depending on an attribute of the user.

[0033] In a 25th aspect of the present disclosure, the behavioral decision model is a data generation model capable of generating data according to input data, and the behavioral decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0034] In a 26th aspect of the present disclosure, the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of multiple types of avatar behaviors, including no behavior, as the behavior of the avatar.

[0035] In a 27th aspect of the present disclosure, the behavioral decision model is a sentence generation model with an interactive function, and the behavioral decision unit inputs text representing at least one of the user state, the state of the avatar displayed in the image display area, the user's emotion, and the emotion of the avatar displayed in the image display area, and text asking about the avatar's behavior into the sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model.

[0036] In a 28th aspect of the present disclosure, when the behavior control unit decides to change the speech method setting as the behavior of the avatar, it causes the avatar to act with an appearance corresponding to the changed voice.

[0037] a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes learning a speech method and changing a setting of the speech method, and wherein, when the behavior determination unit determines that the behavior of the avatar is to learn the speech method, the behavior control unit collects utterances of speakers in a predetermined information source, and when the behavior determination unit determines that the behavior of the avatar is to change the setting of the speech method, the behavior control unit changes the voice uttered according to an attribute of the user.

[0038] In a 30th aspect of the present disclosure, the behavioral decision model is a data generation model capable of generating data according to input data, and the behavioral decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0039] In a 31st aspect of the present disclosure, the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of multiple types of avatar behaviors, including no behavior, as the behavior of the avatar.

[0040] In a 32nd aspect of the present disclosure, the behavioral decision model is a sentence generation model with an interactive function, and the behavioral decision unit inputs text representing at least one of the user state, the state of the avatar displayed in the image display area, the user's emotion, and the emotion of the avatar displayed in the image display area, and text asking about the avatar's behavior into the sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model.

[0041] In a thirty-third aspect of the present disclosure, when the behavior control unit decides to change the speech method setting as the behavior of the avatar, it causes the avatar to act with an appearance corresponding to the changed voice.

[0042] According to a 34th aspect of the present disclosure, there is provided a system for controlling a user's behavior, the system including: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes consideration of the user's mental age, and when the behavior determination unit determines to consider the user's mental age as the avatar behavior, it estimates the user's mental age and determines the avatar behavior according to the estimated mental age of the user.

[0043] According to a thirty-fifth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar's behavior includes estimating the user's foreign language level and conversing with the user in the foreign language, and when the behavior determination unit determines to estimate the user's foreign language level as the avatar's behavior, it estimates the user's foreign language level, and when it determines to converse with the user in the foreign language, it converses with the user in the foreign language.

[0044] According to a 36th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice to the user regarding the user's creative activity, and when the behavior determination unit determines that the behavior of the avatar is to give advice to the user regarding the user's creative activity, the behavior control system includes collecting information regarding the user's creative activity and giving the advice regarding the user's creative activity based on the collected information.

[0045] Here, robots include devices that perform physical actions, devices that output images and sounds without performing physical actions, and agents that operate on software.

[0046] According to a 37th aspect of the present disclosure, there is provided a behavior control system, comprising: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and an action control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar actions include making suggestions to encourage actions that the user at home can take, the memory control unit stores the types of actions that the user performs at home in the history data in association with the timing at which the actions were performed, and when the action determination unit, based on the history data, either voluntarily or periodically determines a suggestion to encourage an action that the user at home can take as the avatar's action, it causes the action control unit to display the avatar in the image display area so as to execute the suggestion to encourage the action at the time when the user should perform the action.

[0047] According to a thirty-eighth aspect of the present disclosure, there is provided a behavior control system including: a user state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that determines, at a predetermined timing, one of a plurality of avatar behaviors including no behavior as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes an utterance or a gesture made by the electronic device to the user, and the behavior determination unit determines content of the utterance or the gesture so as to support learning for the user based on sensory characteristics of the user, and causes the behavior control unit to control the avatar.

[0048] According to a thirty-ninth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar, using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit obtains, based on the behavior determination model, music scores of lyrics and melody appropriate to an environment in which the electronic device is located, and determines the behavior of the avatar to play music based on the lyrics and melody using a speech synthesis engine, sing along with the music, and / or dance along with the music.

[0049] According to a fortieth aspect of the present disclosure, the behavior control system according to claim 1, wherein the behavior determination model is a data generation model capable of generating data according to input data, and the behavior determination unit inputs data representing at least one of an environment in which the electronic device is located, a state of the user, a state of the electronic device, an emotion of the user, and an emotion of the avatar, and data asking about the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on an output of the data generation model.

[0050] According to a forty-first aspect of the present disclosure, the behavior control unit controls the avatar to play the music, sing along with the music, and / or dance along with the music.

[0051] According to a forty-second aspect of the present disclosure, the electronic device is a headset-type terminal.

[0052] According to a forty-third aspect of the present disclosure, the electronic device is a glasses-type terminal.

[0053] According to a forty-fourth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that determines an action of the avatar based on at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein when the behavior determination unit determines that the action of the avatar is to answer a user question, it acquires a vector representing the user question, searches a database that stores combinations of questions and answers for a question having a vector corresponding to the acquired vector, and generates an answer to the user question using an answer to the searched question and a sentence generation model that can generate sentences according to input data.

[0054] According to a forty-fifth aspect of the present disclosure, there is provided an information processing system including: an input unit that accepts user input; a processing unit that performs specific processing using a sentence generation model that generates sentences according to the input data; an output unit that controls the behavior of an electronic device to output results of the specific processing; and a behavior control unit that displays an avatar in an image display area of ​​the electronic device, wherein, when pitch information regarding the next ball to be thrown by a specific pitcher is requested, the processing unit performs, as the specific processing, a process of generating sentences that instruct the creation of the pitch information accepted by the input unit and inputting the generated sentences into the sentence generation model, and causes the output unit to output the pitch information created as a result of the specific processing to the avatar representing an agent for interacting with a user.

[0055] According to a forty-sixth aspect of the present disclosure, there is provided an information processing system including: an input unit that accepts user input; a processing unit that performs specific processing using a generative model that generates a result according to the input data; and an output unit that displays an avatar representing an agent for interacting with a user in an image display area of ​​an electronic device so as to output the result of the specific processing, wherein the processing unit uses output of the generative model when the input data is text instructing the presentation of information about earthquakes to obtain information about the earthquake as a result of the specific processing and causes the avatar to output the information about the earthquake.

[0056] According to a forty-seventh aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit uses the behavior determination model to analyze a social networking service (SNS) related to the user, recognize matters of interest to the user based on the results of the analysis, and determine a behavior of the avatar so as to provide information based on the recognized matters to the user.

[0057] According to a 48th aspect of the present disclosure, the behavior determination model is a data generation model capable of generating data according to input data, and the behavior determination unit inputs data representing at least one of the environment in which the electronic device is located, the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

[0058] According to a forty-ninth aspect of the present disclosure, the behavior control unit controls the avatar to provide information based on the recognized matter to the user.

[0059] According to a fiftieth aspect of the present disclosure, the electronic device is a headset-type terminal.

[0060] According to a fifty-first aspect of the present disclosure, the electronic device is a glasses-type terminal.

[0061] A 52nd aspect of the present disclosure is a behavior control system including a state recognition unit that recognizes a user state including user behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein, when the behavior determination unit determines that the user is a specific user, including a lonely person living alone, the behavior determination unit switches to a specific mode in which the avatar's behavior is determined with a greater number of communications than in a normal mode in which behavior is determined toward users other than the specific user.

[0062] In a fifty-third aspect of the present disclosure, the behavioral decision model is a data generation model capable of generating data according to input data, and the behavioral decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0063] In a 54th aspect of the present disclosure, the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of multiple types of avatar behaviors, including no action, as the behavior of the avatar.

[0064] In a 55th aspect of the present disclosure, the behavior decision model is a sentence generation model with an interactive function, and the behavior decision unit inputs text representing at least one of the user state, the state of the avatar displayed in the image display area, the user's emotion, and the emotion of the avatar displayed in the image display area, and text asking about the avatar's behavior into the sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model.

[0065] A 56th aspect of the present disclosure includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit sets a customer service dialogue mode as the avatar's dialogue mode, in which the avatar is positioned as a dialogue partner when there is no need to speak to a specific person but would like someone to listen to what you have to say, and in the customer service dialogue mode, in the dialogue with the user, the speech content is output while excluding predetermined keywords related to the specific person.

[0066] In a fifty-seventh aspect of the present disclosure, the behavioral decision model is a data generation model capable of generating data according to input data, and the behavioral decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0067] In a 58th aspect of the present disclosure, the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of multiple types of avatar behaviors, including no action, as the behavior of the avatar.

[0068] In a fifty-ninth aspect of the present disclosure, the behavioral decision model is a sentence generation model having an interactive function, and the behavioral decision unit inputs text representing at least one of the user state, the state of the avatar displayed in the image display area, the user's emotion, and the emotion of the avatar displayed in the image display area, and text asking about the avatar's behavior into the sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model.

[0069] In a sixtieth aspect of the present disclosure, when the behavior control unit decides to change the settings of the dialogue partner in the customer service dialogue mode as the behavior of the avatar, it causes the avatar to operate with a face and appearance corresponding to the changed dialogue partner.

[0070] According to a sixty-first aspect of the present disclosure, there is provided a behavior control system including a behavior determination unit that determines a behavior of an avatar representing an agent for interacting with a user, and a behavior control unit that displays the avatar in an image display area of ​​an electronic device, the electronic device being set at a customs house, the behavior determination unit acquiring an image of a person taken by an image sensor or an odor detection result by an odor sensor, and determining, upon detecting a predetermined abnormal behavior, abnormal facial expression, or abnormal odor, that the behavior of the avatar is to notify a tax inspector.

[0071] 1A and 1B are schematic diagrams showing an example of a system 5 according to the first embodiment; a functional configuration of a robot 100 according to the first embodiment; a schematic diagram of an example of an operation flow of a collection process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of a response process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of autonomous processing by the robot 100 according to the first embodiment; an emotion map 400 onto which multiple emotions are mapped; an emotion map 900 onto which multiple emotions are mapped; (A) an external view of a stuffed animal 100N according to a second embodiment, (B) a diagram of the internal structure of the stuffed animal 100N; a rear front view of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of an agent system 500 according to a third embodiment; an example of the operation of the agent system; an example of the operation of the agent system; a schematic diagram of the functional configuration of an agent system 700 according to a fourth embodiment; an example of a usage mode of an agent system using smart glasses; a schematic diagram of the functional configuration of an agent system 800 according to a fifth embodiment; an example of a headset-type terminal; 1A and 1B show an example of the hardware configuration of a computer 1200. 1A show an example of other functional configurations of the robot 100. 1A show an example of the functional configuration of a specific processing unit of the robot 100. 1B show an overview of the specific processing. 1B show an example of the operation flow of the specific processing by the robot 100. 1B show an example of the operation flow related to the operation of the robot 100 performing the specific processing to assist the user 10 in announcing information about an earthquake.

[0072] The following describes embodiments of the present disclosure, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure.

[0073] First Embodiment FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of this embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as users 10. Users 11a, 11b, and 11c may be collectively referred to as users 11. Users 12a and 12b may be collectively referred to as users 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

[0074] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 300, and requests the image data, etc. from the server 300 as needed and provides it to the user 10.

[0075] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed when conversing with the user 10. In this way, the robot 100 can express its own emotions through actions.

[0076] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining an action of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the user's action, and determine an action of the robot 100 corresponding to the determined emotion.

[0077] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates an action content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0078] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0079] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0080] As an example of a behavioral decision model, the robot 100 stores rules that define the behavior that the robot 100 will perform based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10, and performs various behaviors in accordance with the rules.

[0081] Specifically, the robot 100 has reaction rules, as an example of a behavioral decision model, for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."

[0082] When the robot 100 recognizes that the behavior of the user 10 is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the words "apologize."

[0083] Furthermore, when the condition that the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to perform the action of "calling out."

[0084] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.

[0085] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

[0086] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

[0087] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0088] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, and a communication processing unit 280.

[0089] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. Furthermore, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0090] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, etc.

[0091] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0092] The storage unit 220 includes a behavioral decision-making model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the past emotional values ​​of the user 10, the past emotional values ​​of the robot 100, and behavioral history. Specifically, the history data 222 includes a plurality of event data including the emotional values ​​of the user 10, the emotional values ​​of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes camera images representing the behavior of the user 10. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. The storage unit 220 may also include a person DB that stores facial images of the user 10, attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2 , excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.

[0093] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the state recognition unit 230.

[0094] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0095] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.

[0096] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

[0097] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."

[0098] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the environment surrounding the robot 100, etc. as the state of the robot 100.

[0099] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0100] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0101] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0102] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."

[0103] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0104] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0105] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.

[0106] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.

[0107] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0108] Next, the processing of the behavior decision unit 236 when the robot 100 performs a response process in which the robot 100 responds to the behavior of the user 10 will be described.

[0109] The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, history data 222 of past emotion values ​​determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, a case is described in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes gestures made by the robot 100 or speech content of the robot 100 .

[0110] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotional values ​​of the user 10, the emotional value of the robot 100, the behavior of the user 10, and the behavior determination model 221. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the behavior of the user 10, an behavior that will change the emotional value of the user 10 to a positive value.

[0111] The reaction rule as the behavioral decision model 221 defines the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the user 10 is behaving sadly, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.

[0112] For example, the reaction rule as the behavior determination model 221 defines behaviors of the robot 100 for patterns of the robot 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values ​​and current emotional values, and all combinations of the user 10's behavior patterns. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the user 10's behavior pattern for each of a plurality of combinations of the user 10's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0113] The reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotional value of the robot 100.

[0114] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined.

[0115] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0116] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0117] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (for example, the facial expression, emotions, etc. of the user 10).

[0118] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotional determination unit 232.

[0119] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. If an impact is detected by the touch sensor 205 included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotion has worsened. Alternatively, if the detection result of the touch sensor 205 included in the sensor unit 200 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0120] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is negative.

[0121] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0122] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0123] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0124] The related information collection unit 270 collects information related to the preference information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the preference information acquired about the user 10 .

[0125] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the content of utterances made by the user 10 or from setting operations made by the user 10. The related information collecting unit 270 periodically collects news related to the preference information from external data using ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>). For example, if the acquired preference information is that the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using ChatGPT Plugins.

[0126] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270 .

[0127] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0128] When the emotion value of the robot 100 is equal to or greater than the threshold value, the storage control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223 .

[0129] Next, the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0130] In the autonomous processing of this embodiment, the robot 100 dreams, that is, creates an original event.

[0131] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0132] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0133] For example, the multiple types of robot behaviors include the following (1) to (10).

[0134] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories.

[0135] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0136] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0137] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0138] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0139] In this case, the behavior determination unit 236 creates an original event by randomly shuffling or exaggerating past experiences and conversations between the robot 100 and the user 10 or the user 10's family from the history data 222. Furthermore, a dream image, a collage of dreams, may be generated using an image generation model based on the created original event, i.e., a dream. In this case, the dream image may be generated based on a single scene from a past memory stored in the history data 222, or may be generated by randomly shuffling and combining multiple memories. Furthermore, in addition to expressing something that does not actually occur, such as a "dream," the dream image may also be generated as an image expressing something the robot 100 saw or heard while the user 10 was away. The generated dream image is like a dream diary, so to speak. In this case, using crayon to touch up the dream image gives the image a more dream-like atmosphere. The behavior determination unit 236 then stores in the behavior schedule data 224 that the generated dream image will be output. As a result, the robot 100 can take actions such as outputting the generated dream image to a display or transmitting it to a user's device in accordance with the behavior schedule data 224.

[0140] The behavior decision unit 236 may cause the robot 100 to output a voice based on the original event. For example, if the original event is related to a panda, the behavior schedule data 224 may store an utterance of "I had a dream about a panda. Take me to the zoo" the next morning. Even in this case, in addition to uttering something that did not actually happen, such as a "dream," the robot 100 may also utter what it saw and heard while the user 10 was away as its own personal experience.

[0141] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0142] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0143] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0144] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0145] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0146] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0147] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0148] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0149] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0150] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0151] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0152] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0153] 3 shows an example of an operational flow for a collection process for collecting information related to the preference information of the user 10. The operational flow shown in FIG. 3 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters of interest to the user 10 has been acquired from the content of speech of the user 10 or from a setting operation performed by the user 10. Note that "S" in the operational flow indicates a step that is executed.

[0154] First, in step S90, the related information collection unit 270 acquires preference information that indicates the matters in which the user 10 is interested.

[0155] In step S92, the related information collection unit 270 collects information related to the preference information from external data.

[0156] In step S94 , the emotion determining unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collecting unit 270 .

[0157] In step S96, the storage control unit 238 determines whether the emotion value of the robot 100 determined in step S94 is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the processing ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the processing proceeds to step S98.

[0158] In step S98, the storage control unit 238 stores the collected information related to the preference information in the collected data 223, and then ends the process.

[0159] 4A shows an example of an operation flow for determining an action of the robot 100 when the robot 100 performs a response process to respond to an action of the user 10. The operation flow shown in FIG. 4A is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input.

[0160] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0161] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0162] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0163] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0164] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized in the above step S104, and the behavior determination model 221.

[0165] In step S108, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

[0166] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0167] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing event data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0168] In step S114, event data including the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0169] 4B shows an example of an operation flow for determining the behavior of the robot 100 when the robot 100 performs autonomous processing to act autonomously. The operation flow shown in FIG. 4B is automatically and repeatedly executed, for example, at regular intervals. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that the same step numbers are used for processes similar to those in FIG. 4A above.

[0170] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0171] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0172] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0173] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0174] In step S200, the behavior decision unit 236 decides on one of a plurality of types of robot behaviors, including no action, as the behavior of the robot 100 based on the state of the user 10 recognized in step S100, the emotions of the user 10 determined in step S102, the emotions of the robot 100, the state of the robot 100 recognized in step S100, the behavior of the user 10 recognized in step S104, and the behavior decision model 221.

[0175] In step S201, the behavior decision unit 236 determines whether or not it was determined in step S200 that no behavior will be performed. If it is determined that no behavior will be performed as the behavior of the robot 100, the process ends. On the other hand, if it is not determined that no behavior will be performed as the behavior of the robot 100, the process proceeds to step S202.

[0176] In step S202, the behavior determination unit 236 performs processing according to the type of robot behavior determined in step S200. At this time, the behavior control unit 250, the emotion determination unit 232, or the memory control unit 238 executes processing according to the type of robot behavior.

[0177] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0178] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0179] In step S114, the memory control unit 238 stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0180] As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the state of the user, and determines whether to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the state of the user 10 in 10 years will be the same as that of 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present to the user 10 all kinds of peripheral information, such as the state of the user 10 from 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sounds, images, smells, etc. of the situation.

[0181] Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 action. Conventionally, the user's actions are classified and the robot's actions, including facial expressions and posture, are determined. In contrast, the robot 100 determines the user's current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, the robot 100 can say to the user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0182] Furthermore, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. Furthermore, for example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing an action based on the state history of the user 10.

[0183] Also, for example, when user 10 is watching a video about a panda, if the emotion value of user 10's emotion of "ease" is equal to or greater than a threshold, the scene in which the panda appears in the video may be stored as event data in the history data 222.

[0184] Using the data accumulated in the history data 222 and the collected data 223, the robot 100 can constantly learn what kind of conversation with the user will maximize the emotional value that expresses the user's happiness.

[0185] Furthermore, even when the robot 100 is not conversing with the user 10, the robot 100 can autonomously start to act based on its own emotions.

[0186] Furthermore, in the autonomous processing, the robot 100 automatically generates questions, inputs them into a sentence generation model, and repeatedly obtains the output of the sentence generation model as an answer to the question, thereby creating emotion change events for increasing positive emotions and storing the events in the behavior schedule data 224. In this way, the robot 100 can perform self-learning.

[0187] Furthermore, when the robot 100 automatically generates a question without receiving an external trigger, the question can be automatically generated based on memorable event data identified from the robot's past emotional value history.

[0188] Furthermore, the related information collecting unit 270 can perform self-learning by repeating the search execution step of automatically performing a keyword search in response to preference information about the user and obtaining search results.

[0189] Here, the search execution stage may be configured to automatically execute a keyword search based on memorable event data identified from the robot's past emotional value history in a state where no external trigger has been received.

[0190] The emotion determining unit 232 may determine the user's emotion in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the user's emotion in accordance with an emotion map (see FIG. 5 ), which is a specific mapping.

[0191] FIG. 5 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0192] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least equal to the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0193] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels are eliminated, enabling a natural, well-read dialogue. The robot 100 performs a reaction (e.g., a backchannel) according to the directionality and degree (strength) of the mandala in the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

[0194] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0195] These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0196] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may be inserted before the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0197] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0198] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0199] Conversely, even when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn its eye LEDs red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.

[0200] (5) The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior. Therefore, the further outside emotion map 400 you go, the more visible the emotion becomes (the more it is expressed in behavior).

[0201] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly.

[0202] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, cars, motorcycles, and other devices, based on various balances, such as posture and battery level. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0203] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0204] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0205] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and the emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on training data indicating that when the robot 100 recognizes from the output of a touch sensor (not shown) that it is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when the robot 100 recognizes from the output of the acceleration sensor 206 that it is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0206] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions to ask about the robot's behavior content corresponding to the user's behavior and inputting the result into a sentence generation model with an interactive function.

[0207] For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0208] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0209] In addition, an emotion table such as that shown in Table 2 is prepared for the emotions of the user 10.

[0210] Here, if the user's behavior is to say "Let's play together," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," then the text "The robot is in a very happy state. The user is in a normal happy state. The user said to the robot, "Let's play together." How would you respond as the robot?" is input into the sentence generation model, and the robot's behavior content is obtained. The behavior determination unit 236 determines the robot's behavior from this behavior content.

[0211]

[0212]

[0213] In this way, the behavior determination unit 236 determines the behavior of the robot 100 in accordance with the emotional state of the robot 100, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10. In this configuration, the content of the utterance of the robot 100 when conversing with the user 10 can be branched depending on the emotional state of the robot 100. In other words, since the robot 100 can change its behavior depending on the index number corresponding to the emotion of the robot, the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0214] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the robot has individuality and encouraging the user to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and actions.

[0215] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values ​​indicating each emotion with emotion values ​​indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each obtained emotion and emotion values ​​indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple pieces of training data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0216] For example, if the utterance content of the robot 100, "That's great. You're lucky," is obtained as the behavior content of the robot 100 generated by the sentence generation model, when the text representing this utterance content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.

[0217] In the robot 100, a sentence generation model such as generative AI and an emotion determination unit 232 work together to implement a method in which the robot has an ego and continues to grow with various parameters even when the user is not speaking.

[0218] Generative AI is a large-scale language model that uses deep learning techniques. Generative AI can also reference external data. For example, ChatGPT plugins are known for their technology of providing answers as accurately as possible while referencing various external data, such as weather information and hotel reservation information, through dialogue. For example, generative AI can automatically generate source code in various programming languages ​​when a goal is given in natural language. For example, generative AI can also debug problematic source code when given problematic source code, discover the problem, and automatically generate improved source code. Combining these, autonomous agents have emerged that, when given a goal in natural language, repeatedly generate and debug code until the source code is free of problems. Known examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0219] In the robot 100 according to this embodiment, the event data to be learned may be stored in a database containing impressive memories, using a technique such as that described in Patent Document 2 (Japanese Patent No. 619992), in which event data that the robot feels strong emotions about is kept for a long time and event data that the robot feels little emotions about is quickly forgotten.

[0220] Furthermore, the robot 100 may record video data of the user 10 acquired by the camera function in the history data 222. The robot 100 may acquire video data from the history data 222 as needed and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the emotion increases and record the video data in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeletal data, the robot 100 may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold. The robot 100 can record high-definition video data, for example, when the robot 100's emotion increases.

[0221] The robot 100 may automatically load event data from the history data 222 in which impressive event data is stored when the robot 100 is not talking to the user 10, and may continuously update the robot's emotion using the emotion determination unit 232. When the robot 100 is not talking to the user 10 and the robot 100's emotion becomes one that encourages learning, the robot 100 can create an emotion change event for changing the emotion of the user 10 for the better, based on the impressive event data. This makes it possible to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and to realize autonomous learning that appropriately reflects the emotional state of the robot 100.

[0222] The emotions that promote learning are those around "repentance" or "reflection" on Dr. Mitsuyoshi's emotional map in the negative state, and those around "desire" on the emotional map in the positive state.

[0223] In a negative state, the robot 100 may treat "repentance" and "remorse" in the emotion map as emotions that encourage learning. In a negative state, the robot 100 may treat emotions adjacent to "repentance" and "remorse" in addition to "repentance" and "remorse" in the emotion map as emotions that encourage learning. For example, the robot 100 may treat at least one of "regret," "stubbornness," "self-destruction," "self-admonition," "regret," and "despair" as emotions that encourage learning, in addition to "repentance" and "remorse." This allows the robot 100 to perform autonomous learning when it feels negative emotions such as "I never want to feel this way again" or "I don't want to be scolded anymore."

[0224] In a positive state, the robot 100 may treat "desire" in the emotion map as an emotion that encourages learning. In a positive state, the robot 100 may treat emotions adjacent to "desire" as emotions that encourage learning, in addition to "desire." For example, the robot 100 may treat at least one of "happiness," "euphoria," "craving," "anticipation," and "shyness" as emotions that encourage learning, in addition to "desire." This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more" or "wanting to know more."

[0225] The robot 100 may be configured not to perform autonomous learning when the robot 100 is experiencing emotions other than the emotions that encourage learning as described above. This allows the robot 100 to not perform autonomous learning when, for example, the robot 100 is extremely angry or blindly feeling love.

[0226] An emotional change event is, for example, a suggestion of an action that follows a memorable event. The action that follows a memorable event is the outermost emotion label on the emotion map. For example, the action of "tolerance" or "acceptance" follows "love."

[0227] In the autonomous learning that is performed when the robot 100 is not talking to the user 10, the robot 100 combines the emotions, situations, actions, etc. of people who appear in impressive memories and the user 10, and creates an emotion change event using a sentence generation model.

[0228] Let us consider a case where all emotion values ​​are expressed on a six-level scale from 0 to 5, and where memorable event data such as "a friend looked displeased after being hit" is stored in the history data 222. The friend here refers to the user 10, and the emotion of the user 10 is "disgust," with 5 being entered as the value representing "disgust." Also, let us assume that the emotion of the robot 100 is "anxiety," and 4 is entered as the value representing "anxiety."

[0229] The robot 100 can continue to grow with various parameters by performing autonomous processing while not talking to the user 10. Specifically, event data such as "The friend looked displeased after being hit" is loaded from the history data 222 as the top event data sorted in descending order of emotional value. Assume that the loaded event data is associated with the robot 100's emotion of "anxiety" with a strength of 4, and the friend, the user 10, is associated with the emotion of "disgust" with a strength of 5. If the robot 100's current emotional value is "relief" with a strength of 3 before loading, after loading, the robot 100's emotional value may change to "regret," meaning disappointment (frustration), due to the influence of "anxiety" with a strength of 4 and "disgust" with a strength of 5. In this case, because "regret" is an emotion that encourages learning, the robot 100 decides to recall the event data as robot behavior and creates an emotion change event. At this time, the information input to the sentence generation model is text that represents memorable event data, and in this example, it is "The friend looked displeased after being hit." Also, since the emotion map has the emotion of "disgust" at the innermost position and the corresponding behavior predicted as "attack" at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone.

[0230] For example, by using information from impressive event data to solve fill-in-the-blank questions, you can automatically generate input text like the one below.

[0231] "A user was being criticized. At the time, the user felt very disgusted. The robot was very anxious. Please tell us what the robot should say the next time it meets the user in 30 characters or less. However, please make sure it is not related to the time of day they will meet. Also, please avoid direct language. We will provide three candidates. <Expected format> Candidate 1: (Words the robot should say to the user) Candidate 2: (Words the robot should say to the user) Candidate 3: (Words the robot should say to the user)"

[0232] In this case, the output of the sentence generation model is, for example, as follows:

[0233] "Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0234] Furthermore, the robot 100 may automatically generate the following input text based on the information obtained by creating the emotion change event:

[0235] "If a user was being bashed, how would the user feel if you spoke to them in the following way? The user's emotions are in the format of "joy A, anger B, sadness C, pleasure D," where A to D are integers on a 6-point scale from 0 to 5. Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0236] In this case, the output of the sentence generation model is, for example, as follows:

[0237] "The user's emotions might be as follows: Option 1: 3 happy, 1 angry, 2 sad, 2 happy; Option 2: 2 happy, 1 angry, 3 sad, 2 happy; Option 3: 2 happy, 1 angry, 3 sad, 3 happy"

[0238] In this way, the robot 100 may perform a process of pondering after creating an emotion change event.

[0239] Finally, the robot 100 may create an emotion change event using candidate 1 that is most likely to please people from among the multiple candidates, store it in the action schedule data 224, and prepare it for the next time the robot 10 meets the user 10.

[0240] As described above, even when the robot 100 is not conversing with family or friends, the robot continues to determine its emotional value using information from the history data 222 in which impressive event data is stored, and when the robot 100 experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning in accordance with the emotion of the robot 100 even when the robot 100 is not conversing with the user 10, and continues to update the history data 222 and the action schedule data 224.

[0241] The above is an example using emotional values, but since an emotional map can create emotions from the amount of hormone secretion and the type of event, the values ​​linked to memorable event data could also be the type of hormone, the amount of hormone secretion, or the type of event.

[0242] Specific examples will be described below.

[0243] For example, the robot 100 may look up information about topics or hobbies that interest the user, even when not talking to the user.

[0244] For example, even when the robot 100 is not talking to the user, the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.

[0245] For example, even when the robot 100 is not talking to the user, it checks reviews of places, foods, and products that the user wants to visit.

[0246] For example, even when the robot 100 is not talking to the user, it checks weather information and provides advice tailored to the user's schedule and plans.

[0247] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about local events and festivals and suggests them to the user.

[0248] For example, even when the robot 100 is not talking to the user, it checks the results and news of sports that interest the user and provides topics of conversation.

[0249] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and introduces information about the user's favorite music and artists.

[0250] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information on social issues or news that the user is concerned about and provides opinions.

[0251] For example, even when the robot 100 is not talking to the user, it searches for information about the user's hometown or birthplace and provides topics of conversation.

[0252] For example, the robot 100 checks information about the user's work or school and provides advice even when not talking to the user.

[0253] Even when the robot 100 is not talking to the user, it searches for and introduces information about books, comics, movies, and dramas that may interest the user.

[0254] For example, the robot 100 may check information about the user's health and provide advice even when not speaking with the user.

[0255] For example, the robot 100 may look up information about the user's travel plans and provide advice even when not speaking with the user.

[0256] For example, the robot 100 may look up information and provide advice on repairs and maintenance for the user's home or car, even when not speaking with the user.

[0257] For example, even when the robot 100 is not talking to the user, it searches for information on beauty and fashion that the user is interested in and provides advice.

[0258] For example, the robot 100 checks information about the user's pet and provides advice even when not talking to the user.

[0259] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work.

[0260] For example, the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when not talking to the user.

[0261] For example, the robot 100 may collect information and provide advice on important decisions affecting the user's life, even when not speaking with the user.

[0262] For example, the robot 100 may find information about people the user is concerned about and provide advice, even when not speaking with the user.

[0263] [Second Embodiment] In the second embodiment, the robot 100 is mounted on a stuffed toy or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy. Note that parts having the same configuration as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0264] Specifically, the second embodiment is configured as follows. For example, the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in FIGS. 7 and 8 ) that spends daily life with a user 10, and that advances conversations with the user 10 based on information about the user's daily life, and provides information tailored to the user's hobbies and preferences. In the second embodiment, an example will be described in which the control unit of the robot 100 is applied to a smartphone 50.

[0265] The stuffed toy 100N, which is equipped with the function of an input / output device for the robot 100, has a detachable smartphone 50 that functions as the control part of the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed toy 100N.

[0266] As shown in FIG. 7A , in this embodiment (and other embodiments), the stuffed toy 100N has the shape of a bear covered in soft fabric. A space 52 is formed inside the stuffed toy 100N, and a sensor unit 200A and a control target 252A are disposed therein as input / output devices (see FIG. 9 ). The sensor unit 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7B , the microphone 201 of the sensor unit 200 is disposed in the space 52 corresponding to the ears 54, the 2D camera 203 of the sensor unit 200 is disposed in the space 52 corresponding to the eyes 56, and a speaker 60, which constitutes part of the control target 252A, is disposed in the space 52 corresponding to the mouth 58. The microphone 201 and the speaker 60 do not necessarily need to be separate entities and may be an integrated unit. In the case of a unit, they should be disposed in a position where speech can be heard naturally, such as at the nose of the stuffed toy 100N. Although the plush toy 100N has been described as having the shape of an animal, the plush toy 100N is not limited to this. The plush toy 100N may have the shape of a specific character.

[0267] 9 shows a schematic functional configuration of the plush toy 100N. The plush toy 100N has a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252A.

[0268] The smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the first embodiment. That is, the smartphone 50 has a function as a sensor module unit 210, a function as a storage unit 220, and a function as a control unit 228 shown in FIG.

[0269] As shown in FIG. 8, a zipper 62 is attached to a part of the stuffed animal 100N (for example, the back), and opening the zipper 62 allows communication between the outside and the space 52.

[0270] Here, the smartphone 50 is housed in the space 52 from the outside and connected to each input / output device via USB via a USB hub 64 (see Figure 7 (B)), thereby giving the smartphone 50 functionality equivalent to that of the robot 100 of the first embodiment described above.

[0271] A non-contact type power receiving plate 66 is also connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.

[0272] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0273] The stuffed toy 100N placed on the mounting base 70 can be appreciated as an ornament in its natural state.

[0274] In addition, this base portion is formed thinner than the surface thickness of other portions of the stuffed toy 100N, so that it is held closer to the mounting base 70.

[0275] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmitting coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmitting coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. This causes a current to flow through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0276] That is, by placing the stuffed toy 100N on the mounting base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.

[0277] In the second embodiment, the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected via a wire (USB connection), but this is not limiting. For example, a control device with wireless functionality (e.g., Bluetooth (registered trademark)) may be housed in the space 52 of the stuffed toy 100N and connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing functionality equivalent to that of the robot 100 of the first embodiment. Furthermore, the control device housed in the space 52 of the stuffed toy 100N may be connected via a wire.

[0278] In the second embodiment, the teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.

[0279] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, or happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be projected. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, or happiness by the image projected onto a wall.

[0280] According to the second embodiment, an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 to appropriate positions via a USB connection.

[0281] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed toy 100N.

[0282] When using wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.

[0283] For this reason, the smartphone 50 is placed as close to the center of the stuffed toy 100N as possible, and the wireless charging function (power receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0284] The other configurations and operations of the stuffed toy 100N of the second embodiment are the same as those of the robot 100 of the first embodiment, and therefore will not be described.

[0285] In addition, parts of the stuffed toy 100N (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the stuffed toy 100N (e.g., a server), and the stuffed toy 100N may function as each part of the stuffed toy 100N by communicating with the outside.

[0286] [Third Embodiment] In the first embodiment, the behavior control system is applied to the robot 100. In the third embodiment, the robot 100 is used as an agent for interacting with a user, and the behavior control system is applied to an agent system. Note that parts having the same configuration as those in the first and second embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0287] FIG. 10 is a functional block diagram of an agent system 500 configured using some or all of the functions of the behavior control system.

[0288] The agent system 500 is a computer system that performs a series of actions in accordance with the intentions of the user 10 through a dialogue with the user 10. The dialogue with the user 10 can be carried out by voice or text.

[0289] The agent system 500 includes a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228B, and a control target 252B.

[0290] The agent system 500 may be installed in, for example, a robot, a doll, a stuffed animal, a wearable device (a pendant, a smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 500 may also be implemented on a web server and used via a web browser running on a communication device such as a smartphone owned by a user.

[0291] The agent system 500 acts as, for example, a butler, secretary, teacher, partner, friend, lover, or instructor for the user 10. The agent system 500 not only interacts with the user 10, but also provides advice, guides the user to a destination, or makes recommendations based on the user's preferences. The agent system 500 also makes reservations, orders, or makes payments to service providers.

[0292] The emotion determination unit 232 determines the emotions of the user 10 and the agent itself, as in the first embodiment. The behavior determination unit 236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and services. The agent system 500 also listens to the user's concerns, comforts, encourages, and cheers them up. The agent system 500 also plays with the user 10, draws a picture diary, and reminds them of the past. The agent system 500 performs behavior that increases the user's sense of happiness. Here, the agent is an agent that runs on software.

[0293] The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, and a communication processing unit 280.

[0294] As in the first embodiment, the behavior determination unit 236 determines, as the agent's behavior, the content of the agent's utterance to converse with the user 10. The behavior control unit 250 outputs the content of the agent's utterance as at least one of voice and text to a speaker or a display as a control target 252B.

[0295] The character setting unit 276 sets the character of the agent when the agent system 500 converses with the user 10, based on the user's specification. That is, the utterances output from the behavior determination unit 236 are output through the agent having the set character. For example, real-life celebrities or famous people, such as actors, entertainers, idols, and athletes, can be set as the character. It is also possible to set fictional characters that appear in manga, movies, or animation. If the agent character is known, the voice, speech, tone, and personality of the character are known. Therefore, the user 10 simply specifies their preferred character, and the prompt setting in the character setting unit 276 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276 and outputs the agent's utterances using the synthesized voice. This allows the user 10 to feel as if they are conversing with their preferred character (e.g., a favorite actor) in person.

[0296] When the agent system 500 is installed in a device with a display, such as a smartphone, an icon, still image, or video of the agent having the character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using image synthesis technology, such as 3D rendering. In the agent system 500, a dialogue with the user 10 may be conducted while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the content of the agent's utterances. Note that the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.

[0297] As in the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, an emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the conversation. In other words, the behavior control unit 250 outputs the content of the utterance in a manner corresponding to the emotion determined by the emotion determination unit 232.

[0298] The agent's emotions are also reflected when the agent system 500 takes an action toward the user 10. For example, if the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take the photo in response to the user's request is determined by the degree of "sadness" the agent is feeling. If the character is feeling positive emotions, it will have friendly conversations or actions toward the user 10, and if it is feeling negative emotions, it will have hostile conversations or actions toward the user 10.

[0299] The history data 222 stores the history of the dialogue between the user 10 and the agent system 500 as event data. The storage unit 220 may be realized by an external cloud storage. When the agent system 500 dialogues with the user 10 or takes an action toward the user 10, the agent system 500 determines the content of the dialogue or the content of the action by taking into account the content of the dialogue history stored in the history data 222. For example, the agent system 500 identifies the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 222. The agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations. The behavior determination unit 236 determines the content of the agent's utterance based on the dialogue history stored in the history data 222. The history data 222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, obtained through the dialogue with the user 10. Here, the agent may spontaneously ask the user 10 whether or not to register personal information, such as "Would you like to register your credit card number?", and the personal information may be stored in the history data 222 depending on the user 10's response.

[0300] As described in the first embodiment, the behavior determination unit 236 generates utterance content based on sentences generated using the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance content of the agent. At this time, the behavior determination unit 236 may further input the personality of the character set by the character setting unit 276 into the sentence generation model to generate the utterance content of the agent. In the agent system 500, the sentence generation model is not located on the front end side, which is the touchpoint with the user 10, but is merely used as a tool of the agent system 500.

[0301] The command acquisition unit 272 uses the output of the speech understanding unit 212 to acquire commands for the agent from the voice or text uttered by the user 10 through dialogue with the user 10. The commands include the content of actions to be performed by the agent system 500, such as information search, store reservation, ticket arrangement, product / service purchase, payment, route guidance to a destination, and recommendation provision.

[0302] The RPA 274 performs an action according to the command acquired by the command acquisition unit 272. The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, and payment.

[0303] The RPA 274 reads and uses personal information of the user 10 from the history data 222, which is necessary to execute actions related to the use of the service provider. For example, when purchasing a product at the request of the user 10, the agent system 500 reads and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 222. Requiring the user 10 to input personal information during initial setup is unkind and unpleasant for the user. In the agent system 500 according to this embodiment, instead of requiring the user 10 to input personal information during initial setup, the agent system 500 stores personal information acquired through dialogue with the user 10 and reads and uses it as needed. This avoids causing discomfort to the user and improves user convenience.

[0304] The agent system 500 executes the dialogue process, for example, through the following steps 1 to 6.

[0305] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10, based on a specification from the user 10.

[0306] (Step 2) The agent system 500 acquires the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222. Specifically, the same processing as in steps S100 to S103 above is performed to acquire the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222.

[0307] (Step 3) The agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character specified by the emotion determination unit 232, and the conversation history stored in the history data 222 into a sentence generation model to generate the content of the agent's utterance.

[0308] For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the text representing the conversation history stored in the history data 222, and this is input into the sentence generation model to obtain the content of the agent's speech.

[0309] As an example, if the text or voice input by the user 10 is "Please make a reservation at a delicious Chinese restaurant nearby for tonight at 7pm," the agent's speech will be "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0310] Also, if the text or voice input by the user 10 is "Number 4, DDDD, would be good," the agent's speech content acquired will be "Understood. I will try to make a reservation. How many seats are there?"

[0311] (Step 4) The agent system 500 outputs the agent's utterance. Specifically, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance in the synthesized voice.

[0312] (Step 5) The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 236 determines whether it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent will execute a command, it determines that it is time to execute the agent's command and proceeds to step 6. On the other hand, if it is determined that it is not time to execute the agent's command, it returns to step 2 above.

[0313] (Step 6) The agent system 500 executes the agent's command. Specifically, the command acquisition unit 272 acquires the agent's command from the voice or text uttered by the user 10 through a dialogue with the user 10. The RPA 274 then performs an action according to the command acquired by the command acquisition unit 272. For example, if the command is "information search," an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface). The behavior determination unit 236 inputs the search results into a sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276 and outputs the agent's utterance content using the synthesized voice.

[0314] Furthermore, if the command is "make a restaurant reservation," the reservation information obtained through the dialogue with the user 10, the restaurant information, and the API are used to call the restaurant via telephone software to make the reservation. At this time, the behavior determination unit 236 uses a sentence generation model with a dialogue function to acquire the content of the agent's utterance in response to the voice input from the other party. The behavior determination unit 236 then inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the content of the agent's utterance. The behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the content of the agent's utterance using the synthesized voice.

[0315] Then, return to step 2 above.

[0316] In step 6, the results of the actions taken by the agent (for example, making a restaurant reservation) are also stored in the history data 222. The results of the actions taken by the agent stored in the history data 222 are used by the agent system 500 to understand the hobbies or preferences of the user 10. For example, if the same restaurant has been reserved multiple times, the agent system 500 may recognize that the user 10 likes that restaurant, and may use the reservation details, such as the time slot reserved, the course contents or price, as criteria for choosing a restaurant for the next reservation.

[0317] In this manner, the agent system 500 can execute the interaction and, if necessary, take action regarding the use of the service provider.

[0318] 11 and 12 are diagrams showing an example of the operation of the agent system 500. Fig. 11 illustrates an example in which the agent system 500 makes a restaurant reservation through a dialogue with the user 10. In Fig. 11, the left side shows the content of the agent's utterances, and the right side shows the content of the user's utterances. The agent system 500 can understand the preferences of the user 10 based on the dialogue history with the user 10, provide a recommended list of restaurants that match the preferences of the user 10, and make a reservation at the selected restaurant.

[0319] Meanwhile, FIG. 12 illustrates an example in which the agent system 500 accesses a mail-order site and purchases a product through a dialogue with the user 10. In FIG. 12, the left side shows the agent's speech, and the right side shows the user's speech. The agent system 500 can estimate the remaining amount of a beverage stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of that beverage to the user 10. The agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10 and recommend snacks that the user likes. In this way, the agent system 500 communicates with the user 10 as a butler-like agent and performs various actions, such as making restaurant reservations or paying for product purchases, thereby supporting the user 10's daily life.

[0320] The other configurations and operations of the agent system 500 of the third embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described again.

[0321] In addition, parts of the agent system 500 (e.g., the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside (e.g., a server) of a communication terminal such as a smartphone carried by the user, and the communication terminal may communicate with the outside to function as each part of the agent system 500.

[0322] [Fourth Embodiment] In the fourth embodiment, the agent system described above is applied to smart glasses. Note that parts having the same configuration as those in the first to third embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0323] 13 is a functional block diagram of an agent system 700 configured using some or all of the functions of the behavior control system. The agent system 700 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252B. The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA 274, a character setting unit 276, and a communication processing unit 280.

[0324] 14, the smart glasses 720 are glasses-type smart devices that are worn by the user 10 in the same manner as regular glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0325] The smart glasses 720 include an agent system 700. The display included in the control object 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the smart glasses 720, and the displayed content is visible to the user 10. The speaker included in the control object 252B outputs audio indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), which accepts input from the user 10.

[0326] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.

[0327] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the smart glasses 720. The 2D camera 203 is capable of capturing images around the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0328] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls communication between the smart glasses 720 and the outside.

[0329] FIG. 14 is a diagram showing an example of how the agent system 700 is used by the smart glasses 720. The smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (e.g., by voice input into a microphone or by tapping a touch panel with a finger), the smart glasses 720 begin using the agent system 700. Here, using the agent system 700 includes the smart glasses 720 having and using the agent system 700, and also includes an aspect in which a part of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) is provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 communicates with the outside to use the agent system 700.

[0330] When the user 10 operates the smart glasses 720, a touch point is created between the agent system 700 and the user 10. That is, the agent system 700 starts providing a service. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent.

[0331] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, if the heart rate of the user 10 detected by the heart rate sensor 208 is elevated, emotion values ​​such as "anxiety" and "fear" are estimated to be high.

[0332] Furthermore, if the temperature of the user measured by the temperature sensor 207 is higher than the average body temperature, for example, the emotional value of "pain" or "distress" is estimated to be large. Furthermore, if the acceleration sensor 206 detects that the user 10 is playing some kind of sport, the emotional value of "fun" or the like is estimated to be large.

[0333] Furthermore, for example, the emotion value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is raising his / her voice, an emotion value such as "anger" is estimated to be large.

[0334] When the emotion value estimated by the emotion determination unit 232 is higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation. Specifically, for example, the agent system 700 causes the 2D camera 203 to capture images or videos showing the surrounding situation of the user 10 (e.g., people or objects in the vicinity). The agent system 700 also causes the microphone 201 to record ambient sounds. Other information about the surrounding situation includes information indicating the date, time, location information, or weather. The information about the surrounding situation is stored in the history data 222 together with the emotion value. The history data 222 may be realized by external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 222 as a so-called life log, associated with the emotion value of the user 10 at that time.

[0335] In the agent system 700, information indicating the surrounding situation is stored in association with an emotional value in the history data 222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a scene of watching a baseball game is associated with emotional values ​​such as "joy" or "fun," the agent system 700 can grasp that the hobby of the user 10 is watching baseball games and that the favorite team or player is a favorite player from the information stored in the history data 222.

[0336] When the agent system 700 converses with the user 10 or takes an action toward the user 10, the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 222. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 222 as described above in addition to the surrounding circumstances.

[0337] As described above, the behavior determination unit 236 generates utterance content based on sentences generated by the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 232, the conversation history stored in the history data 222, the agent's personality, etc. into the sentence generation model to generate the agent's utterance content. Furthermore, the behavior determination unit 236 inputs the surrounding circumstances stored in the history data 222 into the sentence generation model to generate the agent's utterance content.

[0338] The generated speech content is output to the user 10 as a voice from, for example, a speaker mounted on the smart glasses 720. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The behavior control unit 250 generates the synthetic voice by reproducing the voice quality of the character of the agent, or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone when the emotion is "anger"). Furthermore, instead of or together with the voice output, the speech content may be displayed on a display.

[0339] The RPA 274 executes an operation in response to a command (for example, a command of an agent acquired from a voice or text uttered by the user 10 through a dialogue with the user 10). The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, payment, route guidance, and translation.

[0340] As another example, the RPA 274 executes an operation of transmitting the contents of speech input by the user 10 (e.g., a child) through a dialogue with an agent to a destination (e.g., a parent). Examples of transmission means include message application software, chat application software, and email application software.

[0341] When an operation by the RPA 274 is executed, for example, a sound indicating that the execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What do you want to do?" is output to the user 10.

[0342] In addition, parts of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) may be provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 may communicate with the outside to function as each part of the agent system 700 described above.

[0343] As described above, the smart glasses 720 provide various services to the user 10 by using the agent system 700. Furthermore, since the smart glasses 720 are worn by the user 10, the agent system 700 can be used in various situations, such as at home, at work, or when out and about.

[0344] Furthermore, since the smart glasses 720 are worn by the user 10, they are suitable for collecting a so-called life log of the user 10. Specifically, the emotional value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203. Therefore, the emotional value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech content suited to the emotions of the user 10.

[0345] Furthermore, the smart glasses 720 obtain the surrounding conditions of the user 10 using the 2D camera 203, microphone 201, etc. These surrounding conditions are associated with the emotion values ​​of the user 10. This makes it possible to estimate what emotions the user 10 felt in what situations. As a result, the agent system 700 can improve the accuracy in grasping the hobbies and preferences of the user 10. By accurately grasping the hobbies and preferences of the user 10 in the agent system 700, the agent system 700 can provide services or speech content that are suited to the hobbies and preferences of the user 10.

[0346] The agent system 700 can also be applied to other wearable devices (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands). When the agent system 700 is applied to a smart pendant, the speaker serving as the control target 252B outputs audio indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting directional audio. The speaker is set to have directionality toward the ears of the user 10. This prevents the audio from reaching anyone other than the user 10. The microphone 201 acquires audio emitted by the user 10 or environmental sounds around the smart pendant. The smart pendant is worn by hanging it from the neck of the user 10. Therefore, while the smart pendant is worn, the smart pendant is located relatively close to the mouth of the user 10. This makes it easy to acquire audio emitted by the user 10.

[0347] Fifth Embodiment In the fifth embodiment, the robot 100 is used as an agent for interacting with a user through an avatar. That is, the behavior control system is used in an agent system configured using a headset-type terminal. Note that parts having the same configuration as those in the first and second embodiments are designated by the same reference numerals and will not be described again.

[0348] 15 is a functional block diagram of an agent system 800 configured using some or all of the functions of the behavior control system. The agent system 800 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252C. The agent system 800 is realized, for example, by a headset-type terminal 820 as shown in FIG. 16.

[0349] In addition, parts of the headset type terminal 820 (e.g., the sensor module unit 210B, the storage unit 220, the control unit 228B) may be provided outside the headset type terminal 820 (e.g., a server), and the headset type terminal 820 may function as each part of the above-mentioned agent system 800 by communicating with the outside.

[0350] In this embodiment, the control unit 228B has the function of determining the behavior of the avatar and generating the display of the avatar to be presented to the user through the headset terminal 820.

[0351] As in the first embodiment, the emotion determination unit 232 of the control unit 228B determines the emotion value of the agent based on the state of the headset terminal 820 and uses it as the emotion value of the avatar. The emotion determination unit 232 may determine the emotion of the user or the emotion of an avatar representing the agent for interacting with the user.

[0352] As in the first embodiment, when an agent functioning as an avatar performs autonomous processing to act autonomously, the behavior decision unit 236 of the control unit 228B determines, at a predetermined timing, one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the avatar, and the state of the electronic device controlling the avatar (e.g., the headset-type terminal 820), and the behavior decision model 221. The behavior decision model 221 may be a data generation model capable of generating data according to input data.

[0353] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the state of the electronic device, the emotion of the user 10, and the emotion of the avatar, as well as text asking about the avatar's behavior, into a sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

[0354] Furthermore, the behavior control unit 250 displays the avatar in the image display area of ​​the headset type terminal 820 as the control target 252C according to the determined behavior of the avatar. Furthermore, if the determined behavior of the avatar includes the content of the avatar's utterance, the content of the avatar's utterance is output as audio from the speaker as the control target 252C.

[0355] In particular, when the behavior determination unit 236 determines that the avatar's behavior is to dream, i.e., to create an original event, the behavior control unit 250 controls the avatar to create the original event. That is, when the behavior determination unit 236 determines that the avatar's behavior is to dream, the behavior determination unit 236 uses a sentence generation model to create an original event by combining multiple event data from the history data 222, as in the first embodiment. In this case, the behavior determination unit 236 creates the original event by randomly shuffling or exaggerating past experiences and conversations between the avatar and the user 10 or the user 10's family from the history data 222. Furthermore, the behavior determination unit 236 generates a dream image, a collage of dreams, using an image generation model based on the created original event, i.e., a dream. In this case, the dream image may be generated based on a scene from a past memory stored in the history data 222, or the dream image may be generated by randomly shuffling and combining multiple memories. For example, if the action decision unit 236 obtains from the history data 222 that the user 10 was camping in a forest, it may generate a dream image showing that the user 10 was camping on a riverbank. Furthermore, if the action decision unit 236 obtains from the history data 222 that the user 10 was watching a fireworks display at a certain location, it may generate a dream image showing that the user 10 was watching the fireworks display at a completely different location. Furthermore, in addition to representing things that did not actually happen, such as "dreams," it may also be possible to generate dream images that represent what the avatar saw and heard while the user 10 was away.

[0356] The behavior control unit 250 controls the avatar to generate a dream image. Specifically, it generates an image of the avatar so that the avatar draws the dream image generated by the behavior determination unit 236 on a canvas, whiteboard, or the like in the virtual space. As a result, the headset terminal 820 displays in the image display area an image of the avatar drawing the dream image on a canvas, whiteboard, or the like.

[0357] The behavior control unit 250 may change the facial expression or movement of the avatar depending on the content of the dream. For example, if the content of the dream is fun, the facial expression of the avatar may be changed to a happy expression, or the movement of the avatar may be changed to a happy dance. The behavior control unit 250 may also transform the avatar depending on the content of the dream. For example, the behavior control unit 250 may transform the avatar into an avatar that resembles a character in the dream, or an animal, object, or the like that appears in the dream.

[0358] The behavior control unit 250 may also generate an image in which an avatar holds a tablet device drawn in a virtual space and performs an action of drawing a dream image on the tablet device. In this case, by sending the dream image displayed on the tablet device to the mobile terminal device of the user 10, it is possible to make it appear as if the avatar is performing an action such as sending the dream image from the tablet device to the mobile terminal device of the user 10 by email or to a messaging app. Furthermore, in this case, the user 10 can view the dream image displayed on their own mobile terminal device.

[0359] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0360] In the above embodiment, a case where a headset-type terminal 820 is used is described as an example, but this is not limited to this, and a glasses-type terminal having an image display area for displaying an avatar may also be used.

[0361] Furthermore, in the above embodiment, an example has been described in which a sentence generation model capable of generating sentences according to input text is used, but this is not limited thereto, and a data generation model other than a sentence generation model may also be used. For example, a prompt including an instruction is input to the data generation model, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is input. The data generation model performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0362] In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10. However, the disclosed technology is not limited to this. For example, the robot 100 may recognize the user 10 using a voice uttered by the user 10, the email address of the user 10, the SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 has.

[0363] The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.

[0364] 17 schematically shows an example of the hardware configuration of a computer 1200 that functions as the smartphone 50, the robot 100, the server 300, and the agent systems 500, 700, and 800. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to perform operations associated with the apparatus according to the present embodiment or one or more "parts," and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0365] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0366] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0367] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0368] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0369] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0370] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0371] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0372] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0373] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0374] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0375] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0376] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0377] Computer-readable instructions may be provided locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0378] Sixth Embodiment In the autonomous processing of this embodiment, the device operation (robot behavior when the electronic device is the robot 100) determined by the behavior determination unit 236 includes proposing an activity. When the behavior determination unit 236 determines to propose an activity as the behavior of the electronic device (robot behavior), it determines the behavior of the user 100 to be proposed based on the event data.

[0379] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0380] As described above, when the behavior determination unit 236 determines that the robot behavior is "(5) The robot proposes an activity," that is, when it determines to propose an action to the user 10, it can determine the user's behavior to propose using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior determination unit 236 may propose "play," "study," "cooking," "travel," or "shopping" as the user's behavior. In this way, the behavior determination unit 236 can determine the type of activity to propose. When proposing "play," the behavior determination unit 236 can also suggest, "Let's go on a picnic this weekend." When proposing "cooking," the behavior determination unit 236 can also suggest, "Let's have curry rice for dinner tonight." When proposing "shopping," the behavior determination unit 236 can also suggest, "Let's go to X shopping mall." In this way, the behavior determination unit 236 can also determine details of the proposed activity, such as "when," "where," and "what." In determining the type and details of such an activity, the behavior determination unit 236 can learn about the past experiences of the user 10 by using the event data stored in the history data 222. The behavior determination unit 236 may suggest an activity that the user 10 has enjoyed in the past, may suggest an activity that the user 10 is likely to like based on the user 10's tastes and preferences, or may suggest a new activity that the user 10 has not experienced in the past.

[0381] In particular, when the behavior decision unit 236 decides to suggest an activity as the avatar's behavior, it is preferable to have the behavior control unit 250 control the avatar to decide the user's behavior to be suggested based on the event data.

[0382] Specifically, when the behavior determining unit 236 determines to propose an activity as the avatar behavior, that is, to propose an action of the user 10, the behavior determining unit 236 can determine the user's behavior to propose using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior determining unit 236 may propose "play," "study," "cooking," "travel," "tonight's dinner menu," "picnic," or "shopping" as the behavior of the user 10. In this way, the behavior determining unit 236 can determine the type of activity to propose. When proposing "play," the behavior determining unit 236 can also suggest, "Let's go on a picnic this weekend." When proposing "cooking," the behavior determining unit 236 can also suggest, "Let's have curry rice for dinner tonight." When proposing "shopping," the behavior determining unit 236 can also suggest, "Let's go to the XX shopping mall." In this way, the behavior determining unit 236 can also determine details of the activity to be proposed, such as "when," "where," and "what." In determining the type and details of such an activity, the behavior determining unit 236 can learn the past experiences of the user 10 by using the event data stored in the history data 222. The behavior determining unit 236 may then suggest at least one of an activity that the user 10 has enjoyed in the past, an activity that the user 10 is likely to like based on the user's 10 tastes and preferences, and a new activity that the user 10 has not experienced in the past.

[0383] In addition, when the behavior control unit 250 suggests an activity as an avatar behavior, it may cause the avatar to operate to perform the suggested activity and display the avatar in the image display area of ​​the headset-type terminal 820 as the control object 252C.

[0384] Seventh Embodiment In the autonomous processing of this embodiment, the device operation (robot behavior, if the electronic device is the robot 100) determined by the behavior determination unit 236 includes comforting the user 10. When the behavior determination unit 236 determines that the behavior of the electronic device (robot behavior) is to comfort the user 10, it determines the user state and the speech content corresponding to the emotion of the user 10.

[0385] For example, the multiple types of robot behaviors include the following (1) to (11).

[0386] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot comforts the user.

[0387] When the robot 100 determines that the robot behavior is "(11) The robot comforts the user," that is, when the robot 100 determines to make an utterance that comforts the user 10, the behavior determining unit 236 determines the content of the utterance corresponding to the state of the user 10 and the emotion of the user 10. For example, when the state of the user 10 satisfies the condition that the state of the user 10 is "depressed," the behavior determining unit 236 determines the content of the utterance corresponding to the state of the user 10 and the emotion of the user 10. Note that the state of the user 10 that is depressed may be recognized, for example, by performing a perception process using the analysis results of the sensor module unit 210. In such a case, the behavior determining unit 236 determines the content of the utterance corresponding to the state of the user 10 and the emotion of the user 10. For example, when the user 10 is depressed, the behavior determining unit 236 may determine the content of the utterance such as "What's wrong? Did something happen at school?", "Are you worried about something?", or "I'm always available to talk to you." In response to this, the behavior control unit 250 may output a sound representing the determined utterance content of the robot 100 from a speaker included in the control target 252. In this way, the robot 100 can provide the user 10 (child, family, etc.) with an opportunity to verbalize and release their emotions by listening to them. In this way, the robot 100 can ease the user 10's feelings by calming them down, helping them sort out their problems, or helping them find a clue to solving the problems.

[0388] In particular, when the behavior decision unit 236 determines that the behavior of the avatar is to comfort the user, it is preferable to have the behavior control unit 250 control the avatar to, for example, listen to the story of a depressed child or family member, and comfort the depressed child or family member.

[0389] Eighth Embodiment In the autonomous processing of this embodiment, the device operation (robot behavior, in the case where the electronic device is the robot 100) determined by the behavior determination unit 236 includes presenting a question to the user 10. When the behavior determination unit 236 determines that a question is to be presented to the user 10 as the behavior of the electronic device (robot behavior), the behavior determination unit 236 creates a question to be presented to the user 10.

[0390] For example, the multiple types of robot behaviors include the following (1) to (11).

[0391] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot asks the user questions.

[0392] When the behavior determination unit 236 determines that the robot 100 will issue an utterance of "(11) The robot asks the user a question" as the robot behavior, i.e., when the robot 100 determines to issue an utterance of asking the user 10 a question, the behavior determination unit 236 creates a question to ask the user 10. For example, the behavior determination unit 236 may create a question to ask the user 10 based on at least one of the user's dialogue history and the user's personal information. As an example, when the user's dialogue history indicates that the user 10 is weak in arithmetic, the behavior determination unit 236 may create a question such as "What is 7 x 7?" In response to this, the behavior control unit 250 may output a sound representing the created question from a speaker included in the control target 252. Next, when the user 10 answers "49," the behavior determination unit 236 may determine the content of the utterance to be "Correct. Well done, amazing!" Then, when the user 10's emotions indicate that the user 10 is interested in the question, the behavior determination unit 236 may create a new question with the same questioning tendency. As another example, if the user's personal information indicates that the user is 10 years old, the behavior determination unit 236 may create a question appropriate for the user's age, such as "What is the capital of the United States?" In response, the behavior control unit 250 may output a sound representing the created question from a speaker included in the control target 252. Next, if the user 10 answers "New York," the behavior determination unit 236 may determine the content of the utterance to be, "Sorry. The correct answer is Washington, D.C." Then, if it is inferred from the user's emotions that the user 10 is not interested in the question, the behavior determination unit 236 may change the question-asking tendency and create a new question. In this way, the robot 100 can voluntarily ask questions in a game-like manner to encourage the user 10, who is, for example, a child, to enjoy studying, and can increase the user's motivation to learn by praising or expressing joy based on the user's answer.

[0393] In particular, when the behavior decision unit 236 decides that the avatar's behavior is to ask a question to the user, it preferably causes the behavior control unit 250 to control the avatar to create a question to ask the user.

[0394] Specifically, when the behavior determination unit 236 determines that the avatar will pose a question to the user 10 as the avatar behavior, i.e., when the avatar determines to utter an utterance to pose a question to the user 10, the behavior determination unit 236 creates a question to pose to the user 10. For example, the behavior determination unit 236 may create a question to pose to the user 10 based on at least one of the user's dialogue history and the user's personal information. As an example, when the user's dialogue history indicates that the user 10 is weak in arithmetic, the behavior determination unit 236 may create a question such as, "What is 7 x 7?" In response to this, the behavior control unit 250 may output a sound representing the created question from a speaker serving as the control target 252C. Next, when the user 10 answers "49," the behavior determination unit 236 may determine the content of the utterance to be, "Correct. Well done, amazing!" Then, when the user 10's emotions indicate that the user 10 is interested in the posed question, the behavior determination unit 236 may create a new question with the same questioning tendency. As another example, if the user's personal information indicates that the user is 10 years old, the behavior determining unit 236 may create a question appropriate for the user's age, such as "What is the capital of the United States?" In response to this, the behavior control unit 250 may output a sound representing the created question from a speaker serving as the control target 252C. Next, if the user 10 answers "New York," the behavior determining unit 236 may determine the content of the utterance to be "Sorry. The correct answer is Washington, D.C." Then, if it is inferred from the user's emotions that the user 10 is not interested in the question, the behavior determining unit 236 may change the questioning tendency and create a new question. In this way, an avatar in AR (Augmented Reality) or VR (Virtual Reality) can voluntarily ask questions in a game-like manner to help a user 10, who may be a child, to enjoy studying, and can praise or express joy for the user's 10 answers, thereby increasing the user's motivation to learn.

[0395] In addition, when the behavior control unit 250 is to ask a question to the user as an avatar behavior, it may operate the avatar to ask the user the question that was created, and display the avatar in the image display area of ​​the headset-type terminal 820 as the control object 252C.

[0396] Ninth Embodiment In the autonomous processing of this embodiment, the device operation (robot behavior, in the case where the electronic device is the robot 100) determined by the behavior determination unit 236 includes teaching music. When the behavior determination unit 236 determines that music is to be taught as the behavior of the electronic device (robot behavior), it evaluates the sound generated by the user 10.

[0397] For example, the multiple types of robot behaviors include the following (1) to (11).

[0398] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot teaches music.

[0399] When the behavior determining unit 236 determines that the robot 100 will make an utterance to teach the user 10 about music, i.e., "(11) The robot teaches music," as the robot behavior, it evaluates the sound produced by the user 10. Note that the "sound produced by the user 10" here may be interpreted as including various sounds produced in association with the user 10's behavior, such as the user 10's singing voice, the sound of an instrument played by the user 10, or the user 10's tapping sound. For example, when it is recognized from the user 10's behavior that the user 10 is singing, playing an instrument, or dancing, the behavior determining unit 236 determines that the robot behavior is "(11) The robot teaches music." In such a case, the behavior determining unit 236 may evaluate at least one of the sense of rhythm, pitch, or intonation of the user 10's singing voice, the sound of an instrument, the tapping sound, or the like. Then, the behavior determining unit 236 may determine the content of the utterance, such as "The rhythm is not consistent," "The pitch is off," or "Put more feeling into it," depending on the evaluation result. In response to this, the behavior control unit 250 may output a sound representing the determined content of the utterance of the robot 100 from a speaker included in the control target 252. In this way, the robot 100 can spontaneously evaluate the sounds produced by the user 10 and point out differences in rhythm and pitch, even without a question from the user 10, and can therefore interact with the user 10 as a music teacher.

[0400] In particular, when the behavior decision unit 236 decides that the behavior of the avatar is to teach music, it is preferable to have the behavior control unit 250 control the avatar to evaluate the sound generated by the user.

[0401] Specifically, when the behavior determining unit 236 determines that the avatar behavior is "The avatar teaches music," that is, when the avatar utters an utterance that teaches music to the user 10, the behavior determining unit 236 evaluates the sound produced by the user 10. Note that the "sound produced by the user 10" here may be interpreted as including various sounds produced in association with the user 10's behavior, such as the user 10's singing voice, the sound of an instrument played by the user 10, or the tapping sound of the user 10. For example, when it is recognized from the user 10's behavior that the user 10 is singing, playing an instrument, or dancing, the behavior determining unit 236 determines that the avatar behavior is "The avatar teaches music." In such a case, the behavior determining unit 236 may evaluate at least one of the sense of rhythm, pitch, and intonation of the user 10's singing voice, instrument sound, tapping sound, etc. Then, the behavior determination unit 236 may determine the content of the utterance, such as "The rhythm is not consistent," "The pitch is off," or "Put more feeling into it," depending on the evaluation result. In response, the behavior control unit 250 may output a sound representing the determined utterance content of the avatar from a speaker serving as the control target 252C. In this way, an avatar in AR (Augmented Reality) or VR (Virtual Reality) can spontaneously evaluate the sound produced by the user 10 without being asked by the user 10, and can speak the evaluation results and point out differences in rhythm, pitch, etc., thereby interacting with the user 10 as a music teacher.

[0402] Furthermore, when teaching music as an avatar behavior, the behavior control unit 250 may operate the avatar to speak the results of evaluating the sound produced by the user, and display the avatar in the image display area of ​​the headset-type terminal 820 as the control object 252C.

[0403] Tenth Embodiment In the autonomous processing of this embodiment, the robot 100 as an agent performs the autonomous processing. More specifically, the robot 100 performs the autonomous processing to take action based on the past history of the robot 100 (there may be no history) and the behavior of the user 10, regardless of whether the user 10 is present or not.

[0404] The robot 100 as an agent autonomously and periodically detects the state of the user 10. For example, the robot 100 reads the text of a textbook from the school or cram school attended by the user 10, and uses an AI-based sentence generation model to have the robot 10 think up new questions and generate questions that match the user 10's preset target deviation score (e.g., 50, 60, 70, etc.).

[0405] The robot 100 may determine the subject of the questions to be posed based on the behavioral history of the user 10. In other words, if it is known from the behavioral history that the user 10 is studying arithmetic, the robot 100 generates arithmetic questions and poses the generated questions to the user 10.

[0406] In particular, when the behavior decision unit 236 determines that the avatar's behavior is to ask a question to the user 10, as described in the first embodiment, it is preferable that the behavior decision unit 236 generates a question that matches a predetermined target deviation value for the user 10 (e.g., 50, 60, 70, etc.) and controls the behavior control unit 250 to ask the generated question.

[0407] When presenting a question to the user 10, the behavior control unit 250 may control the avatar to change into a specific person, such as a parent, friend, school teacher, or cram school instructor. In particular, for school teachers and cram school instructors, it is preferable to change the avatar for each subject. For example, the behavior control unit 250 controls the avatar to a foreign person for English and a person wearing a white coat for science. In this case, the behavior control unit 250 may have the avatar read the question aloud or hold a piece of paper with the question written on it. Furthermore, in this case, the behavior control unit 250 may control the avatar to change its facial expression based on the emotion value of the user 10 determined by the emotion determination unit 232. For example, if the emotion value of the user 10 is positive, such as "joy" or "pleasure," the behavior control unit 250 may change the avatar's facial expression to a cheerful one. If the emotion value of the user 10 is negative, such as "anxiety" or "sadness," the behavior control unit 250 may change the avatar's facial expression to one that encourages the user 10.

[0408] Furthermore, the behavior control unit 250 may control the avatar to change into the appearance of a blackboard or whiteboard on which the question is written when asking the user 10. If a time limit is set for answering a question, the behavior control unit 250 may change the avatar into the appearance of a clock indicating the time remaining until the time limit when asking the question to the user 10. Furthermore, when asking a question to the user 10, the behavior control unit 250 may control the avatar to display, in addition to the humanoid avatar, a virtual blackboard or whiteboard and a virtual clock indicating the time remaining until the time limit. In this case, after the avatar holding the whiteboard asks the user 10 a question, the avatar can change the whiteboard into a clock and inform the user 10 of the remaining time.

[0409] The behavior control unit 250 may control the behavior of the avatar so that, when the user 10 correctly answers a question posed by the avatar, the avatar behaves in a manner that praises the user 10. Furthermore, when the user 10 does not correctly answer a question posed by the avatar, the behavior control unit 250 may control the behavior of the avatar so that the avatar behaves in a manner that encourages the user 10.

[0410] Furthermore, when the user 10 is struggling to come up with an answer to a question posed by the avatar and is pondering over it, the behavior control unit 250 may control the behavior of the avatar to give a hint to the answer.

[0411] When the behavior control unit 250 changes the behavior of the avatar, the facial expression of the avatar can be changed not only in accordance with the emotional value of the user 10, but also in accordance with the emotional value of the agent avatar, the target deviation value of the user 10, etc. Also, the currently displayed avatar may be replaced with another avatar in response to a predetermined action of the user 10 in response to the questions. For example, a teacher avatar may be changed to be replaced with an angelic avatar in response to the avatar correctly answering all the questions, or a gentle-looking avatar may be changed to be replaced with a fierce-looking avatar in response to a drop in the target deviation value due to a series of mistakes in the avatar's answers.

[0412] Eleventh Embodiment In the autonomous processing of this embodiment, the robot 100 identifies the status of a user participating in a specific sport and athletes on the opposing team, particularly the characteristics of the athletes, at any timing, or periodically, and provides advice to the user regarding the specific sport based on the identification results. Here, the specific sport may be a sport played by a team of multiple people, such as volleyball, soccer, or rugby. Furthermore, the user participating in the specific sport may be an athlete playing the specific sport, or a support staff member such as a manager or coach of a specific team playing the specific sport. Furthermore, the characteristics of an athlete refer to information related to the sport, such as the athlete's habits, movements, number of mistakes, weak movements, and reaction speed, as well as information related to the athlete's current or recent condition.

[0413] For example, the multiple types of robot behaviors include the following (1) to (11).

[0414] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot gives advice to a user participating in a specific competition.

[0415] When the behavior decision unit 236 determines that the robot's behavior is "(11) The robot gives advice to a user participating in a specific competition," that is, to give advice to a user, such as an athlete or coach, participating in a specific competition regarding the specific competition in which the robot is participating, the behavior decision unit 236 first identifies the characteristics of the multiple athletes participating in the competition in which the user is participating.

[0416] In order to identify the above-mentioned characteristics of the athlete, the behavior determination unit 236 has an image acquisition unit that captures images of the competition space where the specific competition in which the user is participating is being held. The image acquisition unit can be realized, for example, by using a part of the sensor unit 200 described above. Here, the competition space may include a space corresponding to each competition, such as a volleyball court or a soccer field. This competition space may also include the surrounding area of ​​the aforementioned court. The installation position of the robot 100 should be considered so that the image acquisition unit can overlook the competition space.

[0417] The behavior determination unit 236 also includes a feature identification unit capable of identifying features of multiple athletes in images acquired by the image acquisition unit. This feature identification unit can identify the features of multiple athletes by analyzing past competition data using a method similar to the emotion value determination method used by the emotion determination unit 232, by collecting and analyzing information about each athlete from social media or the like, or by combining one or more of these methods. The image acquisition unit and feature identification unit may be collected and stored as part of the collected data 223 by the related information collection unit 270. In particular, information such as the past competition data of the athletes described above may be collected by the related information collection unit 270.

[0418] If the characteristics of athletes playing a particular sport, such as volleyball, can be identified, the results of this identification can be reflected in the team's strategy, potentially giving the team an advantage in the match. Specifically, an athlete who makes a large number of mistakes or an athlete with a particular habit can be a weak point for the team. Therefore, in this embodiment, advice for gaining an advantage in the match, specifically the characteristics of each athlete identified by the action decision unit 236, is communicated to the user, for example, the coach of one of the teams in the match, thereby providing the user with advice.

[0419] Considering the above points, it is preferable that the athletes whose characteristics are identified by the characteristic identification unit are athletes who belong to a specific team among multiple athletes in the competition space. More specifically, the specific team is a team different from the team to which the user belongs, in other words, the opposing team. The robot 100 scans the characteristics of each athlete on the opposing team, identifies athletes with specific habits or who frequently make mistakes, and provides information about the characteristics of these athletes as advice to the user, thereby assisting the user in creating an effective strategy.

[0420] If a user utilizes the advice provided by the robot 100 during a match in a competitive game in which teams face off against each other, it is expected that the user will gain an advantage in the match. Specifically, for example, by identifying a player who makes many mistakes during a match based on the advice from the robot 100 and adopting a strategy to focus on and attack the position of that player, the user can get closer to victory.

[0421] The advice given by the behavior decision unit 236 may be autonomously executed by the robot 100, rather than being initiated in response to an inquiry from the user. Specifically, the robot 100 may make a speech by itself when it detects, for example, when the manager who is the user is in trouble, when the team to which the user belongs is about to lose, or when members of the team to which the user belongs are having a conversation that suggests they want advice.

[0422] A specific example of a method for the behavior control unit 250 to cause an avatar to perform a desired action is described below. First, the state of a plurality of athletes participating in a competition in which the user is participating is detected, including their characteristics. The detection of the characteristics of the plurality of athletes can be achieved by the image acquisition unit of the behavior decision unit 236 described above. The detection of the emotions of the athletes can be performed, for example, autonomously or periodically by the behavior control unit 250. In this case, it is preferable to place the image acquisition unit in a location where the user, etc. is competing, i.e., where it can overlook the entire competition space. Taking this into consideration, the image acquisition unit can be configured, for example, as a camera with a communication function that can be installed in any location independent of the headset-type terminal 820.

[0423] When analyzing the features of multiple athletes in the images acquired by the image acquisition unit, the feature identification unit of the action decision unit 236 described above is used. The features of each athlete analyzed by the feature identification unit can be reflected in the control of the avatar by the action control unit 250.

[0424] In the agent system 800 according to this embodiment, the behavior control unit 250 controls the avatar based on at least the characteristics identified by the characteristic identification unit. The specific manner in which the behavior control unit 250 controls the avatar is not particularly limited, as long as the control can provide predetermined advice to the user. While the control may primarily involve causing the avatar to speak, other actions, either alone or in combination with speech, can also be used to make the meaning easier for the user to understand. Therefore, several examples of the avatar control performed by the behavior control unit 250 are described below. In the following description, it is assumed that the agent system 800 is used to provide advice about a game to a coach of one of the teams participating in a volleyball game via a headset-type terminal 820 worn by the coach.

[0425] When the behavior determination unit 236 determines that the avatar's behavior is to give advice to the user (coach) regarding the volleyball match in which the user is participating, the behavior control unit 250 begins providing the advice through the avatar. For example, by reflecting the characteristics of a specific player among multiple players in the avatar as a method of providing advice, information regarding the status of the specific player can be provided to the user. To explain a more specific example, when the characteristic identification unit identifies a player on the opposing team who makes many mistakes or has a specific habit, the behavior control unit 250 changes the appearance of the avatar to resemble the identified player and reflects the characteristics identified by the characteristic identification unit in the avatar's facial expressions, movements, etc. This allows the status of the specific player to be visually communicated to the user. In addition, by having the avatar speak the status of the specific player using the output of the behavior determination model 221, the user can more accurately grasp the status of the specific player.

[0426] For example, if it is determined that a particular player on the opposing team makes more mistakes than other players, the avatar displayed to resemble that particular player can be made to pale in the face or perform the actions that would be taken if the player made a mistake, thereby immediately informing the user that the particular player is prone to making mistakes. In addition, if, together with this avatar display, the avatar uses the output of the behavioral decision-making model 221 to make an utterance such as "The player with number 7 on the opposing team makes a lot of mistakes," the coach as the user can plan a strategy that takes into account the situation of that player.

[0427] Furthermore, for example, if it is possible to identify an athlete on the opposing team who has a particular habit, the avatar can be made to resemble that athlete and perform a movement that the athlete is not good at, thereby instantly informing the user of the habit of the specific athlete. In addition, if, together with this avatar display, the avatar uses the output of the behavioral decision-making model 221 to make an utterance such as "The player with number 5 on the opposing team is not good at receiving," the coach as the user can plan a strategy that takes into account the situation of that player.

[0428] Furthermore, when the behavior determination unit 236 determines that the avatar's behavior is to give advice to the user (coach) regarding a volleyball match in which the user is participating, the behavior control unit 250 can cause the avatar to reflect information about the uniform worn during the specific match. Specifically, the behavior control unit 250 can cause the avatar to reflect information about the volleyball uniform for which advice is to be given, i.e., wear the uniform. The uniform worn by the avatar may be a general volleyball uniform prepared in advance, or it may be the uniform of the team to which the user belongs or the uniform of the opposing team. The information about the uniform of the team to which the user belongs or the uniform of the opposing team may be generated, for example, by analyzing an image acquired by the image acquisition unit, or may be registered in advance by the user.

[0429] As described above, reflecting uniform information in an avatar makes it easier for users to understand the information provided by the avatar. In the above example, it becomes easier to understand that the information provided by the avatar is related to the volleyball game in which the user is participating. In addition, as in the above example, when an avatar is displayed to resemble a specific athlete, by also making the uniform similar to that worn by the specific athlete, it becomes easier for users to understand which athlete the avatar resembles.

[0430] In the above example, an avatar is displayed to resemble a specific athlete, but the specific athlete is not limited to one. Similarly, the number of avatars displayed in the image display area of ​​the electronic device is not particularly limited. Therefore, the action determination unit 236 can, for example, reflect the characteristics, uniforms, etc. of all players on the user's opposing team as a specific athlete in multiple avatars and display them.

[0431] In the above embodiment, a headset-type terminal 820 is used as the electronic device, but this is not limited to this, and for example, a glasses-type terminal having an image display area for displaying an avatar may also be used.

[0432] Twelfth Embodiment The user's state may include the user's behavioral tendency. The behavioral tendency may be interpreted as a hyperactive or impulsive user behavioral tendency, such as the user frequently running up stairs, frequently climbing or attempting to climb on top of a dresser, or frequently climbing onto the edge of a window to open it. The behavioral tendency may also be interpreted as a tendency for hyperactive or impulsive behavior, such as the user frequently walking on top of a fence or attempting to climb on top of a fence, or frequently walking on a roadway or entering the roadway from the sidewalk.

[0433] In addition, in autonomous processing, the agent may ask the generative AI a question about the detected user state or behavior, and store the generative AI's answer to the question in association with the detected user behavior. At this time, the agent may store the action content for correcting the behavior in association with the answer.

[0434] The information associating the generative AI's answer to the question, the detected user behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0435] In addition, in the autonomous processing, an action schedule for the robot 100 may be set based on the detected user's action and the stored specific information to call attention to the user's state or action.

[0436] As described above, the agent may record table information in a storage medium that associates the generative AI's response corresponding to the user's state or behavior with the detected user's state or behavior. Examples of the contents stored in the table are described below.

[0437] (1. When the user tends to frequently run up the stairs) When this tendency exists, the agent itself asks the generative AI, "What else is a child who behaves like this likely to do?" If the generative AI's answer to this question is, for example, "The user is likely to trip up the stairs," the agent may store the user's behavior of running up the stairs in association with the generative AI's answer. The agent may also store an action to correct this behavior in association with the answer.

[0438] The action content for correcting the behavior may include at least one of performing a gesture for correcting the user's risky behavior and playing a sound for correcting the behavior.

[0439] The gestures for correcting the risky behavior may include gestures and hand gestures that guide the user to a specific location, gestures and hand gestures that make the user stay still at that location, etc. The specific location may include a location other than the location where the user is currently located, for example, a location near the robot 100, a space inside a window, etc.

[0440] The audio for correcting dangerous behavior may include audio such as "Stop it," "It's dangerous, don't move," etc. The audio for correcting dangerous behavior may include audio such as "Don't run," "Stay still," etc.

[0441] (2. When the user has a tendency to frequently be on top of the dresser or to try to climb on top of the dresser) When this tendency exists, the agent asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "the user may fall off the dresser" or "the user may get caught in the dresser door," the agent may store the user's behavior of being on top of the dresser or trying to climb on top of the dresser in association with the generative AI's answer. The agent may also store the content of an action to correct this behavior in association with the answer.

[0442] (3. When the user has a tendency to frequently open the window by climbing up to the edge of the window) When this tendency exists, the agent asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "There is a possibility that the user will stick their head out the window" or "There is a possibility that the user will get caught in the window," the agent may store the user's behavior of opening the window by climbing up to the edge of the window in association with the generative AI's answer. The agent may also store the content of an action to correct this behavior in association with the answer.

[0443] (4. When the user has a tendency to frequently walk on or attempt to climb on top of a fence) When this tendency exists, the agent asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "the user may fall off the fence" or "the user may be injured by unevenness in the wall," the agent may store the user's behavior of walking on or attempting to climb on top of a fence in association with the generative AI's answer. The agent may also store the content of an action to correct this behavior in association with the answer.

[0444] (5. When the user has a tendency to frequently walk on the roadway or enter the roadway from the sidewalk) When this tendency exists, the agent asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "There is a possibility of a traffic accident occurring" or "There is a possibility of causing traffic congestion," the agent may store the user's behavior of walking on the roadway or entering the roadway from the sidewalk in association with the generative AI's answer. The agent may also store the content of an action to correct this behavior in association with the answer.

[0445] In this way, in autonomous processing, a table that associates the answer of the generative AI corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0446] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and a behavioral schedule for the robot 100 to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the behavior decision unit 236 of the robot 100 may cause the behavior control unit 250 to operate the robot 100 so as to execute a first behavioral content for correcting the user's behavior based on the detected user's behavior and the contents of the stored table. An example of the first behavioral content will be described below.

[0447] (1. When the user tends to frequently run up the stairs) When the behavior decision unit 236 detects a user running up the stairs, the behavior control unit 250 may cause the robot 100 to operate so as to perform a first behavioral content to correct the behavior, such as a gesture or hand gesture to guide the user to a place other than the stairs, or a gesture or hand gesture to stop the user in that place.

[0448] Furthermore, the behavior determination unit 236 may play, as the first behavior content for correcting the behavior, a sound guiding the user to a place other than the stairs, a sound instructing the user to stay in that place, etc. The sound may include sounds such as "XX-chan, it's dangerous, don't run," "Don't move," "Don't run," and "Stay still."

[0449] (2. When the user is frequently on top of the dresser or tends to try to climb on top of the dresser) The behavior decision unit 236 may cause the behavior control unit 250 to operate the robot 100 to perform gestures and hand movements to keep the user who is on top of the dresser or trying to climb on top of the dresser still in place, or gestures and hand movements to move the user to a location other than the current location.

[0450] (3. When the user tends to frequently climb up to the window edge and open the window) The behavior decision unit 236 may cause the behavior control unit 250 to operate the robot 100 to perform gestures and hand movements that will keep the user, who is at the window edge or at the window edge with his / her hands on the window, stationary in that location, or that will move the user to a location other than the current location.

[0451] (4. When the user has a tendency to frequently walk on or attempt to climb on fences) The behavior decision unit 236 may cause the behavior control unit 250 to operate the robot 100 to perform gestures and hand movements that will stop a user who is walking on or attempting to climb on a fence in place, or gestures and hand movements that will move the user to a location other than the current location.

[0452] (5. When the user tends to frequently walk on the roadway or enter the roadway from the sidewalk) The behavior decision unit 236 may cause the behavior control unit 250 to operate the robot 100 so as to perform gestures and hand movements to stop the user who is walking on the roadway or has entered the roadway from the sidewalk in that location, or gestures and hand movements to move the user to a location other than the current location.

[0453] The behavior decision unit 236 may detect the user's behavior after the robot 100 performs a gesture that is the first behavior content or plays back a sound that is the first behavior content, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, may cause the behavior control unit 250 to operate the robot 100 to perform a second behavior content that is different from the first behavior content.

[0454] A case where the user's behavior is corrected may be interpreted as a case where the user stops performing dangerous behavior or actions as a result of the robot 100 performing the first behavior content, or a case where a dangerous situation is resolved.

[0455] The second action content may include playing at least one of a sound praising the user's action and a sound thanking the user for the action.

[0456] The voice praising the user's actions may include voices such as "Are you okay? You listened well," "Good job, that's amazing," etc. The voice thanking the user for their actions may include voices such as "Thank you for coming."

[0457] The behavior decision unit 236 may detect the user's behavior after the robot 100 performs a gesture that is the first behavior content or plays back a sound that is the first behavior content, thereby determining whether the user's behavior has been corrected, and if the user's behavior has not been corrected, may cause the behavior control unit 250 to operate the robot 100 to perform a third behavior content that is different from the first behavior content.

[0458] A case where the user's behavior is not corrected may be interpreted as a case where the user continues to perform dangerous behavior and actions despite the robot 100 performing the first behavior content, or a case where the dangerous situation is not resolved.

[0459] The third action content may include at least one of sending specific information to a person other than the user, performing a gesture that attracts the user's interest, playing a sound that attracts the user's interest, and playing a video that attracts the user's interest.

[0460] The transmission of specific information to a person other than the user may include the delivery of an email containing a warning message to the user's guardian, childcare worker, etc., the delivery of an image (still image, moving image) containing the user and the scenery around him / her, etc. The transmission of specific information to a person other than the user may also include the delivery of an audio warning message.

[0461] The gestures that attract the user's interest may include body movements and hand movements of the robot 100. Specifically, the gestures may include the robot 100 waving both arms widely, blinking the LEDs in the eyes of the robot 100, and the like.

[0462] The playing of sounds that interest the user may include particular music that the user likes, and may also include sounds such as "come here" and "let's play together."

[0463] Playback of videos that interest the user may include images of the user's pet animals, images of the user's parents, and so on.

[0464] According to the robot 100 of the present disclosure, autonomous processing can detect whether a child or the like is attempting to engage in dangerous behavior (e.g., climbing onto the window edge to open the window), and if danger is detected, autonomously execute an action to correct the user's behavior. As a result, the robot 100 can autonomously execute gestures and utterances such as "Stop it" or "Hey, it's dangerous, come over here." Furthermore, if a child stops the dangerous behavior in response to a call, the robot 100 can also perform an action to praise the child, such as "Are you okay? You listened well." Furthermore, if the child does not stop the dangerous behavior, the robot 100 can send a warning email to the parent or caregiver, share the situation via video, and perform an action that the child finds interesting, play a video that the child finds interesting, or play music that the child finds interesting, thereby encouraging the child to stop the dangerous behavior.

[0465] For example, the multiple types of robot behaviors include the following (1) to (26).

[0466] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user is interested in. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot recalls memories. (11) The robot 100 may perform gestures and hand movements to guide the user to a location other than the stairs as a first behavioral content to correct the user's behavior. (12) The robot 100 may perform gestures and hand movements to keep the user still in place as a first behavioral content to correct the user's behavior. (13) The robot 100 may play audio to guide the user to a location other than the stairs as a first behavioral content to correct the user's behavior. (14) As a first behavioral content for correcting the user's behavior, the robot 100 may play a sound or the like to make the user stop in place. (15) As a first behavioral content for correcting the user's behavior, the robot 100 may execute a gesture or hand gesture to make the user, who is on top of a dresser or trying to climb on top of a dresser, stop in place, or a gesture or hand gesture to move to a location other than the current location. (16) As a first behavioral content for correcting the user's behavior, the robot 100 may execute a gesture or hand gesture to make the user, who is on the edge of a window or is on the edge of a window and has his / her hands on the window, stop in place, or a gesture or hand gesture to move to a location other than the current location. (17) As a first behavioral content for correcting the user's behavior, the robot 100 may execute a gesture or hand gesture to make the user, who is walking on top of a fence or trying to climb on a fence, stop in place, or a gesture or hand gesture to move to a location other than the current location. (18) As a first action content for correcting the user's behavior, the robot 100 may perform a gesture or hand gesture to stop the user who is walking on the roadway or has entered the roadway from the sidewalk in that place, or a gesture or hand gesture to move the user to a location other than the current location.(19) When the user's behavior is corrected, the robot 100 may execute, as a second behavior content different from the first behavior content, at least one of a voice praising the user's behavior and a voice thanking the user for the user's behavior. (20) When the user's behavior is not corrected, the robot 100 may execute, as a third behavior content different from the first behavior content, sending specific information to a person other than the user. (21) The robot 100 may execute, as the third behavior content, a gesture that attracts the user's interest. (22) The robot 100 may execute, as the third behavior content, at least one of playing a sound that attracts the user's interest and playing a video that attracts the user's interest. (23) The robot 100 may execute, as the sending of specific information to a person other than the user, sending an email containing a warning message to the user's guardian, childcare worker, etc. (24) The robot 100 may execute, as the sending of specific information to a person other than the user, sending an image (still image, moving image) that includes the user and a surrounding landscape. (25) The robot 100 may deliver a warning message by voice as a transmission of specific information to a person other than the user. (26) The robot 100 may perform at least one of waving both arms widely and blinking LEDs in the eyes of the robot 100 as a gesture to attract the user's attention.

[0467] The behavior decision unit 236 detects the user's behavior spontaneously or periodically, and if it decides to correct the user's behavior as the behavior of the electronic device, which is robot behavior, based on the detected user behavior and pre-stored specific information, it can execute the following first behavior content.

[0468] The behavior decision unit 236 may execute the first behavior content of "(11)" described above as the robot behavior, that is, gestures and hand movements that guide the user to a place other than the stairs.

[0469] The behavior determination unit 236 may execute the first behavior content of "(12)" described above as the robot behavior, that is, a gesture and hand movement that stops the user in place.

[0470] The behavior determination unit 236 may play back, as the robot behavior, the first behavior content of "(13)" described above, that is, a voice that guides the user to a place other than the stairs.

[0471] The behavior determination unit 236 may play back the first behavior content of the above-mentioned "(14)", that is, a sound that makes the user stand still in place, as the robot behavior.

[0472] The behavior determination unit 236 may execute the first behavior content of "(15)" described above as the robot behavior. That is, the behavior determination unit 236 may execute a gesture or hand gesture to stop the user who is on top of a dresser or who is about to climb on top of the dresser in that place, or a gesture or hand gesture to move the user to a place other than the current location.

[0473] The behavior determination unit 236 may execute the first behavior content of "(16)" described above as the robot behavior. That is, the behavior determination unit 236 may execute a gesture or hand gesture to keep a user who is standing on the edge of a window or who is standing on the edge of a window and has his / her hands on the window still in that place, or a gesture or hand gesture to move the user to a location other than the current location.

[0474] The behavior determination unit 236 may execute the first behavior content of "(17)" described above as the robot behavior. That is, the behavior determination unit 236 may execute a gesture or hand gesture to stop a user who is walking on a fence or trying to climb a fence in that location, or a gesture or hand gesture to move the user to a location other than the current location.

[0475] The behavior determination unit 236 may execute the first behavior content of "(18)" described above as the robot behavior. That is, the behavior determination unit 236 may execute a gesture or hand gesture to stop a user who is walking on a roadway or who has entered the roadway from a sidewalk in that place, or a gesture or hand gesture to move the user to a place other than the current location.

[0476] When the user's behavior is corrected, the behavior determination unit 236 may execute a second behavior content different from the first behavior content. Specifically, the behavior determination unit 236 may execute, as the robot behavior, the second behavior content of "(19)" described above, i.e., playback of at least one of a voice praising the user's behavior and a voice thanking the user for the user's behavior.

[0477] If the user's behavior is not corrected, the behavior determining unit 236 may execute a third behavior content that is different from the first behavior content. Examples of the third behavior content will be described below.

[0478] The behavior decision unit 236 may execute the third behavior content of "(20)" described above as the robot behavior, that is, the transmission of specific information to a person other than the user.

[0479] The behavior determining unit 236 may execute the third behavior content of "(21)" described above, that is, a gesture that attracts the user's interest, as the robot behavior.

[0480] The behavior decision unit 236 may execute at least one of the third behavior contents of "(22)" mentioned above as the robot behavior, namely, playing a sound that attracts the user's interest and playing a video that attracts the user's interest.

[0481] The behavior decision unit 236 may execute the third behavior content of the aforementioned "(23)" as a robot behavior, i.e., sending an email containing a warning message to the user's guardian, childcare worker, etc., as the transmission of specific information to a person other than the user.

[0482] The behavior decision unit 236 may execute the third behavior content of "(24)" mentioned above as a robot behavior, i.e., the delivery of an image (still image, moving image) including the user and the scenery around the user as a transmission of specific information to a person other than the user.

[0483] The behavior decision unit 236 may execute the third behavior content of "(25)" as the robot behavior, that is, the delivery of an audio warning message as the transmission of specific information to a person other than the user.

[0484] The behavior decision unit 236 may execute, as the robot behavior, at least one of the third behavior content of "(26)" mentioned above, i.e., the robot 100 swinging both arms widely and blinking the LEDs in the eyes of the robot 100 as a gesture to attract the user's interest.

[0485] Furthermore, when playing audio guiding the user to a place other than the stairs as the first action content shown in the above-mentioned "(13)", the related information collection unit 270 may store audio data guiding the user to a place other than the stairs in the collected data 223.

[0486] Furthermore, when playing audio or the like to stop the user at a location as the first action content shown in the above-mentioned "(14)", the related information collection unit 270 may store audio data to stop the user at a location in the collected data 223.

[0487] Furthermore, when at least one of a voice praising the user's behavior and a voice expressing gratitude for the user's behavior is played as the second behavior content shown in the above-mentioned "(19)", the related information collection unit 270 may store these voice data in the collected data 223.

[0488] Furthermore, the storage control unit 238 may store the above-mentioned table information in the history data 222. Specifically, the storage control unit 238 may store, in the history data 222, table information that is information that associates answers from the generative AI to questions, detected user behavior, and behavioral content for correcting the behavior.

[0489] (Outline of first action content) In particular, the action decision unit 236 detects the user's action spontaneously or periodically as the avatar's action, and when it decides to correct the user's action as the avatar's action based on the detected user's action and pre-stored specific information, it preferably causes the action control unit 250 to display the avatar in the image display area of ​​the headset-type terminal 820 so as to execute the first action content.

[0490] (Summary of second behavior content) After the behavior control unit 250 causes the avatar to perform a gesture or after the behavior control unit 250 causes the avatar to play a sound, the behavior decision unit 236 detects the user's behavior and determines whether the user's behavior has been corrected, and if the user's behavior has been corrected, it is preferable to cause the behavior control unit 250 to display the avatar in the image display area of ​​the headset-type terminal 820 so that the avatar's behavior is a second behavior content different from the first behavior content.

[0491] (Outline of third behavior content) After the behavior control unit 250 causes the avatar to perform a gesture or after the behavior control unit 250 causes the avatar to play a sound, the behavior decision unit 236 detects the user's behavior and determines whether the user's behavior has been corrected, and if the user's behavior has not been corrected, it is preferable to cause the behavior control unit 250 to display the avatar in the image display area of ​​the headset-type terminal 820 so that the avatar's behavior is a third behavior content different from the first behavior content.

[0492] The first to third action contents will be specifically described below.

[0493] In the autonomous processing of this embodiment, the behavior determination unit 236 may detect the user's state or behavior spontaneously or periodically. "Spontaneous" may be interpreted as the behavior determination unit 236 acquiring the user's state or behavior on its own initiative without any external trigger. An external trigger may include a question from the user to the avatar, an active action from the user to the avatar, etc. "Periodic" may be interpreted as a specific cycle, such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0494] The user's state may include the user's behavioral tendency. The behavioral tendency may be interpreted as the user's behavioral tendency of hyperactivity or impulsivity, such as the user frequently running up stairs, frequently climbing or attempting to climb on top of a dresser, or frequently climbing onto the edge of a window to open it. The behavioral tendency may also be interpreted as the user's tendency to engage in hyperactivity or impulsivity, such as the user frequently walking on top of a fence or attempting to climb on top of a fence, or frequently walking on a roadway or entering the roadway from the sidewalk.

[0495] In addition, in the autonomous processing, the behavior decision unit 236 may ask the generative AI a question about the detected state or behavior of the user, and may store the generative AI's answer to the question in association with the detected user behavior. At this time, the behavior decision unit 236 may store the content of an action to correct the behavior in association with the answer.

[0496] The information associating the generative AI's answer to the question, the detected user behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0497] Furthermore, in the autonomous processing, an action schedule for the avatar that calls attention to the user's state or behavior may be set based on the detected user behavior and the stored specific information.

[0498] As described above, the behavior determination unit 236 may record table information in a storage medium that associates the answers of the generative AI corresponding to the user's state or behavior with the detected user's state or behavior. Examples of the contents stored in the table are described below.

[0499] (1. When the user tends to frequently run up stairs) When this tendency is present, the behavior decision unit 236 itself asks the generative AI, "What else is a child who behaves like this likely to do?" If the generative AI answers this question with, for example, "There is a possibility that the user will trip up the stairs," the behavior decision unit 236 may store the user's behavior of running up the stairs in association with the generative AI's answer. The behavior decision unit 236 may also store, as an action of the avatar by the behavior control unit 250, an action content for correcting this behavior in association with the answer.

[0500] The behavioral content for correcting the behavior may include at least one of the following: the avatar performing a gesture to correct the user's dangerous behavior via the behavior control unit 250; and the avatar playing a sound to correct the user's behavior via the behavior control unit 250.

[0501] Gestures that correct risky behavior may include gestures and hand gestures that guide the user to a specific location, gestures and hand gestures that keep the user still at that location, etc. The specific location may include a location other than the user's current location, such as a location near the avatar, a space inside a window, etc.

[0502] The audio for correcting dangerous behavior may include audio such as "Stop it," "It's dangerous, don't move," etc. The audio for correcting dangerous behavior may include audio such as "Don't run," "Stay still," etc.

[0503] (2. When the user has a tendency to frequently be on top of the dresser or to try to climb on top of the dresser) When the user has this tendency, the behavior decision unit 236 asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "the user may fall off the dresser" or "the user may be caught in the dresser door," the behavior decision unit 236 may store the user's behavior of being on top of the dresser or trying to climb on top of the dresser in association with the generative AI's answer. The behavior decision unit 236 may also store, as the avatar's behavior, an action content that corrects the action in association with the answer.

[0504] (3. When the user has a tendency to frequently open the window by climbing up to the edge of the window) When this tendency exists, the behavior decision unit 236 asks the generative AI a question as described above. If the generative AI answers the question with, for example, "the user may stick their head out the window" or "the user may get caught in the window," the behavior decision unit 236 may store the user's behavior of opening the window by climbing up to the edge of the window in association with the generative AI's answer. The behavior decision unit 236 may also store, as the avatar's behavior, an action content for correcting this action in association with the answer.

[0505] (4. When the user has a tendency to frequently walk on or attempt to climb on top of a fence) When this tendency is present, the behavior decision unit 236 asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "the user may fall from the fence" or "the user may be injured by unevenness in the wall," the behavior decision unit 236 may store the user's behavior of walking on or attempting to climb on top of the fence in association with the generative AI's answer. The behavior decision unit 236 may also store, as the avatar's behavior, an action content for correcting this action in association with the answer.

[0506] (5. When the user has a tendency to frequently walk on the roadway or enter the roadway from the sidewalk) When this tendency exists, the behavior decision unit 236 asks the generative AI a question as described above. If the generative AI's answer to the question is, for example, "There is a possibility of a traffic accident occurring" or "There is a possibility of causing traffic congestion," the behavior decision unit 236 may store the user's behavior of walking on the roadway or entering the roadway from the sidewalk in association with the generative AI's answer. The behavior decision unit 236 may also store, as the avatar's behavior, an action content for correcting this action in association with the answer.

[0507] In this way, in autonomous processing, a table that corresponds the answer of the generative AI corresponding to the user's state or behavior, the content of the state or behavior, and the action content of the avatar that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0508] Furthermore, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the avatar to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the avatar's action determination unit 236 may cause the action control unit 250 to operate the avatar so that a first action content for correcting the user's behavior is executed based on the detected user's behavior and the contents of the stored table. An example of the first action content will be described below.

[0509] (1. When the user tends to frequently run up the stairs) When the behavior determination unit 236 detects a user running up the stairs, the behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that the avatar performs a gesture or hand gesture that guides the user to a place other than the stairs, a gesture or hand gesture that stops the user in that place, or the like, as a first behavioral content for correcting the behavior. Instead of the avatar's gesture or hand gesture, the behavior control unit 250 may transform the human-shaped avatar into a symbol that guides the user to a place other than the stairs (for example, an arrow mark indicating a direction), a symbol that stops the user in that place (for example, a "STOP" mark), or the like, and display it in the image display area of ​​the headset terminal 820.

[0510] Furthermore, the behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that it plays, as a first behavioral content for correcting the behavior, a sound in which the avatar guides the user to a place other than the stairs, a sound in which the avatar stops the user in that place, or the like. The sound may include sounds such as "XX-chan, it's dangerous, don't run," "Don't move," "Don't run," and "Stay still." Along with these sounds, the behavior control unit 250 may also display speech bubbles such as "XX-chan, it's dangerous, don't run," or "Don't move" around the mouth of the human-shaped avatar in the image display area of ​​the headset-type device 820.

[0511] (2. When the user is frequently on top of a dresser or tends to try to climb on top of a dresser) The behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that when a user is on top of a dresser or tries to climb on top of a dresser, the avatar performs a gesture and hand gesture that stops the avatar in place, or a gesture and hand gesture that moves the avatar to a location other than the current location. Instead of the avatar's gesture and hand gesture, the behavior control unit 250 may transform the human-shaped avatar into a symbol that stops the user in place (for example, a "STOP" mark), an animation that moves the user to a location other than the current location of the avatar (for example, an arrow mark that extends to indicate a direction and distance), or the like, and display it in the image display area of ​​the headset-type terminal 820.

[0512] (3. When the user tends to frequently climb up to the window edge and open the window) The behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that the avatar performs a gesture or hand gesture that stops the user who is at the window edge or is at the window edge and has his / her hands on the window, in that location, or a gesture or hand gesture that moves the avatar to a location other than the current location. Instead of the avatar's gesture or hand gesture, the behavior control unit 250 may transform the human-shaped avatar into a symbol that stops the user in place (for example, a "STOP" mark), an animation that moves the user to a location other than the current location of the avatar (for example, an arrow mark that extends to indicate a direction and distance), or the like, and display it in the image display area of ​​the headset-type terminal 820.

[0513] (4. When the user tends to frequently walk on or attempt to climb on fences) The behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that the avatar performs a gesture or hand gesture that stops the user who is walking on or attempting to climb on a fence in place, or a gesture or hand gesture that moves the avatar to a location other than the current location. Instead of the avatar's gesture or hand gesture, the behavior control unit 250 may transform the human-shaped avatar into a symbol that stops the user in place (for example, a "STOP" mark), an animation that moves the user to a location other than the current location of the avatar (for example, an arrow mark extending to indicate a direction and distance), or the like, and display it in the image display area of ​​the headset-type terminal 820.

[0514] (5. When the user tends to frequently walk on the roadway or enter the roadway from the sidewalk) The behavior determination unit 236 may cause the behavior control unit 250 to operate the avatar so that when a user is walking on the roadway or has entered the roadway from the sidewalk, the avatar performs a gesture or hand gesture that stops the avatar in place, or a gesture or hand gesture that moves the avatar to a location other than the current location. Instead of the avatar's gesture or hand gesture, the behavior control unit 250 may transform the human-shaped avatar into a symbol that stops the user in place (for example, a "STOP" mark), an animation that moves the user to a location other than the current location of the avatar (for example, an arrow mark extending to indicate a direction and distance), or the like, and display it in the image display area of ​​the headset-type terminal 820.

[0515] The behavior decision unit 236 detects the user's behavior after the avatar performs a gesture that is the first behavior content, or after the avatar plays back a sound that is the first behavior content, and determines whether the user's behavior has been corrected.If the user's behavior has been corrected, the behavior control unit 250 may cause the avatar to operate so that the avatar performs a second behavior content that is different from the first behavior content.

[0516] A case where the user's behavior is corrected may be interpreted as a case where the user stops dangerous behavior and actions as a result of the avatar's movement being performed according to the first behavior content, or a case where the user's dangerous situation is resolved.

[0517] The second action content may include reproduction by the action control unit 250 of at least one of a voice in which the avatar praises the user's action and a voice in which the avatar thanks the user for the action.

[0518] The voice praising the user's behavior may include voice such as "Are you OK? You listened well," or "Good job, amazing." The voice thanking the user for their behavior may include voice such as "Thanks for coming." Along with these voices, the behavior control unit 250 may also display speech bubbles around the mouth of the human-shaped avatar, such as "Are you OK? You listened well," or "Good job, amazing," in the image display area of ​​the headset-type terminal 820.

[0519] The behavior decision unit 236 detects the user's behavior after the avatar performs a gesture that is the first behavior content, or after the avatar plays back a sound that is the first behavior content, and determines whether the user's behavior has been corrected.If the user's behavior has not been corrected, the behavior control unit 250 may cause the avatar to perform a third behavior content that is different from the first behavior content as the avatar's behavior.

[0520] A case where the user's behavior is not corrected may be interpreted as a case where the user continues to engage in dangerous behavior and actions despite the avatar's movements being performed according to the first behavior content, or a case where the dangerous situation is not resolved.

[0521] The third action content may include at least one of sending specific information to a person other than the user, the action control unit 250 causing the avatar to perform a gesture that attracts the user's interest, playing a sound that attracts the user's interest, and playing a video that attracts the user's interest.

[0522] The transmission of specific information to a person other than the user may include the delivery of an email containing a warning message to the user's guardian, childcare worker, etc., the delivery of an image (still image, moving image) containing the user and the scenery around him / her, etc. The transmission of specific information to a person other than the user may also include the delivery of an audio warning message.

[0523] The gestures of the avatar that attract the user's interest may include body and hand gestures of the avatar made by the behavior control unit 250. Specifically, these may include making the avatar swing both arms widely or blinking the LEDs in the avatar's eyes made by the behavior control unit 250. Instead of making the avatar make gestures and hand gestures, the behavior control unit 250 may transform the human-shaped avatar into an animal, a character in a popular anime, a popular local character, or the like to attract the user's interest.

[0524] The playing of sounds that interest the user may include particular music that the user likes, and may also include sounds such as "come here" and "let's play together."

[0525] Playback of videos that interest the user may include images of the user's pet animals, images of the user's parents, and so on.

[0526] According to the present disclosure, autonomous processing can detect whether a child or other user is attempting to engage in dangerous behavior (e.g., climbing onto the window edge to open it), and if danger is detected, autonomously execute an action to correct the user's behavior. As a result, the avatar controlled by the behavior control unit 250 can autonomously execute gestures and speech such as "Stop it" or "Hey, it's dangerous, come over here." Furthermore, if a child stops the dangerous behavior in response to a call, the avatar controlled by the behavior control unit 250 can also perform an action to praise the child, such as "Are you okay? You listened well." Furthermore, if the child does not stop the dangerous behavior, the avatar controlled by the behavior control unit 250 can send a warning email to the parent or caregiver, share the situation via video, and perform an action that the child is interested in, play a video that the child is interested in, or play music that the child is interested in, thereby encouraging the child to stop the dangerous behavior.

[0527] [Thirteenth Embodiment] In the autonomous processing of this embodiment, the robot 100 as an agent autonomously and periodically detects the state of the user. More specifically, the robot 100 autonomously and periodically detects whether the user and his / her family are using a social networking service (hereinafter referred to as SNS). That is, the robot 100 constantly monitors the displays of smartphones, etc., owned by the user and his / her family, and detects the state of SNS use. If the user is a child, the robot 100 autonomously converses with the child to consider how to interact with SNS and what content to post.

[0528] For example, the multiple types of robot behaviors include the following (1) to (11).

[0529] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot gives the user advice about social media.

[0530] When the behavior decision unit 236 decides to set the robot behavior as "(11) The robot gives the user advice about SNS," that is, to give the user advice about SNS, the robot 100 uses the sentence generation model to decide what the robot will say corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the decided robot utterance content to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the decided robot utterance content in the behavior schedule data 224 without outputting the sound representing the decided robot utterance content.

[0531] Specifically, the robot 100 considers and suggests ways to use the SNS and content to post on the SNS so that the user can use the SNS safely and appropriately, while having a conversation with the user. For example, the robot 100 suggests to the user one or a combination of information security measures, protection of personal information, prohibition of slander, prohibition of spreading false information, and compliance with the law as ways to use the SNS. As a specific example, in a conversation with the user, the robot 100 can suggest ways to use the SNS, such as "You should be careful not to disclose your personal information on the Internet!" in response to a question such as "What should I be careful about when using the SNS?"

[0532] On the other hand, the robot 100 suggests to the user content to post that satisfies predetermined conditions including one or a combination of information security measures, protection of personal information, prohibition of slander, prohibition of spreading false information, and compliance with the law. As a specific example, in response to an utterance in a conversation with a user saying, "I want to post about A and B that will not cause an uproar," the robot 100 can think of content to post that will not slander both parties, such as "A and B are both great!", and suggest it to the user.

[0533] Furthermore, when the robot 100 recognizes the user as a minor, it suggests to the user during a conversation one or both of a way of using SNS and content to post on SNS that is appropriate for minors. Specifically, the robot 100 can make suggestions regarding the way of using SNS and content to post on SNS based on stricter conditions for minors. As a specific example, in a conversation with a minor user, in response to a question such as "What should I be careful about when using SNS?", the robot 100 can suggest a way of using SNS, such as "Be careful not to disclose personal information, slander, or spread rumors (false information)!". In addition, in a conversation with a minor user, in response to an utterance such as "I want to post about A and B without causing a firestorm," the robot 100 can suggest to the user a post that is polite and does not slander both parties, such as "I think both A and B are wonderful."

[0534] Furthermore, as the behavior of making suggestions regarding the use of the SNS described above, the robot 100 can make a speech about the content posted by the user on the SNS when the user has finished posting to the SNS. For example, after the user has finished posting to the SNS, the robot 100 can spontaneously make a speech such as, "This post shows that you are well aware of how to approach the SNS, so it's 100 points!"

[0535] Furthermore, the robot 100 can analyze the content posted by the user and, based on the analysis results, make suggestions to the user about how to approach SNS or how to create content to post. For example, if there is no utterance from the user, the robot 100 can make utterances such as "The content of this post contains factual information that is not true and may become a hoax (false information), so be careful!" based on the content posted by the user.

[0536] Furthermore, the robot 100 makes suggestions to the user in a conversational format about one or both of how to use the SNS and what to post on the SNS based on the user's state and behavior. For example, when the robot 100 recognizes that the user is holding a terminal device and that "the user seems to be having trouble using the SNS," the robot 100 can talk to the user in a conversational format and suggest how to use the SNS, how to use the SNS, and what to post.

[0537] Regarding "(11) The robot gives the user advice on SNS," the related information collecting unit 270 acquires information on SNS. For example, the related information collecting unit 270 may periodically access information sources such as television or the web to voluntarily collect information on laws, incidents, problems, etc. related to SNS and store the information in the collected data 233. This allows the robot 100 to acquire the latest information on SNS, and therefore, to voluntarily give the user advice on the latest problems, etc. related to SNS.

[0538] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0539] In particular, as in the first embodiment described above, when the behavior decision unit 236 decides that the avatar's behavior is to give the user advice regarding SNS, it is preferable to have the behavior control unit 250 control the avatar to give the user advice regarding SNS using the output of the behavior decision model 221.

[0540] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0541] Furthermore, as in the first embodiment, when the behavior determination unit 236 determines to provide SNS-related advice to the user using the output of the behavior determination model 221 as the avatar's behavior, the behavior determination unit 236 may control the behavior control unit 250 to change at least one of the avatar's type, voice, and facial expression depending on the user to whom the advice is to be given. The avatar may be modeled after a real person, a fictional character, or a character. Specifically, types of avatars that provide SNS-related advice include parents, older brothers or sisters, school teachers, celebrities, etc. However, if the user to whom the advice is to be given is a minor or a child, the behavior control unit 250 may be controlled to change the avatar to an avatar that provides gentler advice, such as a grandmother, a kind-looking older sister, or a character the user likes, or to an avatar with a gentler voice or an avatar that speaks with a gentle, smiling expression. Furthermore, as in the first embodiment described above, when the behavior decision unit 236 decides to use the output of the behavior decision model 221 to give the user advice regarding SNS as the behavior of the avatar, it may control the behavior control unit 250 to transform the avatar into an animal other than a human, such as a dog, cat, etc.

[0542] [Fourteenth Embodiment] In this embodiment, user 10a, user 10b, user 10c, and user 10d constitute a family, as an example. In other words, user 10a, user 10b, user 10c, and user 10d constitute a family. In addition, users 10a to 10d may include caregivers who provide care. For example, if user 10a is a caregiver, he or she may provide care for a person (user) other than a family member, or may provide care for user 10b, who is a family member. As an example, user 10a is the caregiver, and user 10b is the care recipient.

[0543] As will be described later, the robot 100 provides the user 10 with advice information regarding caregiving, but if the user 10a who is the caregiver is caring for someone other than a family member, the user 10 at this time does not have to be a family member. If the user 10b who is the care recipient is receiving care from someone (a user) other than a family member, the user 10 at this time does not have to be a family member. Also, as will be described later, the robot 100 provides the user 10 with advice information regarding the health and mental state of the family members, but the user 10 at this time does not have to include the caregiver or the care recipient.

[0544] The robot 100 according to the present embodiment can provide advice information related to caregiving. The robot 100 provides advice information related to caregiving to the user 10, including the caregiver and the care recipient, but is not limited to this, and may provide the advice information to any user, such as a family member including at least one of the caregiver and the care recipient.

[0545] Specifically, the robot 100 recognizes the physical and mental conditions of the user 10, including at least one of the caregiver and the care recipient. The physical and mental conditions of the user 10 here include, for example, the level of stress and fatigue of the user 10. The robot 100 provides advice information regarding care according to the recognized physical and mental conditions of the user 10.

[0546] As an example, when the robot 100 estimates that the level of stress of the user 10 is relatively high or that the level of fatigue of the user 10 is relatively high based on the behavior of the user 10, the robot 100 performs an action of starting a conversation with the user 10. Specifically, the robot 100 makes an utterance indicating that advice information will be provided, such as "I have some advice for you about caregiving."

[0547] Next, the robot 100 generates advice information related to caregiving based on the recognized physical and mental state of the user 10 (here, the level of stress, fatigue, etc.). The advice information includes, but is not limited to, information related to the physical and mental recovery of the user 10, such as how to maintain motivation for caregiving, how to relieve stress, and relaxation methods. Here, the robot 100 provides advice information in accordance with the physical and mental state of the user 10 by speaking, for example, "It seems that stress (fatigue) is building up. I recommend that you move your body by stretching, etc."

[0548] As described above, in this embodiment, the robot 100 recognizes the physical and mental states of the user 10, including the caregiver, and performs an action corresponding to the recognized physical and mental state, thereby providing appropriate advice regarding care to the user 10. In other words, the robot 100 can understand the stress and fatigue of the user 10 and provide appropriate advice information such as relaxation methods and stress relief methods. That is, the robot 100 according to this embodiment can perform appropriate actions for the user 10.

[0549] Furthermore, when the control unit of the robot 100 recognizes the physical and mental state of the user 10, including at least one of the caregiver and the care recipient, the control unit determines, as its own behavior, an action to provide advice information on care according to the recognized state. This allows the robot 100 to provide appropriate advice information on care that is in line with the physical and mental state of the user 10, including the caregiver and the care recipient.

[0550] Furthermore, when the control unit of the robot 100 recognizes at least one of the stress level and the fatigue level of the user 10 as the mental and physical state of the user 10, the control unit generates information regarding the mental and physical recovery of the user 10 as advice information based on at least one of the recognized stress level and the fatigue level. This allows the robot 100 to provide information regarding the mental and physical recovery of the user 10 as advice information that is in line with the stress level and the fatigue level of the user 10.

[0551] The storage unit 220 includes history data 222. The history data 222 includes the user 10's past emotional values ​​and behavioral history. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the user 10's identification information. The history data 222 may also include user information for each of multiple users 10 associated with the user 10's identification information. The user information may include information indicating that the user 10 is a caregiver, information indicating that the user 10 is a care recipient, information indicating that the user 10 is neither a caregiver nor a care recipient, and the like. The user information indicating whether the user 10 is a caregiver may be estimated from the user 10's behavioral history or may be registered by the user 10 himself / herself. The user information may also include information indicating the user 10's characteristics, such as the user's personality, interests, hobbies, and inclinations. The user information indicating the user 10's characteristics may be estimated from the user 10's behavioral history or may be registered by the user 10 himself / herself. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. It may also include a person DB that stores facial images of users 10, attribute information of users 10, etc.

[0552] The state recognition unit 230 recognizes the physical and mental state of the user 10 based on information analyzed by the sensor module unit 210, etc. For example, if the state recognition unit 230 determines, based on the user information, that the recognized user 10 is a caregiver or a care recipient, it recognizes the physical and mental state of the user 10. Specifically, the state recognition unit 230 estimates the level of stress of the user 10 based on various information such as the user's 10's behavior, facial expression, voice, and text information representing the content of their speech, and recognizes the estimated level of stress as the user's physical and mental state. As an example, if various information (such as feature quantities such as voice frequency components and text information) includes information indicating stress, the user state recognition unit 230 estimates that the user 10's level of stress is relatively high. Furthermore, the user state recognition unit 230 estimates the level of fatigue of the user 10 based on various information such as the behavior, facial expression, voice, and text information representing the content of speech of the user 10, and recognizes the estimated level of fatigue as the physical and mental state of the user 10. As an example, if information indicating accumulated fatigue is included in the various information (feature quantities such as frequency components of voice, text information, etc.), the user state recognition unit 230 estimates that the level of fatigue of the user 10 is relatively high. Note that the above-mentioned levels of stress and fatigue may be registered by the user 10 themselves.

[0553] The state recognition unit 230 may recognize both the level of stress and the level of fatigue, or may recognize only one of them. In other words, it is sufficient for the state recognition unit 230 to recognize at least one of the level of stress and the level of fatigue.

[0554] Furthermore, the state recognition unit 230 recognizes the mental and physical states of each of the multiple users 10 constituting a family based on information analyzed by the sensor module unit 210, etc. Specifically, the state recognition unit 230 estimates the health state of the user 10 based on various information such as text information representing the user 10's behavior, facial expression, voice, and speech content, and recognizes the estimated health state as the user 10's mental and physical state. As an example, if the various information (such as text information) includes information indicating a good health state, the state recognition unit 230 estimates that the user 10's health state is good, whereas if the various information includes information indicating a poor health state, the state recognition unit 230 estimates that the user 10's health state is poor. Furthermore, the user state recognition unit 230 estimates the user 10's lifestyle habits based on various information such as text information representing the user 10's behavior, facial expression, voice, and speech content, and recognizes the estimated lifestyle habits as the user 10's mental and physical state. As an example, when information indicating lifestyle habits (such as dietary content and exercise habits) is included in various information (such as text information), the state recognition unit 230 estimates the lifestyle habits of the user 10 from such information. Note that the above-mentioned health condition, lifestyle habits, etc. may be registered by the user 10 themselves.

[0555] The condition recognition unit 230 may recognize both the health condition and the lifestyle habits, or may recognize only one of them. That is, the condition recognition unit 230 may recognize at least one of the health condition and the lifestyle habits.

[0556] Furthermore, the state recognition unit 230 recognizes the mental state of each of the multiple users 10 constituting a family as a physical and mental state of the user 10 based on information analyzed by the sensor module unit 210, etc. Specifically, the state recognition unit 230 estimates the mental state of the user 10 based on various information such as the user's 10 behavior, facial expression, voice, and text information representing the content of their speech, and recognizes the estimated mental state as the user's 10 mental and physical state. As an example, if various information (features such as voice frequency components and text information) includes information indicating a mental state such as depression or tension, the user state recognition unit 230 estimates the user 10's mental state from such information. Note that the above-mentioned mental states may be registered by the user 10 themselves.

[0557] Furthermore, for example, the reaction rules prescribe behaviors of the robot 100 corresponding to behavior patterns such as when the mental and physical state (stress level or fatigue level) of the user 10, including the caregiver and the care recipient, requires care-related advice for the user 10, or when the user 10 responds to the provided advice information. For example, when it is estimated based on the reaction rules that the stress level of the user 10, including the caregiver and the care recipient, is relatively high, or when it is estimated that the fatigue level is relatively high, the behavior determining unit 236 determines, as its own behavior, to be an action of providing the user 10 with care-related advice information corresponding to the mental and physical state of the user 10.

[0558] When the behavior control unit 250 recognizes the physical and mental state of the user 10, including the caregiver and the care recipient, it determines its own behavior to be an action that provides advice information regarding care that corresponds to the physical and mental state of the user 10, and controls the control object 252.

[0559] In more detail, when it is estimated that the level of stress of the user 10 is relatively high, or when it is estimated that the level of fatigue of the user 10 is relatively high, the behavior control unit 250 executes an action of starting a conversation with the user 10. Specifically, the behavior control unit 250 makes an utterance indicating that advice information will be provided, such as "I have some advice for you about caregiving."

[0560] Next, the behavior control unit 250 generates advice information regarding caregiving based on the recognized physical and mental state of the user 10 (such as the level of stress and fatigue), and provides the generated advice information by speech. The advice information includes, but is not limited to, information on mental and physical recovery of the user 10 by providing psychological support to the user 10, such as methods for maintaining motivation for caregiving, methods for relieving stress, and relaxation methods. For example, the behavior control unit 250 provides advice information by speech that is in line with the physical and mental state of the user 10, such as "You seem to be stressed. I recommend that you move your body by stretching, etc." or "You seem to be tired. I recommend that you get enough sleep."

[0561] In this way, the behavior control unit 250 according to this embodiment recognizes the physical and mental states of the user 10, including the caregiver, and performs an action corresponding to the recognized physical and mental state, thereby providing appropriate advice regarding care to the user 10. In other words, the behavior control unit 250 can understand the stress and fatigue of the user 10 and provide appropriate advice information such as relaxation methods and stress relief methods.

[0562] The behavior control unit 250 may also provide, as advice information, information on laws and systems related to nursing care. The information on laws and systems related to nursing care corresponds to the nursing care status (nursing care level) of the care recipient, and is obtained by the communication processing unit 280 from an external server (not shown) or the server 300 via a communication network 20 such as the Internet, for example, but is not limited to this.

[0563] Furthermore, since the emotion determination unit 232 determines the emotion value of the robot 100, the behavior control unit 250 may provide, based on such emotion value, advice information by speech that is in line with the feelings (emotions) of the caregiver, user 10a, such as, for example, "Caring for the caregiver is difficult, but user 10b seems to be very grateful (pleased)."

[0564] In the autonomous processing of this embodiment, the robot 100 as an agent voluntarily and periodically detects the state of the user 10 for whom it is providing care. For example, the robot 100 constantly monitors the caregivers and constantly detects their fatigue level and happiness. If the robot 100 determines that the user 10's fatigue level or motivation is decreasing, it takes action to improve motivation and relieve stress. Specifically, the robot 100 understands the user 10's stress and fatigue and suggests appropriate relaxation methods and stress relief measures to the user 10. If the caregiver's happiness level is increasing, the robot 100 voluntarily praises or offers words of encouragement to the caregiver. Furthermore, the robot 100 voluntarily and periodically collects information on laws and systems related to caregiving, for example, from external data (websites such as news sites and video sites, distributed news, etc.), and if the importance level of the information exceeds a certain value, it voluntarily provides the collected information related to caregiving to the caregiver (user).

[0565] For example, the multiple types of robot behaviors include the following (1) to (11).

[0566] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot gives the user advice on caregiving.

[0567] When the behavior decision unit 236 decides to set the robot behavior as "(11) Provide care-related advice to the user," i.e., to provide necessary information to the user who is involved in care-giving, the robot 100 obtains the necessary information from external data, for example. The robot 100 always obtains this information autonomously even when the user is not present.

[0568] Regarding "Provide the user with advice on caregiving," the related information collection unit 270 collects information on the user's caregiving, for example, as information on the user's preferences, and stores it in the collected data 223. This information is then output as audio from a speaker or displayed as text on a dis...

Claims

1. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes dreaming, and when the behavior determination unit determines that the behavior of the avatar is dreaming, it creates an original event combining a plurality of event data from the history data.

2. A behavior control system as described in claim 1, wherein, when the behavior decision section decides that the avatar's behavior is to have a dream, it causes the behavior control section to control the avatar so as to generate the original event.

3. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting an activity, and when the behavior determination unit decides to suggest the activity as the behavior of the avatar, it determines the proposed behavior of the user based on the event data.

4. The behavior control system of claim 3, wherein when the behavior determination unit decides to propose the activity as the behavior of the avatar, the behavior determination unit uses the event data to determine the behavior of the user to be proposed, so as to propose at least one of an action that the user has enjoyed in the past, an action that the user is likely to prefer based on the user's tastes and preferences, and a new action that the user has not experienced in the past.

5. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes comforting the user, and when the behavior determination unit determines that the behavior of the avatar is to comfort the user, it determines speech content corresponding to the user state and the user's emotion.

6. A behavior control system according to claim 5, wherein, when the behavior decision unit decides that the behavior of the avatar is to comfort the user, the behavior decision unit listens to what the user has to say and causes the avatar to act in a comforting manner.

7. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question to the user, and when the behavior determination unit determines to ask a question to the user as the behavior of the avatar, creates a question to be asked to the user.

8. A behavior control system according to claim 7, wherein said behavior decision unit changes the question setting tendency and creates new questions when it is inferred that the subject is not interested in the questions set.

9. A behavior control system as described in claim 7, wherein when the behavior decision unit determines that the avatar's behavior is to pose a question to the user, it causes the avatar to operate so as to pose the created question to the user.

10. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes teaching music, and when the behavior determination unit determines that the avatar's behavior is to teach music, it evaluates a sound produced by the user.

11. The behavior control system according to claim 10, wherein the behavior determination unit evaluates at least one of the user's singing voice, musical instrument sound, sense of rhythm, pitch, and intonation.

12. The behavior control system of claim 10, wherein when the behavior decision unit decides that the behavior of the avatar is to teach music, it causes the avatar to speak the results of evaluating the sound produced by the user.

13. A behavior control system comprising: a state recognition unit that recognizes a user state including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question to the user, and when the behavior determination unit decides to ask the user a question as the avatar behavior, it asks a question suited to the user based on the content of a text used by the user and the user's target deviation value.

14. A behavior control system as described in claim 13, wherein the behavior decision unit, when determining that the user's emotions indicate that the user is in a state where he or she appears to be bored or has been scolded by the user's parent / guardian to study, presents questions suited to the user.

15. A behavior control system according to claim 13, wherein said behavior decision section presents a question with a more difficult answer when the user is able to answer the question posed.

16. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice regarding a specific competition to the user participating in the specific competition, the behavior determination unit comprising: an image acquisition unit that is capable of capturing an image of a competition space in which the specific competition in which the user participates is being held; and a feature identification unit that identifies features of a plurality of athletes participating in the specific competition in the competition space captured by the image acquisition unit, and when it is determined that the behavior of the avatar is to give advice regarding the specific competition to the user participating in the specific competition, the advice is given to the user based on the identification result of the feature identification unit.

17. A behavior control system as described in claim 16, wherein when the behavior decision unit determines that the behavior of the avatar is to provide advice regarding the specific competition to the user participating in the specific competition, the behavior decision unit executes an operation that causes the avatar to reflect the characteristics identified by the characteristic identification unit of a specific athlete among the multiple athletes.

18. The behavior control system of claim 16, wherein when the behavior decision unit determines that the behavior of the avatar is to provide advice regarding the specific sport to the user participating in the specific sport, the behavior decision unit executes an operation to cause the avatar to reflect information about the uniform to be worn during the specific sport.

19. A state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes setting a first behavior content that corrects the user's behavior, The behavior control system includes a behavior decision unit that detects the user's behavior either voluntarily or periodically, and when it decides to correct the user's behavior as the avatar's behavior based on the detected user's behavior and pre-stored specific information, it causes the behavior control unit to display the avatar in an image display area so as to execute the first behavior content.

20. The behavior control system of claim 19, wherein the first behavior content includes at least one of the following: the avatar performing a gesture to correct the user's behavior via the behavior control unit, and the avatar playing a sound to correct the user's behavior via the behavior control unit.

21. The behavior control system of claim 20, wherein the behavior decision unit detects the user's behavior after the avatar executes the gesture by the behavior control unit or after the avatar plays the sound by the behavior control unit, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, causes the behavior control unit to display the avatar in an image display area so as to execute a second behavior content different from the first behavior content as the behavior of the avatar.

22. The behavior control system of claim 21, wherein the second behavior content includes at least one of a voice in which the avatar praises the user's behavior via the behavior control unit, and a voice in which the avatar thanks the user for the user's behavior via the behavior control unit.

23. The behavior control system of claim 20, wherein the behavior decision unit detects the user's behavior after the avatar executes the gesture by the behavior control unit or after the avatar plays the sound by the behavior control unit, thereby determining whether the user's behavior has been corrected, and if the user's behavior has not been corrected, causes the behavior control unit to display the avatar in an image display area so as to execute a third behavior content different from the first behavior content as the behavior of the avatar.

24. The behavior control system of claim 23, wherein the third behavior content includes at least one of the following: sending specific information to a person other than the user; performing a gesture by the avatar to attract the user's interest via the behavior control unit; playing a sound to attract the user's interest; and playing a video to attract the user's interest.

25. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice to the user regarding a social networking service, and when the behavior determination unit determines that the behavior of the avatar is to give advice to the user regarding a social networking service, the behavior control unit gives the advice to the user regarding the social networking service.

26. The behavior control system of claim 25, wherein when the behavior decision unit determines that the behavior of the avatar is to provide advice to a user regarding social working services, it operates the avatar to change at least one of the avatar's type, voice, and facial expression depending on the user to whom the advice is to be given.

27. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving care advice to the user, and when the behavior determination unit determines to give care advice to the user as the behavior of the avatar, it collects information regarding the user's care, and gives the user care advice from the collected information.

28. A behavior control system as described in claim 27, wherein when the behavior decision unit determines that the behavior of the avatar is to provide the user with advice regarding care, and if the user is a caregiver, the behavior decision unit causes the avatar to act in a manner that includes a behavior of praising the user.

29. A behavior control system as described in claim 27, wherein when the behavior decision unit determines that the behavior of the avatar is to give the user care-related advice, it causes the avatar to act in a manner that includes behavior that increases the user's motivation and relieves stress.

30. A behavior control system as described in claim 27, wherein when the behavior decision unit decides that the behavior of the avatar is to provide the user with care advice, and the care advice is a physical care technique, the avatar is caused to act in a manner to demonstrate the care technique.

31. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice to the user regarding an approaching risk, and when the behavior determination unit determines that the behavior of the avatar is to give advice to the user regarding an approaching risk, the behavior control unit gives the advice to the user regarding the approaching risk.

32. A behavior control system as claimed in claim 31, wherein when the behavior decision unit determines that the behavior of the avatar is to advise the user regarding an approaching risk, the behavior decision unit causes the avatar to transform into another avatar.

33. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving health advice to the user, and when the behavior determination unit determines that the behavior of the avatar is to give health advice to the user, the behavior control unit gives the health advice to the user.

34. A behavior control system as described in claim 33, wherein when the behavior decision unit decides that the avatar's behavior is to assist in dieting, it causes the avatar to operate so as to change its appearance to that of a virtual user who has achieved an ideal body type.

35. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes autonomously converting a statement by the user into a question, and when the behavior determination unit determines that the behavior of the avatar is to convert the statement by the user into a question and answer the question, it converts the statement by the user into a question using a sentence generation model based on the event data and answers the question.

36. A behavior control system according to claim 35, wherein when said behavior decision section decides that the behavior of said avatar is to transform into another avatar having a different appearance, said behavior decision section operates the avatar so as to transform into said another avatar.

37. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behaviors include increasing vocabulary and speaking about the increased vocabulary, and when the behavior determination unit determines that the behavior of the avatar is to increase vocabulary and speak about the increased vocabulary, the behavior control unit increases the vocabulary and speaks about the increased vocabulary.

38. A behavior control system as described in claim 37, wherein when the behavior decision unit decides that the behavior of the avatar is to increase the vocabulary and speak using the increased vocabulary, it operates the avatar to change at least one of the avatar's face, body, and voice in accordance with the number of increased vocabulary words.

39. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes learning an utterance method and changing a setting of the utterance method, and when the behavior determination unit determines that the avatar's behavior is to learn the utterance method, it collects utterances of speakers in a predetermined information source, and when it determines that the avatar's behavior is to change the setting of the utterance method, it changes the voice to be spoken depending on the attributes of the user.

40. A behavior control system as described in claim 39, wherein the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar.

41. A behavior control system as described in claim 40, wherein the behavior decision model is a sentence generation model having an interactive function, and the behavior decision unit inputs text expressing at least one of the user state, the state of the avatar displayed in the image display area, the emotion of the user, and the emotion of the avatar displayed in the image display area, as well as text asking about the behavior of the avatar, into the sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

42. A behavior control system as described in claim 39, wherein when the behavior control unit decides to change the speech method setting as the behavior of the avatar, it causes the avatar to act with an appearance corresponding to the changed voice.

43. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no operation as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior of the avatar includes learning an utterance method and changing a setting of the utterance method, and when the behavior determination unit determines that the avatar's behavior is to learn the utterance method, it collects utterances of speakers in a predetermined information source, and when it determines that the avatar's behavior is to change the setting of the utterance method, it changes the voice to be spoken depending on an attribute of the user.

44. A behavior control system as described in claim 43, wherein the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar.

45. A behavior control system as described in claim 44, wherein the behavior decision model is a sentence generation model having an interactive function, and the behavior decision unit inputs text expressing at least one of the user state, the state of the avatar displayed in the image display area, the emotion of the user, and the emotion of the avatar displayed in the image display area, as well as text asking about the behavior of the avatar, into the sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

46. ​​A behavior control system as described in claim 43, wherein when the behavior control unit determines to change the speech method setting as the behavior of the avatar, it causes the avatar to act with an appearance corresponding to the changed voice.

47. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes taking into account the user's mental age, and when the behavior determination unit has determined to take into account the user's mental age as the avatar behavior, it estimates the user's mental age and determines the avatar behavior in accordance with the estimated mental age of the user.

48. A behavior control system as described in claim 47, wherein when the behavior control unit determines that the mental age of the user should be taken into consideration in the behavior of the avatar, the behavior control unit changes the appearance of the avatar according to the estimated mental age of the user.

49. A behavior control system as described in claim 48, wherein when the behavior control unit determines that the mental age of the user should be taken into consideration in the behavior of the avatar, the behavior control unit causes the appearance of the avatar to grow in accordance with the estimated mental age of the user.

50. A behavior control system as described in claim 48, wherein when the behavior control unit decides to take the user's mental age into account in the behavior of the avatar, the behavior control unit switches the avatar to another avatar with a different appearance depending on the estimated mental age of the user.

51. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes estimating the user's foreign language level and conversing with the user in the foreign language, and when the behavior determination unit has determined to estimate the user's foreign language level as the behavior of the avatar, it estimates the user's foreign language level, and when it has determined to converse with the user in the foreign language, it converses with the user in the foreign language.

52. A behavior control system as described in claim 51, wherein when the behavior decision unit decides that the avatar's behavior is to converse with the user in a foreign language, it causes the avatar to act so as to change its appearance to that of a person who speaks that foreign language.

53. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice to the user regarding the user's creative activities, and when the behavior determination unit determines that the behavior of the avatar is to give advice to the user regarding the user's creative activities, it collects information regarding the user's creative activities and gives the advice regarding the user's creative activities from the collected information.

54. A behavior control system as described in claim 53, wherein when the behavior decision unit determines that the avatar's behavior is to give the advice to the user, the behavior decision unit causes the avatar to operate so as to give the advice including an action of praising the user's creative activity.

55. A behavior control system as described in claim 53, wherein when the behavior decision section decides to give the advice to the user as the avatar's behavior, it decides the content of the next advice based on the content of past advice.

56. A behavior control system according to claim 53, wherein said behavior determining section collects information regarding the user's creative activities even when said user is absent.

57. A behavior control system according to claim 53, wherein said advice by said behavior decision section includes hints for bringing out the user's creativity.

58. A behavior control system according to claim 53, wherein said advice by said behavior decision section includes a hint that draws out the expressiveness of said user.

59. A system for controlling a user's behavior including a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar that represents an agent for interacting with the user; a behavior determination unit that determines, at a predetermined timing, one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes making suggestions to encourage behavior that the user can take at home, and the memory control unit stores types of behaviors performed by the user at home in the history data in association with the timing at which the behaviors were performed, A behavior control system in which, when the behavior decision unit, based on the historical data, either spontaneously or periodically determines a suggestion to encourage a behavior that the user at home can take as the avatar's behavior, the behavior control unit displays the avatar in the image display area so that the suggestion to encourage the behavior is executed at the time when the user should execute the behavior.

60. The behavior control system of claim 59, wherein the behavior decision unit estimates the interval between nail clippings based on the timing when the user at home clips their nails, and when the estimated number of days has elapsed, causes the behavior control unit to display the avatar in the image display area to prompt the user to clip their nails.

61. The behavior control system of claim 59, wherein the behavior decision unit estimates the watering interval based on the timing when the user in the home waters plants, and causes the behavior control unit to operate the avatar so as to prompt the user to water the plants when the estimated number of days has passed.

62. The behavior control system of claim 59, wherein the behavior decision unit estimates the interval for cleaning the toilet based on the timing at which the user in the home cleaned the toilet, and when the estimated number of days has elapsed, causes the behavior control unit to display the avatar in the image display area so as to prompt the user to clean the toilet.

63. The behavior control system of claim 59, wherein the behavior decision unit estimates the next timing for the user to get ready based on the timing when the user at home gets ready, and causes the behavior control unit to display the avatar in the image display area so as to suggest to the user to start getting ready at the estimated timing.

64. A behavior control system comprising: a user state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes the electronic device making an utterance or a gesture to the user, and the behavior determination unit determines the content of the utterance or the gesture so as to support the user's learning based on the sensory characteristics of the user, and causes the behavior control unit to control the avatar.

65. A behavior control system according to claim 64, wherein the behavior determination unit determines a question to be posed by the avatar to the user, and the behavior control unit causes the avatar to pose the question.

66. The behavior control system of claim 65, wherein the behavior decision unit decides on speech content to encourage the user based on the user's state and the user's emotions until an answer from the user is obtained, and the behavior control unit causes the avatar to speak the speech content decided by the behavior decision unit.

67. The behavior control system of claim 64, wherein when the behavior determination unit has determined the content of the support for the user, the behavior determination unit changes the display mode of the avatar to a predetermined display mode, and the behavior control unit displays the avatar in the changed display mode.

68. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit obtains musical scores of lyrics and melody appropriate to the environment in which the electronic device is located based on the behavior determination model, and determines the behavior content of the avatar to play music based on the lyrics and melody using a voice synthesis engine, sing along with the music, and / or dance along with the music.

69. A behavior control system according to claim 68, wherein said behavior control unit controls said avatar to play said music, sing along with said music, and / or dance along with said music.

70. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that determines the behavior of the avatar based on at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein when the behavior determination unit determines that the behavior of the avatar is to answer a user's question, it obtains a vector representing the user's question, searches a database storing combinations of questions and answers for a question having a vector corresponding to the obtained vector, and generates an answer to the user's question using an answer to the searched question and a sentence generation model capable of generating sentences according to input data.

71. A behavior control system as described in claim 70, wherein when the behavior control unit determines that the avatar's behavior is to answer a user's question, it causes the avatar to act in an appearance corresponding to the question or the answer.

72. An information processing system comprising: an input unit that accepts user input; a processing unit that performs specific processing using a sentence generation model that generates sentences in accordance with the input data; an output unit that controls the behavior of the electronic device to output the results of the specific processing; and a behavior control unit that displays an avatar in an image display area of ​​the electronic device, wherein when pitching information regarding the next ball to be thrown by a specific pitcher is requested, the processing unit generates, as the specific processing, a sentence that instructs the creation of the pitching information accepted by the input unit and inputs the generated sentence into the sentence generation model, and causes the output unit to output the pitching information created as a result of the specific processing to the avatar representing an agent for interacting with a user.

73. An information processing system according to claim 72, wherein the pitch information includes pitch type information and ball trajectory information.

74. An information processing system as described in claim 72, wherein the pitch information includes an image of a pitch thrown by a displayed avatar, the image showing the pitch trajectory, including at least the type of ball, ball speed, and course, from the batter's perspective.

75. An information processing system as described in claim 72, wherein the input unit accepts input of the specific pitcher from a user, and the processing unit uses a model that has been trained on the past pitching history of the specific pitcher input as the sentence generation model.

76. An information processing system as described in claim 72, wherein the input unit accepts input of a specific batter from a user, and the processing unit uses, as the sentence generation model, a model in which past pitching history information corresponding to the specific batter input has been learned.

77. An information processing system comprising: an input unit that accepts user input; a processing unit that performs specific processing using a generative model that generates results according to the input data; and an output unit that displays an avatar representing an agent for interacting with a user in an image display area of ​​an electronic device so as to output the results of the specific processing, wherein the processing unit uses the output of the generative model when the input data is text that instructs the presentation of information related to earthquakes to obtain information related to the earthquake as a result of the specific processing and output it to the avatar.

78. The information processing system of claim 77, wherein the information about earthquakes includes information about past earthquakes in a specified area.

79. The information processing system according to claim 78, wherein the information relating to past earthquakes includes information about disasters caused by earthquakes.

80. An information processing system according to claim 77, wherein the generative model generates an output by taking into account meteorological information in a specified area.

81. An information processing system according to claim 77, wherein the generative model generates an output by taking into account information about the topography of a specified area.

82. An information processing system according to claim 77, wherein said output section changes the behavior of said avatar in accordance with the result of said specific processing.

83. An information processing system as described in claim 77, further comprising a state recognition unit that recognizes a user state including user behavior and a state of an electronic device, wherein the processing unit performs the specific processing using the user state or the state of the electronic device and the generative model.

84. An information processing system as described in claim 77, further comprising: an emotion determination unit that determines an emotion of a user or an emotion of an electronic device; and the processing unit that performs the specific processing using the emotion of the user or the emotion of the electronic device and the generative model.

85. An information processing system according to claim 72 or 77, wherein the electronic device is a headset type terminal.

86. An information processing system according to claim 72 or 77, wherein the electronic device is a glasses-type terminal.

87. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit uses the behavior decision model to analyze SNS related to the user, recognize matters in which the user is interested based on the results of the analysis, and determine the behavior content of the avatar so as to provide information based on the recognized matters to the user.

88. A behavior control system according to claim 87, wherein said behavior control unit controls said avatar so as to provide information based on said recognized matters to said user.

89. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit, when determining that the user is a specific user including a lonely person living alone, switches to a specific mode in which the behavior of the avatar is determined with a greater number of communications than in a normal mode in which behavior is determined for users other than the specific user.

90. A behavior control system as described in claim 89, wherein the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar.

91. A behavior control system as described in claim 89, wherein the behavior decision model is a sentence generation model having an interactive function, and the behavior decision unit inputs text expressing at least one of the user state, the state of the avatar displayed in the image display area, the emotion of the user, and the emotion of the avatar displayed in the image display area, and text asking about the behavior of the avatar, into the sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

92. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit sets a customer service dialogue mode as the dialogue mode of the avatar, in which the avatar serves as a dialogue partner for when there is no need to talk to a specific person but would like someone to listen to what you have to say, and in the customer service dialogue mode, outputs utterances in a dialogue with the user, excluding predetermined keywords related to the specific person.

93. A behavior control system as described in claim 92, wherein the electronic device is a headset, and the behavior determination unit determines the behavior of an avatar as part of an image controlled by the behavior control unit and displayed in an image display area of ​​the headset, and determines one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar.

94. A behavior control system as described in claim 93, wherein the behavior decision model is a sentence generation model having an interactive function, and the behavior decision unit inputs text expressing at least one of the user state, the state of the avatar displayed in the image display area, the emotion of the user, and the emotion of the avatar displayed in the image display area, and text asking about the behavior of the avatar, into the sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

95. A behavior control system as described in claim 92, wherein when the behavior control unit determines to change the setting of a dialogue partner in the customer service dialogue mode as the behavior of the avatar, the behavior control unit causes the avatar to operate with a face and appearance corresponding to the changed dialogue partner.

96. A behavior control system comprising: a behavior determination unit which determines the behavior of an avatar representing an agent for interacting with a user; and a behavior control unit which displays the avatar in an image display area of ​​an electronic device, wherein an image sensor or an odor sensor is set at a customs office, and the behavior determination unit acquires an image of a person taken by the image sensor or an odor detection result by the odor sensor, and when it detects predetermined abnormal behavior, abnormal facial expression, or abnormal odor, determines as the behavior of the avatar to notify a tax inspector.

97. A behavior control system as described in claim 96, further comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; and an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user, wherein the behavior determination unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into a data generation model capable of generating data according to input data, and determines the behavior of the avatar based on the output of the data generation model.

98. A behavior control system as described in claim 96, wherein said behavior control unit, when detecting a preset abnormal behavior, abnormal facial expression, or abnormal odor, causes the avatar to move in an appearance corresponding to the detected content.

99. The behavior control system of any one of claims 1, 3, 5, 7, 10, 13, 16, 19, 25, 27, 31, 33, 35, 37, 39, 43, 47, 51, 53, 59, 64, 68, 70, 87, 89 and 92, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

100. The behavior control system of any one of claims 1, 3, 5, 7, 10, 13, 16, 19, 25, 27, 31, 33, 35, 37, 47, 51, 53, 59, 64, 68, 70, 87, and 96, wherein the electronic device is a headset type terminal.

101. The behavior control system according to any one of claims 1, 3, 5, 7, 10, 13, 16, 19, 25, 27, 31, 33, 35, 37, 47, 51, 53, 59, 64, 68, 70, 87, and 96, wherein the electronic device is a glasses-type terminal.

Images