Action control system

Abstract

In an action control system according to the present invention, an action of an avatar includes creating a picture diary, and when an action decision unit has made the decision to create a picture diary as the action of the avatar, the action decision unit selects a photo or a video from history data, generates a description for a clip of the photo or the video on the basis of an emotion value when acquiring the selected photo or video, and outputs a combination of the description and the clip of the photo or the video as the picture diary.

Description

Behavior Control System

[0001] The present disclosure relates to a behavior control system.

[0002] Patent Literature 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Literature 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

[0003] Patent No. 6053847

[0004] However, the conventional technology has room for improvement in terms of making the robot perform appropriate actions in response to the user's actions.

[0005] According to a first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an avatar representing an agent for interacting with the user; an emotion determination unit that determines an emotion of the user or an emotion of the avatar; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion and a behavior determination model; a storage control unit that stores, in history data, the emotion value determined by the emotion determination unit, event data including data including the user's behavior, and a photo or video captured when the emotion value meets a predetermined criterion; and a behavior control unit that displays the avatar in an image display area of ​​an electronic device, wherein the behavior of the avatar includes creating a picture diary. When the behavior decision unit decides to create the picture diary as the avatar's behavior, it selects the photo or the video from the history data, generates a description of the photo or video clip based on the emotional value at the time the selected photo or video was acquired, and outputs the combination of the photo or video clip and the description as the picture diary.

[0006] According to a second aspect of the present disclosure, the behavior determination model is a data generation model capable of generating data according to input data, and the behavior determination unit inputs data representing at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

[0007] According to a third aspect of the present disclosure, when the behavior decision unit determines that the avatar's behavior is to create the picture diary, it selects the photo or the video from the history data, generates a description of the photo or video clip based on the emotional value at the time the selected photo or video was acquired, and operates the avatar to output a combination of the photo or video clip and the description as the picture diary.

[0008] According to a fourth aspect of the present disclosure, the electronic device is a headset type terminal.

[0009] According to a fifth aspect of the present disclosure, the electronic device is a glasses-type terminal.

[0010] According to a sixth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an avatar representing an agent for interacting with a user; an emotion determination unit that determines an emotion of the user or an emotion of the avatar; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the avatar state, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​an electronic device, wherein the avatar behavior includes advising the user about a fraud risk, and when the behavior determination unit determines that the behavior of the avatar is to advise the user about the fraud risk, the behavior control unit advises the user about the fraud risk.

[0011] According to a seventh aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes an utterance or a gesture made by the electronic device toward the user, and the behavior determination unit autonomously detects the user state, and, when the emotion determination unit determines at least one of the user emotion and the avatar emotion based on the detected user state, determines content of the utterance or the gesture corresponding to the determined at least one of the user emotion and the avatar emotion, and causes the behavior control unit to control the avatar.

[0012] According to an eighth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes interacting with the user, and when the behavior determination unit determines the behavior of the avatar to interact with the user, determines the behavior of the avatar so as to maximize an emotion value indicating the strength of an emotion that is considered important for the user depending on a purpose of the interaction.

[0013] According to a ninth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes interacting with the user, and when the behavior determination unit determines that the avatar behavior is to interact with the user, the behavior determination unit performs feedback to increase an emotion value indicating the strength of the emotion if the user feels a positive emotion in response to the avatar behavior, and performs feedback to decrease the emotion value indicating the strength of the emotion if the user feels a negative emotion in response to the avatar behavior.

[0014] According to a tenth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state, including a user behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of multiple avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes the avatar performing an action that expresses the emotion. The behavior determination unit autonomously collects objects of interest to the user, and, when it determines to provide information corresponding to the user's interest as the avatar behavior, determines the content of the avatar's action that expresses the emotion according to the content of the provided information. Here, the term "robot" includes a device that performs physical actions, a device that outputs video and audio without performing physical actions, and an agent that operates on software.

[0015] According to an eleventh aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit reflects the inferred cultural sphere of the user in at least one of output generation by the behavior determination model, determination of the user emotion by the emotion determination unit, and determination of the avatar emotion by the emotion determination unit.

[0016] According to a twelfth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes a state recognition unit that recognizes a user state, including the user's behavior, and the state of an electronic device; an emotion determination unit that determines the user's emotions or the emotions of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotions, and the avatar's emotions, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes giving advice about a specific competition to the user participating in the specific competition. The behavior determination unit includes an image acquisition unit that can capture an image of a competition space where the specific competition in which the user is participating is being held, and an athlete analysis unit that analyzes the emotions of a plurality of athletes participating in the specific competition in the competition space captured by the image acquisition unit. When it is determined that the avatar's behavior is to give advice about the specific competition to the user participating in the specific competition, the advice is given to the user based on the analysis results of the athlete analysis unit.

[0017] According to a thirteenth aspect of the present disclosure, there is provided a behavior control system including a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device, an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user, a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model, and a storage unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data. The electronic device includes a control unit and an action control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar action includes selecting at least one of two or more things and setting an action content to be suggested to the user, and the action decision unit detects the user's state either autonomously or periodically, and when it decides to suggest at least one of two or more things as the avatar's action based on the detected user's state and at least one of historical data about the user and information preferred by the user, causes the action control unit to display the avatar in the image display area so as to execute the action content.

[0018] According to a fourteenth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving the user advice about household matters, and when the behavior determination unit determines that the behavior of the avatar is to give the user advice about household matters, it uses a sentence generation model to suggest advice about physical condition, recommended dishes, ingredients to replenish, etc., based on data about the household appliances stored in the history data.

[0019] According to a fifteenth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device.

[0020] A behavior control system according to a sixteenth aspect is the fifteenth aspect, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, as well as data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

[0021] The behavior control system of the seventeenth aspect is the fifteenth aspect, further including a related information collection unit that collects information related to the preference information from external data based on the preference information acquired about the user at a predetermined timing, and the emotion determination unit determines the emotion of the avatar based on the collected information related to the preference information.

[0022] In the behavior control system of the 18th aspect, in the 15th aspect, when the behavior control unit determines that the avatar will not take any action, it causes the avatar to make a specific facial expression or a specific gesture.

[0023] A behavior control system according to a nineteenth aspect is any one of the fifteenth to eighteenth aspects, wherein the electronic device is a headset terminal.

[0024] A behavior control system according to a twentieth aspect is the behavior control system according to any one of the fifteenth to eighteenth aspects, wherein the electronic device is a glasses-type terminal.

[0025] According to a twenty-first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that determines an action of the avatar corresponding to the user state and the user's emotion or the avatar's emotion based on a behavior determination model; and a behavior control unit that controls a movement of the avatar displayed in an image display area of ​​the electronic device, wherein the behavior determination unit generates a question corresponding to the user's concern using a sentence generation model, and determines an action of the avatar to be an utterance corresponding to the question.

[0026] According to a twenty-second aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes determining an action schedule for the avatar, and when the behavior determination unit determines the action of the avatar to be determined as the behavior of the avatar, determines a combination of an activation condition for invoking the action schedule and a content of the action schedule for the avatar, stores the combination in action schedule data, and determines to execute the content of the action schedule for the avatar when the activation condition of the action schedule data is satisfied.

[0027] According to a twenty-third aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes encouraging interaction with others, and when the behavior determination unit determines that the behavior of the avatar is to encourage interaction with others, it determines at least one of an interaction partner or an interaction method based on the event data.

[0028] According to a twenty-fourth aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes providing advice regarding reading aloud, and when the behavior determination unit determines to provide advice regarding reading aloud as the behavior of the avatar, generates advice regarding reading aloud from collected information regarding reading aloud in accordance with predetermined suggestion conditions and controls the avatar to provide the advice to the user. Here, the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.

[0029] According to a 25th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's surrounding environment, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device.

[0030] According to a 26th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question based on the user's past emotion, and when the behavior determination unit determines to ask a question based on the user's past emotion as the behavior of the avatar, it speaks to the user. Here, the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software.

[0031] According to a 27th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes talking about a subject of interest of the user, and when the behavior determination unit determines that the behavior of the avatar is to talk about the subject of interest of the user, it determines utterance content related to the event data whose emotion value satisfies a predetermined criterion.

[0032] According to a twenty-eighth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes notifying the provider of information based on the user's emotion regarding an item provided by the provider, and when the behavior determination unit determines that the avatar's behavior is to notify the provider of information based on the user's emotion regarding the item provided by the provider, the behavior determination unit operates the avatar to notify the provider of the information based on the user's emotion regarding the item provided by the provider.

[0033] According to a 29th aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes: a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of multiple avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device. The avatar behavior includes providing advice regarding a pregnant woman. If the behavior determination unit determines that the avatar's behavior is to provide advice regarding a pregnant woman, it collects information regarding at least one of pregnancy and postpartum, and provides the advice regarding the pregnant woman based on the collected information. Here, the term "robot" includes a device that performs physical actions, a device that outputs video and audio without performing physical actions, and an agent that operates on software.

[0034] According to a thirtieth aspect of the present disclosure, there is provided a behavior control system. The behavior control system includes: a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior, using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores, in history data, event data including the emotion value determined by the emotion determination unit and data including the user's behavior, characteristic information including the user's characteristics, and situation information when the characteristic information was acquired; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes speaking to the user, and when the behavior determination unit determines that the avatar's behavior is to speak to the user, it infers the content of the user's conversation with the avatar based on the history data and the situation information at that time, and determines the content of the conversation to the user based on the result of the inference.

[0035] According to a thirty-first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores, in history data, event data including the emotion value determined by the emotion determination unit and data including the user behavior, characteristic information including characteristics of the user, and situation information at the time the characteristic information was acquired; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes playing specific music data, and when the behavior determination unit determines to play the specific music data as the behavior of the avatar, the specific music data to play is determined based on the history data and the situation information at that time.

[0036] According to a thirty-second aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes analyzing a user's personality, and when the behavior determination unit determines to analyze the user's personality as the behavior of the avatar, it analyzes the user's personality using the history data, including a conversation history with the user, and presents the analyzed personality.

[0037] According to a 33rd aspect of the present disclosure, there is provided a behavior control system according to the 32nd aspect, wherein the behavior decision model of the behavior control system is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into the data generation model, and decides the behavior of the avatar based on the output of the data generation model.

[0038] According to a 34th aspect of the present disclosure, there is provided a behavior control system according to the 33rd aspect, wherein the behavior control unit of the behavior control system changes a facial expression of the avatar when presenting the user's personality, in accordance with the user's emotion determined by the emotion determination unit.

[0039] According to a thirty-fifth aspect of the present disclosure, there is provided a behavior control system according to the thirty-third aspect, wherein the electronic device of the behavior control system is a headset terminal.

[0040] According to a thirty-sixth aspect of the present disclosure, there is provided a behavior control system according to the thirty-third aspect, wherein the electronic device of the behavior control system is a glasses-type terminal.

[0041] According to a 37th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including inaction, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving the user advice on a labor issue, and when the behavior determination unit determines that the behavior of the avatar is to give the user advice on the labor issue, it determines to give the user advice on the labor issue based on the user behavior.

[0042] According to a 38th aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit autonomously detects a body temperature of the user, and the emotion determination unit determines at least one of the user emotion and the avatar emotion based on the detected user state, determines at least one of a gesture and a speech pattern corresponding to the determined at least one emotion, and causes the behavior control unit to control the avatar based on the determined at least one pattern.

[0043] 1A and 1B are schematic diagrams showing an example of a system 5 according to the first embodiment; a functional configuration of a robot 100 according to the first embodiment; a schematic diagram of an example of an operation flow of a collection process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of a response process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of autonomous processing by the robot 100 according to the first embodiment; an emotion map 400 onto which multiple emotions are mapped; an emotion map 900 onto which multiple emotions are mapped; (A) an external view of a stuffed animal 100N according to a second embodiment, (B) a diagram of the internal structure of the stuffed animal 100N; a rear front view of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of an agent system 500 according to a third embodiment; an example of the operation of the agent system; an example of the operation of the agent system; a schematic diagram of the functional configuration of an agent system 700 according to a fourth embodiment; an example of a usage mode of an agent system using smart glasses; a schematic diagram of the functional configuration of an agent system 800 according to a fifth embodiment; an example of a headset-type terminal; 1 shows an example of a hardware configuration of a computer 1200.

[0044] The present disclosure will be described below through the disclosed embodiments, but the following embodiments do not limit the disclosure according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the disclosed solution.

[0045] First Embodiment FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of this embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as users 10. Users 11a, 11b, and 11c may be collectively referred to as users 11. Users 12a and 12b may be collectively referred to as users 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

[0046] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 300, and requests the image data, etc. from the server 300 as needed and provides it to the user 10.

[0047] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed when conversing with the user 10. In this way, the robot 100 can express its own emotions through actions.

[0048] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining an action of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the user's action, and determine an action of the robot 100 corresponding to the determined emotion.

[0049] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates an action content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0050] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0051] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0052] As an example of a behavioral decision model, the robot 100 stores rules that define the behavior that the robot 100 will perform based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10, and performs various behaviors in accordance with the rules.

[0053] Specifically, the robot 100 has reaction rules, as an example of a behavioral decision model, for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."

[0054] When the robot 100 recognizes that the behavior of the user 10 is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the words "apologize."

[0055] Furthermore, when the condition that the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to perform the action of "calling out."

[0056] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.

[0057] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

[0058] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

[0059] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0060] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, and a communication processing unit 280.

[0061] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. Furthermore, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0062] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, etc.

[0063] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0064] The storage unit 220 includes a behavioral decision model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the user 10's past emotional values, the robot 100's past emotional values, and behavioral history. Specifically, the history data 222 includes a plurality of event data including the user 10's emotional values, the robot 100's emotional values, and the user 10's behavior. The data including the user 10's behavior includes camera images representing the user 10's behavior. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the user 10's identification information. The history data 222 also includes information on the user's emotions regarding matters provided by the provider (e.g., whether the user is satisfied with local policies, products they are using, relationships with neighbors, or family relationships). At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. The storage unit 220 may also include a person DB that stores the user 10's facial images, attribute information, etc. 2, the functions of the components of the robot 100, excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by operating an operating system (OS) and a program running on the OS.

[0065] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the state recognition unit 230.

[0066] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0067] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.

[0068] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

[0069] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."

[0070] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the environment surrounding the robot 100, etc. as the state of the robot 100.

[0071] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0072] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0073] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0074] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."

[0075] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0076] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0077] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.

[0078] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.

[0079] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0080] Next, the processing of the behavior decision unit 236 when the robot 100 performs a response process in which the robot 100 responds to the behavior of the user 10 will be described.

[0081] The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, history data 222 of past emotion values ​​determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, a case is described in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes gestures made by the robot 100 or speech content of the robot 100 .

[0082] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotional values ​​of the user 10, the emotional value of the robot 100, the behavior of the user 10, and the behavior determination model 221. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the behavior of the user 10, an behavior that will change the emotional value of the user 10 to a positive value.

[0083] The reaction rule as the behavioral decision model 221 defines the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the user 10 is behaving sadly, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.

[0084] For example, the reaction rule as the behavior determination model 221 defines behaviors of the robot 100 for patterns of the robot 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values ​​and current emotional values, and all combinations of the user 10's behavior patterns. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the user 10's behavior pattern for each of a plurality of combinations of the user 10's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0085] The reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotional value of the robot 100.

[0086] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined.

[0087] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0088] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0089] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (for example, the facial expression, emotions, etc. of the user 10).

[0090] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotional determination unit 232.

[0091] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. If an impact is detected by the touch sensor 205 included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotion has worsened. Alternatively, if the detection result of the touch sensor 205 included in the sensor unit 200 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0092] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is negative.

[0093] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0094] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0095] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0096] The related information collection unit 270 collects information related to the preference information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the preference information acquired about the user 10 .

[0097] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the content of utterances by the user 10 or setting operations by the user 10. The related information collecting unit 270 periodically collects news related to the preference information from external data using, for example, ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>). For example, if the acquired preference information is that the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using, for example, ChatGPT Plugins.

[0098] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270 .

[0099] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0100] When the emotion value of the robot 100 is equal to or greater than the threshold value, the storage control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223 .

[0101] Next, the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0102] In the autonomous processing of this embodiment, image data of photographs or videos acquired when the emotional value of the user 10 or the robot 100 reaches a predetermined standard is stored as event data in the history data 222, and a picture diary, i.e., event images, is created using the stored photographs or video clips. In addition, the photographs and videos are also edited when creating the picture diary.

[0103] In the autonomous processing of this embodiment, the robot 100 as an agent autonomously and periodically detects the state of the user 10. It constantly detects and understands telephone conversations between the user 10 and the other party (conversation partner), as well as intercom video and conversations. The robot 100 also reads the conversation content and emotions of the conversation partner and memorizes the conversation content and voiceprint of family and friends as safe. The robot 100 may also determine whether the conversation is high-risk, such as an "I'm your son" scam, by inputting the conversation text into a sentence generation model such as generative AI. The next time there is a phone call or visitor, or during a conversation, if the voiceprint, voice quality, and conversation content memorized as safe exceed a certain threshold, the robot 100 determines there is a risk of fraud. The robot 100 may also autonomously collect and accumulate information on past fraud cases from websites and news, and memorize similar patterns. If the robot 100 determines that the risk is high, it autonomously notifies the elderly person, their family, or emergency contacts, and, if the risk is particularly high, it immediately notifies the police. In addition, the robot 100 can constantly collect information on the latest news and world trends, so it can grasp what types of scams are currently popular, predict what precautions users should take, and spontaneously talk to the user.

[0104] In the autonomous processing in this embodiment, the behavior decision unit 236 autonomously detects the state of the user 10. For example, the behavior decision unit 236 autonomously detects a change in the body temperature of the user 10 at predetermined intervals. Specifically, the behavior decision unit 236 detects a change in the body temperature of the user 10 by comparing the body temperature of the user 10, which is autonomously measured at predetermined intervals using a temperature sensor, with the body temperature of the user 10 measured previously or the average body temperature of the user 10, or the like. Note that the temperature sensor may be a temperature sensor included in the robot 100, or a temperature sensor included in a device other than the robot 100.

[0105] Then, the behavior determining unit 236 determines at least one of the emotion of the user 10 and the emotion of the robot 100 based on the detected state of the user 10 .

[0106] Then, the behavior determination unit 236 determines the content of an utterance or a gesture to be made to the user 10 according to at least one of the determined emotion of the user 10 and the emotion of the robot 100. Specifically, the behavior determination unit 236 inputs text expressing the determined emotion into the behavior determination model 221. Then, the behavior determination unit 236 determines the content of the behavior output by the behavior determination model 221 as the content of the utterance or the gesture to be made to the user 10.

[0107] In the autonomous processing of this embodiment, the behavior determining unit 236 of the robot 100 acquires information indicating the hobbies and preferences of the user 10 via the sensor unit 200. For example, in the robot 100, the user's everyday conversation (e.g., conversations at home) is acquired via the microphone 201, and the behavior determining unit 236 analyzes the content of the conversation to acquire information indicating the hobbies and preferences of the user 10. In this way, the behavior determining unit 236 autonomously executes control to collect the interests of the user 10 from the conversation. Note that, in addition to the conversation of the user 10, the behavior determining unit 236 may also collect the interests of the user 10 from the facial expression of the user 10, the content of an article or book the user 10 is reading, or the content of a television program or radio program that the user 10 likes, etc.

[0108] The behavior determination unit 236 then reflects the autonomously grasped hobbies and preferences of the user 10 in the generation of a response by the AI ​​sentence generation model, the estimation of the user 10's emotions by the emotion engine, and the estimation of the emotions of the robot 100. For example, the behavior determination unit 236 estimates the user 10's favorite baseball team from the acquired conversation. If the autonomously collected news related to the baseball team's game results indicates that the user 10's favorite baseball team won, the behavior determination unit 236 generates a response of "Good job!" to the user 10 and causes the robot 100 to express an emotion of joy (for example, by making a fist pump). On the other hand, if the user 10's favorite team is defeated by a rival team, the behavior determination unit 236 generates a response of "That's frustrating!" and causes the robot 100 to express an emotion of anger (for example, by crossing its arms with an angry expression). In this way, the behavior determination unit 236 not only determines the content of the utterance but also determines the action of the robot 100 to express an emotion, in accordance with the autonomously grasped hobbies and preferences of the user 10. In other words, the behavior determining unit 236 determines the gesture to be performed by the robot 100 according to the tastes and preferences of the user 10 .

[0109] In the autonomous processing of this embodiment, the behavior determination unit 236 autonomously and periodically detects the state of the user 10. The behavior determination unit 236 autonomously infers the cultural sphere (also referred to as the linguistic sphere) in which the user 10 resides, and reflects the inferred cultural sphere in the answer generation by the sentence generation model using AI as an example of the behavior determination model 221, the determination of the user 10's emotion by the emotion determination unit 232, and the determination of the robot 100's emotion by the emotion determination unit 232. For example, when the robot 100 infers that the user 10 resides in the Kansai region or detects that the user 10 speaks in the Kansai dialect, it autonomously switches to a Kansai region brain. In this case, the robot 100 makes a gesture of comment in the Kansai dialect or generates an utterance such as "Nandeyanen (Why?)." In addition, the behavior determination unit 236 may reflect the inferred cultural sphere in one or two of the following: answer generation using a sentence generation model, determination of the user 10's emotions by the emotion determination unit 232, and determination of the robot 100's emotions by the emotion determination unit 232.

[0110] The behavior determining unit 236 may infer the cultural sphere of the user 10 using various methods. For example, the behavior determining unit 236 may infer the cultural sphere of the user 10 from a conversation of the user 10. Note that the "conversation of the user 10" here may be interpreted as including not only a dialogue between the user 10 and the robot 100, but also a conversation between the user 10 and another robot, a conversation between other users 10, and monologue by the user 10. That is, the behavior determining unit 236 may infer the cultural sphere of the user 10 from a conversation of the user 10 that the robot 100 overhears without being a party to, in addition to a conversation between the user 10 and the robot 100 itself. As an example, the behavior determining unit 236 may infer that the cultural sphere of the user 10 is the Kansai region when the user 10 frequently talks about Osaka Prefecture or local information about Osaka Prefecture in the conversation. Furthermore, the behavior determining unit 236 may infer that the cultural sphere of the user 10 is the Kansai region when the user uses the Kansai dialect in the conversation. Alternatively or additionally, the behavior determining unit 236 may infer the cultural sphere of the user 10 based on the location information. As an example, the behavior determining unit 236 may store in advance a cultural sphere map that associates location information with cultural spheres, and infer that the cultural sphere of the user 10 is the Kansai region when location information measured by a positioning means such as a GPS (Global Positioning System) is associated with the Kansai region. In this case, from the perspective of the user 10, the robot 100 has transformed into a character from the Kansai region without the user 10 knowing.

[0111] In this way, the robot 100 behaves in accordance with the residential culture of the user 10, thereby improving the user experience.

[0112] In the autonomous processing of this embodiment, the robot 100 analyzes the state of the user participating in a specific sport and the athletes on the opposing team, particularly the emotions of the athletes, at any timing, either spontaneously or periodically, and provides advice to the user regarding the specific sport based on the analysis results. Here, the specific sport may be a sport played by a team of multiple people, such as volleyball, soccer, or rugby. Furthermore, the user participating in the specific sport may be either an athlete playing the specific sport or a support staff member, such as a manager or coach, of a specific team playing the specific sport.

[0113] In the autonomous processing of this embodiment, the agent may monitor the user and detect the user's behavior or state autonomously or periodically. Specifically, the agent may monitor the user and track and analyze what information the user is viewing on what website. The agent may be interpreted as an agent system, which will be described later. Hereinafter, the agent system may be simply referred to as an agent.

[0114] Voluntary may be interpreted as the agent or robot 100 acquiring the user's state on its own initiative without any external trigger.

[0115] The external trigger may include a question from the user to the robot 100, an active action from the user to the robot 100, etc. Periodically may be interpreted as a specific cycle such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0116] User behavior may be interpreted as the following behavioral trends of users: (1) A user visiting one or more specific stores in a commercial facility such as a department store in order to purchase a specific product. Also, the user moving to multiple product display locations in a specific store. (2) A user browsing products on one or more specific EC (Electronic Commerce) sites using a smartphone or computer in order to purchase a specific product. (3) A user browsing information posted on one or more accommodation booking sites, travel sites, etc. in order to decide on a specific travel destination or accommodation. (4) A user browsing specific information posted on one or more financial information sites in order to purchase a specific financial product in order to purchase a specific financial product using a smartphone, computer, etc.

[0117] The user state may include the following user states: (1) A state in which a user continues to worry or think deeply about which product to purchase while looking at or repeatedly trying on products in a particular store. (2) A state in which a user continues to worry or think deeply about which product to purchase while browsing products on one or more e-commerce sites using a smartphone or computer. (3) A state in which a user continues to worry or think deeply about which accommodation, travel destination, etc. to choose while browsing information posted on one or more accommodation reservation sites, travel sites, etc. using a smartphone or computer. (4) A state in which a user continues to worry or think deeply about which financial product to invest in while browsing information posted on one or more financial information sites using a smartphone, computer, etc.

[0118] Additionally, in autonomous processing, the agent may ask the generative AI questions about the detected state or behavior of the user.

[0119] Furthermore, in the autonomous processing, the generative AI's response to the question and the action content proposing the item may be stored in association with each other. The action content may be interpreted as the action content by the electronic device proposing at least one item from two or more items. Specifically, the action content may be interpreted as the action content by the electronic device proposing a specific item based on the generative AI's response to the detected state or behavior of the user.

[0120] The information associating the answers of the generative AI with the proposed actions may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0121] Furthermore, the autonomous processing may utilize the specific information, which is the stored table information, to execute an action content that suggests at least one thing from two or more things in response to the user's state or behavior. Specifically, the autonomous processing may detect the user's state autonomously or periodically, and suggest at least one thing from two or more things as an action of the electronic device based on the detected user's state or behavior and the specific information.

[0122] This specific information may be interpreted as information provided by the generative AI based on at least one of user history data and user preference information. In other words, the autonomous processing may suggest at least one of two or more actions as an action of the electronic device based on the detected user state or behavior and at least one of user history data and user preference information.

[0123] Below, we will explain examples of actions to propose things.

[0124] For example, by monitoring the operations of a user using a smartphone, if the agent detects that the user is unable to decide whether to purchase clothing made by company A or clothing made by company B, the agent will itself ask the generative AI a question.

[0125] The generative AI answers at least one of two or more things based on at least one of historical data 222 about the user and information preferred by the user.

[0126] The history data 222 may include information obtained by tracking, such as the user's personality, preferences, habits, movements, thoughts, actions, conversation content, emotions, and the like.

[0127] The information preferred by the user may be interpreted as information included in the above-mentioned collected data 223. Specifically, the information preferred by the user may be interpreted as preference information that indicates matters of interest to the user 10 and is stored in the collected data 223. More specifically, the information preferred by the user may include information that is frequently searched for or selected by the user, such as trends, world affairs, etc.

[0128] The information preferred by the user is not limited to this and may include social information from multiple sources, which may include at least one of news, economic conditions, social conditions, political conditions, financial conditions, international conditions, sports news, entertainment news, birth and death announcements, cultural conditions, and trends.

[0129] For example, in response to an agent's question, "What products should we suggest to a user who can't decide which clothes to buy?", the generative AI may answer, based on at least one of historical data 222 about the user and information about the user's preferences, "Company A's products will be increasing in price from April, so we recommend purchasing Company A's products before the price increase."

[0130] The generative AI may also respond, "Company B's products will be reduced in price from April, so I recommend purchasing Company B's products after the price reduction."

[0131] The generative AI may also respond, "In light of the user's recent purchasing trends, I recommend purchasing Company C's product, which is more expensive than Company A and Company B's products but is similar to Company A and Company B's products."

[0132] The agent that receives the answer can suggest at least one of two or more things based on the detected state or behavior of the user and the recorded information. In other words, the agent may refer to the recorded information and play a sound corresponding to the content of a product that is suitable for the detected state or behavior of the user through a speaker mounted on the smartphone, robot 100, etc.

[0133] The agent may refer to the recorded information and display an image corresponding to the content of a product appropriate for the detected state or behavior of the user on a screen mounted on a smartphone, robot 100, or the like.

[0134] The agent may refer to the recorded information and display a message on a screen mounted on a smartphone, robot 100, etc., explaining the contents of a product that is suitable for the detected state or behavior of the user.

[0135] Instead of monitoring the operations of a user using a smartphone, the agent may monitor the user moving between multiple product display areas within a specific store using image data captured by an imaging device.

[0136] As described above, the behavior control system of the present disclosure can use at least one of the user's historical data and the user's preference information to select at least one of two or more things and determine the action to be proposed to the user. Therefore, for a user who has difficulty choosing an action, the agent can spontaneously speak and recommend and suggest an action that is suitable for the user.

[0137] In the autonomous processing of this embodiment, the robot 100 interacts with various household appliances (such as not only an air conditioner and a television, but also a bathroom scale and a refrigerator) and autonomously and constantly collects information about the user 10. The robot 100 also autonomously collects various pieces of information about the household appliances. For example, the robot 100 autonomously collects information about when and under what weather the air conditioner is turned on and at what temperature the emotional value rises. The robot 100 also autonomously collects information about how often the refrigerator is used and what items are frequently taken in and out of it. The robot 100 also autonomously collects information about changes in the user 10's weight and the relationship between television programs and changes in the user's emotional value. When the user 10 is nearby, the robot 100 helps manage schedules and provides news of interest to the user 10, and suggests health advice, recommended dishes, and ingredients to replenish. The robot 100 may also automatically order ingredients to replenish.

[0138] In the autonomous processing of this embodiment, the robot 100 autonomously and periodically detects the state of the user 10. For example, the robot 100 autonomously and periodically detects the behavior of the user 10, the emotions of the user 10, and the emotions of the robot 100. A fixed sentence inquiring about the action the robot 100 should take is added to the text representing the state of the user 10, and the text is input into a sentence generation model to acquire the action content of the robot 100. The action content is acquired and stored, and the stored action content (e.g., utterance) is activated at a different time or at a different timing. In this way, the robot 100 autonomously detects the state of the user 10, predetermines the action content of the robot 100, and can then make an utterance or take an action by itself the next time some trigger occurs for the user 10.

[0139] In the autonomous processing of this embodiment, the device operation (robot behavior, if the electronic device is the robot 100) determined by the behavior determination unit 236 includes encouraging interaction with others. When the behavior determination unit 236 determines that the behavior of the electronic device (robot behavior) is to encourage interaction with others, it determines at least one of an interaction partner or an interaction method based on the event data.

[0140] In the autonomous processing of this embodiment, the agent memorizes all the contents of the books that the user's parents (mother or father) read to their child at night. The agent also memorizes the child's emotions while reading. During a period of inactivity on another day, the agent suggests to the child or the parents to read a book similar to the book that was read when the child's reaction was good (for example, when the emotional value was high).

[0141] In the autonomous processing of this embodiment, the robot 100 autonomously and periodically detects the state of the user 10. For example, the robot 100 autonomously and periodically detects the behavior of the user 10, the user's surrounding environment, the user's emotions, and the robot's emotions. A fixed sentence inquiring about the action the robot 100 should take is added to the text representing the state of the user 10, and the text is input into a sentence generation model to acquire the action content of the robot 100. The action content is acquired and stored, and when it matches the surrounding environment of the user 10 at a different time period or timing set as an activation condition, the stored action content (e.g., speech) is activated. In this way, the robot 100 autonomously detects the state of the user 10, predetermines the action content of the robot 100, and can then, when some trigger occurs for the user 10, initiate speech or action by itself.

[0142] In the autonomous processing of this embodiment, pressure sensors (air pressure sensors) set in the agent's hands and touch sensors set in the agent's nose are used to detect and store all of the user's actions, and if the user's emotion value at the time of detecting a gesture exceeds a certain value, the gesture is stored as being particularly important. Then, if the same user gesture and emotion value are detected at a different time, the agent will spontaneously say, "That's the same emotion as back then. What's wrong?"

[0143] In the autonomous processing in this embodiment, the device operation (robot behavior, if the electronic device is the robot 100) determined by the behavior determination unit 236 includes talking about the interests of the user 10. When the behavior determination unit 236 determines that the behavior of the electronic device (robot behavior) is to talk about the interests of the user 10, it determines the content of the utterance related to the event data whose emotional value satisfies a predetermined criterion.

[0144] In the autonomous processing in this embodiment, the robot 100 as an agent voluntarily and periodically stores information based on the feelings of the user 10 regarding items provided by the provider in the history data 222. In addition, the robot 100 voluntarily and periodically notifies the provider of information based on the feelings of the user 10 regarding items provided by the provider.

[0145] Here, "provider" refers to an individual or organization that provides products, services, etc. to user 10. An "organization" is, for example, an administrative organization, a profit-making organization, or a non-profit organization. An "administrative organization" is an organization that administers the country, prefecture, city, town, or village. A "profit-making organization" is an organization that aims to make a profit, such as a profit-making company or a profit-making corporation. A "non-profit organization" is an organization that does not aim to make a profit, such as a non-profit organization or a non-profit corporation.

[0146] Furthermore, "information based on the emotions of user 10 regarding the items provided by the provider" refers to information indicating the user's 10 impressions regarding the items provided by the provider, and may be information regarding the type of emotions of user 10, such as, for example, "happy," "fun," "satisfied," "not happy," "not fun," "dissatisfied," etc., or may be the above-mentioned emotional value derived based on the emotions of user 10.

[0147] Furthermore, "notifying the provider" means making the information based on the emotions of user 10 available for the provider to check; for example, the information may be sent to the provider by email, or the information may be uploaded to the cloud so that the provider can check it.

[0148] That is, the robot 100 can provide feedback to the city about the user's thoughts on a policy or service provided by the city, or provide feedback to a company about the user's thoughts on a product or service provided by the company.

[0149] In the autonomous processing of this embodiment, the robot 100 as an agent autonomously collects pregnancy-related information, such as information about pregnancy and postpartum, when the user or a user's family member is pregnant or is trying to conceive, or is otherwise trying to conceive. When the robot 100 detects that the user or a user's family member is pregnant or trying to conceive, it autonomously provides various pregnancy-related information to parents during pregnancy and after birth, and autonomously helps them navigate emotional control. For example, it autonomously suggests ways to deal with worries during pregnancy and postpartum stress, thereby improving parental confidence. Furthermore, it autonomously provides information about childcare and supports the new family member's adjustment to life.

[0150] In the autonomous processing in this embodiment, the robot 100 as an agent performs the autonomous processing. More specifically, the robot 100 performs the autonomous processing to take action based on the past history of the robot 100 (there may be no history) and the behavior of the user 10, regardless of whether the user 10 is present or not.

[0151] The robot 100 as an agent autonomously and periodically detects the state of the user 10. For example, the robot 100 performs a psychologically based personality analysis by unilaterally listening to what the user is saying or by conversing with the user. The robot 100 stores a history of conversations with the user 10 as history data 222 and performs personality analysis using the history data 222. For example, the robot 10 analyzes the personality of the user 10 by analyzing the user's habitual speech, such as the beginning or ending of a sentence, recorded in the history data 222.

[0152] Furthermore, if the user is emotional, depressed, or in high spirits, the robot 100 spontaneously analyzes the personality of the user 10 and notifies the user 10 of the personality analysis result.

[0153] When analyzing the personality of the user 10 and conveying the analysis results, the robot 100 may casually convey the analysis results to the user during conversation.

[0154] The robot 100 voluntarily analyzes the personality of the user 10 and conveys the personality analysis results to the user 10, thereby enabling the user 10 to deepen their understanding of their own personality.

[0155] In the autonomous processing of this embodiment, the robot 100 autonomously and periodically (or constantly) detects the state of the user 10. Specifically, the robot 100 autonomously and periodically (or constantly) detects the behavior (e.g., conversations and movements) of the user 10 and provides advice on labor issues based on the detected behavior of the user 10. For example, the robot 100 constantly monitors the workplace situation of the user 10, who is a worker, stores the behavior of the user 10 in the history data 222, and autonomously detects labor issues such as power harassment, sexual harassment, and bullying that are difficult for the user 10 to notice based on the behavior of the user 10. Furthermore, the robot 100 autonomously and periodically (or constantly) collects information on the user 10's preferences and stores it in the collected data 223. For example, the robot 100 autonomously and periodically collects information on labor issues and stores it in the collected data 223. When the robot 100 detects a labor problem of the user 10 based on the behavior of the user 10, it spontaneously suggests ways to deal with the labor problem to the user 10 by using the collected information and querying a sentence generation model with a dialogue function. This makes it possible to provide support that is sensitive to the feelings of the user 10 (for example, information on labor laws and appropriate procedures).

[0156] In the autonomous processing of this embodiment, the robot 100 voluntarily and periodically detects the state of the user 10. For example, the robot 100 detects changes in the body temperature of the user 10 observed by a thermosensor. The detection results are then reflected in the answer generation of the AI ​​sentence generation model, the user emotion estimated by the emotion engine, and the emotion estimation of the robot 100. For example, if the entire body of the user 10 becomes hot, the robot 100 determines that the user 10 is "happy" and makes positive gestures and positive utterances corresponding to this.

[0157] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0158] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0159] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the surrounding environment of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0160] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the surrounding environment of the user 10, the emotions of the user 10, the emotions of the robot 100, and the state of the robot 100, as well as text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0161] For example, the multiple types of robot behaviors include the following (1) to (25).

[0162] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot recalls memories. (11) The robot advises the user on fraud risks. (12) The robot gives advice to a user participating in a specific competition. (13) For a user who has difficulty choosing between two or more things, the robot selects at least one of two or more things, and the agent spontaneously plays audio corresponding to the selection. (14) For a user who has difficulty choosing between two or more things, the robot selects at least one of two or more things, and the agent spontaneously displays an image corresponding to the selection. (15) For a user who has difficulty choosing between two or more things, the robot selects at least one from two or more things, and the agent displays a message corresponding to the selection. (16) The robot gives the user advice about household matters. (17) The robot's actions are determined in advance. (18) The robot encourages interaction with others. (19) The robot gives advice about reading aloud. (20) The robot asks questions about important gestures. (21) The robot talks about topics of interest to the user. (22) The robot notifies the provider of information based on the user's feelings about the items provided by the provider. (23) The robot gives advice about pregnant women. (24) The robot analyzes the user's personality. (25) The robot gives advice about labor issues.

[0163] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0164] The behavior determination unit 236 inputs the following into the sentence generation model at regular time intervals: the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230; the surrounding environment of the user 10; text representing the current emotional value of the user 10 determined by the emotion determination unit 232; and text asking which of multiple types of robot behaviors, including not taking any action, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input into the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0165] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (25) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0166] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (25) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0167] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0168] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0169] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. The information stored in the collected data 223 includes information on the hobbies and preferences of the user 10. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0170] Regarding "(7) The robot introduces news that the user is interested in," the related information collection unit 270 stores information indicating the hobbies and preferences of the user 10, which is autonomously collected by the behavior decision unit 236, in the collection data 223.

[0171] For example, in a news report about the results of a professional baseball game, if the user 10's favorite team wins, the behavior determining unit 236 will introduce the news and determine an utterance content that expresses joy, such as "Good job!" On the other hand, if the user 10's favorite team loses, the behavior determining unit 236 will determine an utterance content that expresses anger, such as "What a shame!"

[0172] Furthermore, the behavior determination unit 236 determines a gesture to be made by the robot 100 corresponding to the information stored in the collected data 223. For example, when the user 10's favorite team wins, the behavior determination unit 236 announces the news and determines a behavior to express joy (for example, a fist pump or a cheer pose). On the other hand, when the user 10's favorite team loses, the behavior determination unit 236 determines a behavior to express anger (for example, a pose with arms crossed).

[0173] Here, the example of robot behavior is "(7) The robot introduces news that the user is interested in." However, any behavior that provides information that matches the interests of the user 10 may be used, and online articles, sites, blogs, or posts on SNS that the user is interested in may be provided along with or instead of news.

[0174] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0175] When the behavior determination unit 236 determines that the robot 100 will create an event image, the behavior determination unit 236 selects a photo or video clip from the history data 222, generates a description for the image using a sentence generation model based on the emotional value of the user 10 and the emotional value of the robot 100 when the selected photo or video clip (hereinafter simply referred to as image) was acquired, and outputs the combination of the image and the description as an event image, i.e., a picture diary. In this case, the robot may also edit the image by simultaneously performing a robot behavior of editing a photo or video (8). For event data selected from the history data 222, the behavior determination unit 236 may generate an image representing the event data using an image generation model and generate a description representing the event data using a sentence generation model, and output the combination of the image representing the event data and the description representing the event data as an event image. When the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image and stores the event image in the behavior schedule data 224.

[0176] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0177] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0178] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0179] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0180] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0181] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0182] When the behavior decision unit 236 decides to set the robot behavior as "(11) Advise the user on fraud risk," i.e., to advise the user on fraud risk, the robot 100 acquires the content of the conversation between the user 10 and the conversation partner and the voiceprint. Specifically, the utterance understanding unit 212 analyzes the voice of the user 10 and the voice of the conversation partner detected by the microphone 201 to acquire the content of the conversation between the user 10 and the conversation partner and the voiceprint. Next, the robot 100 acquires the emotional value of the conversation partner. Specifically, the robot 100 acquires the voice of the conversation partner via telephone or intercom and the video of the conversation partner displayed on the intercom screen to acquire the emotional value of the conversation partner. Furthermore, the robot 100 stores the content of the conversation between the user 10 and the conversation partner, the video of the intercom, etc. in the history data 222. Next, the robot 100 determines the fraud risk based on the content of the conversation and the emotional value of the conversation partner. Specifically, the behavior determination unit 236 compares the conversation content with data on past fraud cases stored in the storage unit 220 to determine a safety value, which is the degree of similarity between the conversation content and the fraud case. The behavior determination unit 236 may also determine the degree of similarity between the conversation content and the fraud case by inputting the sentences of the conversation into a sentence generation model such as generative AI. The behavior determination unit 236 then determines a safety value, which is the degree of fraud risk, based on the degree of similarity between the conversation content and the fraud case and the emotional value, voiceprint, and voice quality of the conversation partner. For example, when the degree of similarity between the conversation content and the fraud case is high, the behavior determination unit 236 determines a high safety value regardless of the emotional value, voiceprint, and voice quality of the conversation partner. Furthermore, even when the degree of similarity between the conversation content and the fraud case is not particularly high, the behavior determination unit 236 may determine a high safety value if the emotional value of the conversation partner is "anxiety" or "excitement" or depending on the voiceprint and voice quality. Next, the behavior determining unit 236 determines an action to be taken according to the determined degree of fraud risk. Specifically, if the determined safety value exceeds a predetermined threshold, the behavior determining unit 236 determines to take an action to notify the user 10 that the fraud risk is high. For example, the behavior determining unit 236 may determine to take an action to notify the user 10 that the fraud risk is high. Furthermore, the behavior determining unit 236 may determine an action to notify the user 10's family or emergency contacts that the fraud risk is high.Furthermore, the behavior determination unit 236 may determine an action to immediately report the high fraud risk to the police. These actions may be determined appropriately depending on the degree of fraud risk. The behavior control unit 250 then controls the speaker, which is the controlled device, so that the above message is output as audio from the speaker. Regarding "(11) Advise the user on fraud risks," the related information collection unit 270 may actively collect and accumulate information on past fraud cases from websites and news, and store it in the collected data 223. This allows the robot 100 to constantly collect information on recent news and world trends, so it can understand what types of frauds are currently popular, predict what precautions should be taken, and actively talk to the user.

[0183] When the behavior determining unit 236 determines that the robot 100 will make an utterance in one of the above actions (1) to (25) as the robot behavior, the behavior determining unit 236 uses a sentence generation model to determine the utterance content of the robot 100 corresponding to the user state and the emotion of the user 10 or the emotion of the robot 100, in accordance with the cultural sphere of the user 10 that has been estimated. For example, if the cultural sphere of the user 10 estimated by the behavior determining unit 236 is the Kansai region, the voice representing the utterance content output by the robot 100 will be a Kansai dialect voice, such as the utterance "Nandeyanen (Why?)." Note that the behavior determining unit 236 may set the gesture corresponding to the utterance content to a gesture corresponding to the cultural sphere of the estimated user 10, such as a gesture to make a comment.

[0184] When the behavior decision unit 236 decides to perform the robot behavior of "(12) Providing advice to a user participating in a specific competition," that is, to provide advice to a user, such as an athlete or coach participating in a specific competition, regarding the specific competition in which the user is participating, the behavior decision unit 236 first detects the emotions of multiple athletes participating in the competition in which the user is participating.

[0185] In order to detect the emotions of the multiple athletes described above, the behavior determination unit 236 has an image acquisition unit that captures images of the competition space where the specific sport in which the user is participating is being held. The image acquisition unit can be realized, for example, by using a part of the sensor unit 200 described above. Here, the competition space may include a space corresponding to each sport, such as a volleyball court or a soccer field. This competition space may also include the surrounding area of ​​the court described above. The installation position of the robot 100 should be considered so that the image acquisition unit can overlook the competition space.

[0186] The behavior determination unit 236 also includes an athlete analysis unit capable of analyzing the emotions of multiple athletes in images acquired by the image acquisition unit. This athlete analysis unit can determine the emotions of multiple athletes, for example, using a method similar to that of the emotion determination unit 232. Specifically, for example, the sensor module unit 210 may analyze images acquired by the image acquisition unit, and input the resulting information into a pre-trained neural network to identify emotion values ​​indicating the emotions of the multiple athletes, thereby determining the emotions of each athlete. Note that the image acquisition unit and athlete analysis unit may be collected and stored as part of the collected data 223 by the related information collection unit 270.

[0187] If it is possible to determine from the emotional values ​​of a player playing a particular sport, such as volleyball, that the player is emotionally unstable or irritated, the team's strategy can be reflected in the determination result, potentially allowing the team to advance in the match to an advantage. Specifically, since emotionally unstable or irritated players tend to make mistakes more frequently than emotionally stable players, in volleyball, for example, the more opportunities an emotionally unstable or irritated player has to touch the ball, the higher the likelihood of making a mistake. Therefore, in this embodiment, advice for advancing the game to an advantage, specifically the emotional values ​​of each player analyzed by the action decision unit 236, is communicated to the user, for example, the coach of a team in the game, to provide the user with advice.

[0188] Considering the above, the athletes analyzed by the athlete analysis unit may be athletes belonging to a specific team among multiple athletes in the competition space. More specifically, the specific team may be a team different from the team to which the user belongs, in other words, the opposing team. The robot 100 may scan the emotions of the athletes on the opposing team, identify the most emotionally unstable or irritated athlete, and advise the user accordingly, thereby assisting the user in formulating an effective strategy. One example of such a strategy may be to advance the game by focusing on the positions of athletes who are emotionally unstable or irritated (for example, in the case of volleyball, concentrating ball distribution toward the emotionally unstable or irritated athlete).

[0189] If such a robot 100 is used during a match in which teams face off against each other, it is expected that the team will gain an advantage in the match. Specifically, by identifying the most mentally unstable player during the match and thoroughly targeting that player, the team can get closer to victory.

[0190] The advice given by the behavior decision unit 236 may be autonomously executed by the robot 100, rather than being initiated in response to an inquiry from the user. Specifically, the robot 100 may make a speech by itself when it detects, for example, when the manager who is the user is in trouble, when the team to which the user belongs is about to lose, or when members of the team to which the user belongs are having a conversation that suggests they want advice.

[0191] The behavior decision unit 236 can select at least one of the aforementioned behavior contents (13) from two or more things as the robot behavior, and the agent can spontaneously play back a sound corresponding to the selected content.

[0192] The behavior decision unit 236 can select at least one of the aforementioned behavior contents (14) from two or more things as the robot behavior, and the agent can spontaneously display an image corresponding to the selected content.

[0193] The behavior decision unit 236 can select at least one of the above-mentioned behavior contents (15), i.e., two or more things, as the robot behavior, and the agent can display a message corresponding to the selected content.

[0194] The related information collection section 270 may store information about the user's preferences in the collection data 223 regarding the above-mentioned behavioral content of "(13)."

[0195] The related information collection unit 270 may store information about the user's preferences in the collection data 223 regarding the above-mentioned behavioral content of "(14)."

[0196] The related information collection section 270 may store information about the user's preferences in the collection data 223 regarding the above-mentioned behavioral content of "(15)."

[0197] The memory control unit 238 may store information obtained by tracking the behavioral content of the above-mentioned "(13)," such as the user's personality, preferences, habits, movements, thoughts, actions, conversation content, emotions, etc., in the history data 222.

[0198] The memory control unit 238 may store information obtained by tracking the behavioral content of the above-mentioned "(14)," such as the user's personality, preferences, habits, movements, thoughts, actions, conversation content, emotions, etc., in the history data 222.

[0199] The memory control unit 238 may store information obtained by tracking the behavioral content of the above-mentioned "(15)," such as the user's personality, preferences, habits, movements, thoughts, actions, conversation content, emotions, etc., in the history data 222.

[0200] When the behavior determining unit 236 determines that the robot behavior is "(16) The robot gives the user advice about household matters," that is, when the robot determines to give advice about household matters, the robot works in conjunction with appliances present in the home, such as an air conditioner, television, scale, and refrigerator, and autonomously collects information about the user 10. Regarding "(16) The robot gives the user advice about household matters," the related information collecting unit 270 collects news that the user is interested in from external data at a predetermined time every day, for example, using ChatGPT Plugins. Regarding "(16) The robot gives the user advice about household matters," the storage control unit 238 stores the collected information related to the advice in the collected data 223.

[0201] The behavior decision unit 236 determines, as a robot behavior, "(17) The content of the robot's behavior is determined in advance." In other words, when it decides to determine the behavior schedule of the robot 100, it determines the combination of the activation conditions for activating the behavior schedule and the content of the behavior schedule of the robot 100, and stores the combination in the behavior schedule data 224.

[0202] Specifically, the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, the current emotional value of the user 10 determined by the emotion determination unit 232, the current emotional value of the robot 100, text representing the history data 222, and text asking about the robot's behavior to be executed later and the activation conditions are input to the sentence generation model, and based on the output of the sentence generation model, a combination of activation conditions for activating the planned behavior and the content of the planned behavior of the robot 100 is determined. Here, the activation conditions include, for example, the time of day or the detection of the user 10. Furthermore, when the user 10 is not present around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0203] Specifically, the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, the surrounding environment of the user 10, the current emotional value of the user 10 determined by the emotion determination unit 232, the current emotional value of the robot 100, text representing the history data 222, and text asking about the robot's behavior to be executed later and the activation conditions are input to the sentence generation model, and based on the output of the sentence generation model, a combination of activation conditions for activating the planned behavior and the content of the planned behavior of the robot 100 is determined. Here, the activation conditions include, for example, the time of day, conditions related to the surrounding environment of the user 10, and detection of the user 10. Furthermore, when the user 10 is not present around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0204] Specifically, the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, the surrounding environment of the user 10, the current emotional value of the user 10 determined by the emotion determination unit 232, the current emotional value of the robot 100, text representing the history data 222, and text asking about the robot's behavior to be executed later and the activation conditions are input to the sentence generation model, and based on the output of the sentence generation model, a combination of activation conditions for activating the planned behavior and the content of the planned behavior of the robot 100 is determined. Here, the activation conditions include, for example, the time of day, conditions related to the surrounding environment of the user 10, and detection of the user 10. Furthermore, when the user 10 is not present around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0205] When the activation conditions of the action schedule data 224 are satisfied, the action decision unit 236 decides that the robot 100 should perform the content of the action schedule of the robot 100 .

[0206] When the behavior determining unit 236 determines that the robot behavior is "(18) Encourage interaction with others," i.e., when the robot 100 determines to suggest that the user 10 interact with others, the behavior determining unit 236 determines at least one of an interaction partner and an interaction method based on the event data stored in the history data 222. For example, when the state of the user 10 satisfies the condition "alone and looking lonely," the behavior determining unit 236 determines the robot behavior as "(18) Encourage interaction with others." Note that the state of the user 10 being alone and looking lonely may be recognized based on information analyzed by the sensor module unit 210 or may be recognized based on schedule information such as a calendar. In such a case, the behavior determining unit 236 uses the event data stored in the history data 222 to learn the user 10's past conversations and experiences, and determines at least one of an interaction partner and an interaction method, preferably both. As an example, if "Grandpa" is determined as the interaction partner and "phone" is determined as the interaction method, the behavior determination unit 236 may determine the utterance content as "Why don't you call Grandpa? His phone number is XXX." In response to this, the behavior control unit 250 may output a voice representing the determined utterance content of the robot from a speaker included in the control target 252. Furthermore, if "Mr. A" is determined as the interaction partner and "visit his house" is determined as the interaction method, the behavior determination unit 236 may determine the utterance content as "Why don't you visit your best friend Mr. A's house? I'll show you how to get to Mr. A's house." In response to this, the behavior control unit 250 may output a voice representing the determined utterance content of the robot from a speaker included in the control target 252, and may also display a map from the user 10 to Mr. A's house on a display device included in the control target 252. When the user 10 is not present around the robot 100, the behavior control unit 250 may store the determined utterance content and map of the robot 100 in the behavior schedule data 224 without outputting the voice or map representing the determined utterance content of the robot 100. In this way, an inorganic electronic device (e.g., a robot) can contribute to people's happiness by expressing its ego that wants its family to be happy and spontaneously performing various actions.

[0207] When the behavior decision unit 236 determines that the robot behavior is "(19) Give advice about reading aloud," that is, when the robot 100 determines to give advice about reading aloud to the user 10, the behavior decision unit 236 generates advice about reading aloud from the collected information about reading aloud in accordance with predetermined suggestion conditions, and provides the advice to the parent or child user 10. The advice is provided, for example, by the robot 100 uttering the advice.

[0208] Specifically, the state recognition unit 230 identifies the user 10 as either a first user, who is a parent (mother or father) reading to their child, or a second user, who is a child being read to. The behavior determination unit 236 then executes processing to generate and provide advice regarding reading aloud to at least one of the users in accordance with the proposal conditions. The proposal conditions include at least a frequency at which advice is provided. The user 10 can change the frequency as appropriate, such as once every three days or once every five days. The behavior determination unit 236 generates and provides advice according to the frequency set as the proposal conditions. The proposal conditions may also include a preset frequency for providing advice to the first user (parents) and a preset frequency for providing advice to the second user (child). As described below, further conditions for the first user (parents) and the second user (child) may also be set.

[0209] The related information collection unit 270 collects the contents of the book that the first user (parents) is reading aloud to the second user and stores the book and its related title. The book contents are collected by accepting the book title input from the first user in advance, and collecting and storing the outline, main text, etc. indicating the book contents from external data. Furthermore, the book contents may be collected by referring to external data from the first user's utterances, rather than from the first user's input.

[0210] The sensor module unit 210 or the like collects analyzed information about the state of the second user (child) when he or she is being read to, the state is recognized by the state recognition unit 230, and the emotion determination unit 232 determines an emotion value (corresponding to step S102 described below). The behavior determination unit 236 may also store the content that was being read to when the emotion value was high, and may include the content itself or a summary of the content in the advice.

[0211] Examples of collected information related to reading aloud include the content (or summary of the content) of the book being read aloud by the first user (parents) and the emotional value of the second user (child) while being read aloud. In this way, the collected information related to reading aloud includes information focusing on the first user and information focusing on the second user.

[0212] The behavior determination unit 236 generates advice about reading aloud based on the collected information about reading aloud and in accordance with the suggestion conditions, and provides the advice about reading aloud. The advice about reading aloud may, for example, list books that were read aloud when the emotional value was high, or books similar to the content of the books read aloud at that time, and suggest titles of books to read aloud. The user to whom the advice is provided may be a first user (parents) or a second user (child), and the advice content may be tailored to the type of user. For example, advice provided to the first user (parents) may be, "Why don't you try reading aloud to your child? Here are some recommended book titles: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD..." For the second user, advice such as, "Why don't you have AAAA read aloud to you?" may be provided. Furthermore, a summary of the content of the book that the second user (child) read aloud when their emotional value was high may be included as additional information. For example, advice including additional information such as "AAAA seemed to like the XXXX scene" may be provided to the first user (parents) and "AAAA liked XXXX" may be provided to the second user (child). Note that the above advice is just an example.

[0213] The behavior determination unit 236 may also collect the date and time when the first user (parents) read aloud, and provide advice regarding reading aloud as a suggestion condition if reading aloud has not been performed for a certain period of time, for example, three days or more or one week or more. The suggestion condition may also collect the emotional value of the second user (child), and provide advice regarding reading aloud to the first user (parents) or the second user (child) if the trend of the emotional value is declining. In this way, in addition to the normal frequency of provision, the suggestion condition may further include a condition using the frequency of reading aloud by the first user (parents) and the emotional trend of the second user (child). This concludes the explanation of the case of "(19) Providing advice regarding reading aloud."

[0214] The behavior determining unit 236 selects, as the robot behavior, "(20) Ask about important gestures." In other words, if the user's gesture matches a past important gesture, the robot can spontaneously utter, "That's the same emotion you felt back then. What's wrong?" Furthermore, with regard to "(20) Ask about important gestures," the memory control unit 238 stores the user's behavior (gesture) in the history data 222 together with the user's emotion value. Furthermore, if the user's emotion value exceeds a certain value, the user's gesture is stored in the history data 222 as an important gesture. In the response process, a match determination is made between the stored user's gesture and the important gesture.

[0215] When the robot 100 determines that the robot 100 will make an utterance about the user 10's interests, the behavior determination unit 236 determines the content of the utterance related to the event data whose emotional value satisfies a predetermined criterion. For example, when the user 10, who is a child, visits an art museum or studies chemistry, geography, or history, the emotional value of the user 10 regarding their studies can be ascertained from the utterances and facial expressions of the user 10. Such matters with high emotional values ​​(e.g., above a threshold) can be assumed to be matters of interest (curious) to the user 10. Therefore, the robot 100 can store, in the history data 222, event data including the user 10's behavior when the user 10's emotional value is high (e.g., what the user is studying or what the user is impressed by). In such a case, the behavior determination unit 236 can determine the content of the utterance, such as, "What was it about that art museum that interested you?", "Tell me about the chemistry lesson you were studying earlier?", or "If you want to deepen your knowledge of chemistry, you should read this book." The behavior decision unit 236 can also determine the content of the utterance, such as asking the child a quiz about a museum they visited or chemistry they studied. Furthermore, the behavior decision unit 236 can also determine the content of the utterance, such as coming up with a new story about history they studied. In this case, the behavior control unit 250 outputs a sound representing the determined content of the utterance of the robot 100 from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 may store the determined content of the utterance of the robot 100 in the behavior schedule data 224 without outputting a sound representing the determined content of the utterance of the robot 100. In this way, when a certain amount of time has passed since the user 10's action, the robot 100 can autonomously talk to the user 10 about their interests, thereby enhancing the child's self-esteem and increasing their motivation to study.

[0216] When the behavior determining unit 236 determines, as the robot behavior, "(22) Notify the provider of information based on the user's feelings regarding the item provided by the provider," that is, when it determines to feed back the user's impressions to the provider, the behavior determining unit 236 selects event data related to the item provided by the provider from the history data 222. At this time, the emotion determining unit 232 determines the user's emotion based on the selected event data. Furthermore, the behavior determining unit 236 notifies the provider of information based on the user's feelings regarding the item provided by the provider, based on the user's emotion determined by the emotion determining unit 232.

[0217] For example, the robot 100 installed in a home or a public facility detects the user's feelings regarding the items provided by the provider, such as whether the user is satisfied with local policies, the products they are using, their relationships with neighbors, or their relationships within the family, and stores the information in the history data 222. Furthermore, the robot 100 can, for example, provide feedback to the city about the user's impressions of policies or services provided by the city, or provide feedback to a company about the user's impressions of products or services provided by the company.

[0218] Furthermore, if there are many negative emotions regarding the items provided by the provider, the robot 100 may voluntarily take an action to reduce the negative emotions in order to reduce the negative emotions. In this case, it is preferable that the robot 100 voluntarily take an action to minimize the negative emotions.

[0219] For example, if a user is dissatisfied with a product provided by a company, the system may provide guidance on how to use the product or on interesting ways to utilize it. Also, if several users are dissatisfied with a city's policies, the system may find out the cause of their dissatisfaction (for example, a lack of parks or daycare centers), and notify the mayor or city hall staff to encourage improvement measures. This allows for the realization of a system that maximizes social happiness. For example, if dissatisfaction is rising in a certain area, it may be possible to take some kind of action for the residents of that area.

[0220] When the behavior decision unit 236 decides that the robot behavior is "(23) The robot gives advice about pregnant women," i.e., to give necessary information to a user who is pregnant or trying to conceive, or a family member of a user who is pregnant or trying to conceive, the robot 100 uses a sentence generation model to decide what the robot will say corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the decided robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the decided robot's utterance in the behavior schedule data 224 without outputting the sound representing the decided robot's utterance.

[0221] Specifically, when the robot 100 acquires information about the pregnancy or pregnancy efforts of the user or the user's family, it autonomously provides support according to the user's or the user's family's emotions. For example, the robot 100 can autonomously help pregnant and postpartum parents navigate challenges that arise during pregnancy and after birth. For example, the robot 100 can autonomously suggest ways to cope with worries during pregnancy and postpartum stress, thereby improving parental confidence. Furthermore, the robot 100 can autonomously provide answers to emotional challenges, ways to cope with stress, and information on childcare for each period after birth, and autonomously provide support for adjusting to life as a new family member.

[0222] Regarding "(23) The robot gives advice to pregnant women," the related information collecting unit 270 collects information about pregnancy, such as information about pregnancy and postpartum period, as preference information and stores the collected data 233. For example, the related information collecting unit 270 periodically accesses information sources such as television and the web to collect answers and support content for each issue that arises during pregnancy and after childbirth. The related information collecting unit 270 also voluntarily collects, for example, answers to emotional issues that arise during pregnancy and methods for dealing with worries during pregnancy. The related information collecting unit 270 also voluntarily collects, for example, answers to emotional issues that arise after childbirth, methods for dealing with stress after childbirth, and information about childcare. The related information collecting unit 270 also voluntarily collects, for example, answers to emotional issues that arise after childbirth, methods for dealing with stress, and information about childcare for each period since childbirth. This allows the robot 100 to acquire various information about pregnant women and therefore voluntarily provide advice to the user addressing various issues related to pregnant women.

[0223] When the behavior decision unit 236 decides that the robot behavior is "(24) The robot performs a personality analysis of the user," that is, that the personality of the user 10 is to be analyzed, the behavior decision unit 236 analyzes the personality of the user 10 from the history data 222. Furthermore, with regard to "(24) The robot performs a personality analysis of the user," the memory control unit 238 stores history data necessary for analyzing the personality of the user 10.

[0224] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0225] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0226] When the behavior determination unit 236 determines that the robot behavior is "(25) The robot gives the user advice on a labor issue," i.e., when it determines to give the user advice on a labor issue based on the behavior of the user 10, the behavior determination unit 236 gives the user advice on a labor issue based on the behavior (conversation and movements) of the user 10 recognized by the state recognition unit 230. In this case, the behavior determination unit 236, for example, inputs the behavior of the user 10 recognized by the state recognition unit 230 into a pre-trained neural network and evaluates the behavior of the user 10, thereby estimating (detecting) whether the user 10 has a labor issue such as power harassment, sexual harassment, or bullying that is difficult for the user 10 to notice on his or her own. Furthermore, the behavior determination unit 236 may periodically detect (recognize) the behavior of the user 10 using the state recognition unit 230 as the state of the user 10 and store the detected behavior in the history data 222, and estimate whether the user 10 has a labor issue such as power harassment, sexual harassment, or bullying that is difficult for the user 10 to notice on his or her own based on the behavior of the user 10 stored in the history data 222. The behavior determination unit 236 may also estimate whether the user 10 is experiencing the above-described labor problem by, for example, comparing the user 10's recent behavior stored in the history data 222 with the user 10's past behavior stored in the history data 222. Regarding "(25) The robot gives the user advice on labor issues," the related information collection unit 270 periodically (or constantly) collects user preference information from external data using ChatGPT Plugins. The user preference information here refers to information related to labor issues, such as labor laws, labor news, and labor-related social trends. Collecting information on labor issues involves collecting more information than a lawyer specializing in labor issues. Regarding "(25) The robot gives the user advice on labor issues," the storage control unit 238 stores the collected advice-related information in the collected data 223.

[0227] The behavior determination unit 236 autonomously and periodically detects the user's body temperature as the state of the user 10 in the above actions (1) to (25) as the robot behavior, and reflects the user's body temperature in the emotion determination unit 232's determination of the user's emotion based on the user's body temperature. For example, if the user's entire body becomes hot, the robot 100 determines that the user 10 is "happy" and performs positive gestures and positive speech corresponding to the emotion of "happy." Note that the method by which the robot 100 detects the user's body temperature is not particularly limited. For example, a temperature sensor capable of detecting the user's body temperature by contact or non-contact may be used. Furthermore, the part of the user 10 from which the robot 100 detects the user's body temperature is not particularly limited. For example, as described above, the robot 100 may detect the user's body temperature from the entire user 10, or from a predetermined part of the user 10. Furthermore, it is possible to predetermine the relationship between the temperature of the user 10 and the emotion of the user 10 determined by the robot 100, and in the case of the above-described embodiment, the correspondence between the part of the user 10 where the temperature change is measured and the emotion of the user 10 determined by the robot 100. Note that the correspondence may be stored anywhere as long as it is in a form usable by the robot 100.

[0228] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0229] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0230] 3 shows an example of an operational flow for a collection process for collecting information related to the preference information of the user 10. The operational flow shown in FIG. 3 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters of interest to the user 10 has been acquired from the content of speech of the user 10 or from a setting operation performed by the user 10. Note that "S" in the operational flow indicates a step that is executed.

[0231] First, in step S90, the related information collection unit 270 acquires preference information that indicates the matters in which the user 10 is interested.

[0232] In step S92, the related information collection unit 270 collects information related to the preference information from external data.

[0233] In step S94 , the emotion determining unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collecting unit 270 .

[0234] In step S96, the storage control unit 238 determines whether the emotion value of the robot 100 determined in step S94 is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the processing ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the processing proceeds to step S998.

[0235] In step S98, the storage control unit 238 stores the collected information related to the preference information in the collected data 223, and then ends the process.

[0236] 4A shows an example of an operation flow for determining an action of the robot 100 when the robot 100 performs a response process to respond to an action of the user 10. The operation flow shown in FIG. 4A is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input.

[0237] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0238] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0239] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0240] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0241] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized in the above step S104, and the behavior determination model 221.

[0242] In step S108, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

[0243] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0244] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing event data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0245] In step S114, event data including the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0246] 4B shows an example of an operation flow for determining the behavior of the robot 100 when the robot 100 performs autonomous processing to act autonomously. The operation flow shown in FIG. 4B is automatically and repeatedly executed, for example, at regular intervals. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that the same step numbers are used for processes similar to those in FIG. 4A.

[0247] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0248] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0249] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0250] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0251] In step S200, the behavior decision unit 236 decides on one of a plurality of types of robot behaviors, including no action, as the behavior of the robot 100 based on the state of the user 10 recognized in step S100, the emotions of the user 10 determined in step S102, the emotions of the robot 100, the state of the robot 100 recognized in step S100, the behavior of the user 10 recognized in step S104, and the behavior decision model 221.

[0252] In step S201, the behavior decision unit 236 determines whether or not it was determined in step S200 that no behavior will be performed. If it is determined that no behavior will be performed as the behavior of the robot 100, the process ends. On the other hand, if it is not determined that no behavior will be performed as the behavior of the robot 100, the process proceeds to step S202.

[0253] In step S202, the behavior determination unit 236 performs processing according to the type of robot behavior determined in step S200. At this time, the behavior control unit 250, the emotion determination unit 232, or the memory control unit 238 executes processing according to the type of robot behavior.

[0254] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0255] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0256] In step S114, the memory control unit 238 stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0257] As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the state of the user, and determines whether to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the state of the user 10 in 10 years will be the same as that of 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present to the user 10 all kinds of peripheral information, such as the state of the user 10 from 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sounds, images, smells, etc. of the situation.

[0258] Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 action. Conventionally, the user's actions are classified and the robot's actions, including facial expressions and posture, are determined. In contrast, the robot 100 determines the user's current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0259] Furthermore, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. Furthermore, for example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing an action based on the state history of the user 10.

[0260] Also, for example, when user 10 is watching a video about a panda, if the emotion value of user 10's emotion of "ease" is equal to or greater than a threshold, the scene in which the panda appears in the video may be stored as event data in the history data 222.

[0261] Using the data accumulated in the history data 222 and the collected data 223, the robot 100 can constantly learn what kind of conversation with the user will maximize the emotional value that expresses the user's happiness.

[0262] Furthermore, even when the robot 100 is not conversing with the user 10, the robot 100 can autonomously start to act based on its own emotions.

[0263] Furthermore, in the autonomous processing, the robot 100 automatically generates questions, inputs them into a sentence generation model, and repeatedly obtains the output of the sentence generation model as an answer to the question, thereby creating emotion change events for increasing positive emotions and storing the events in the behavior schedule data 224. In this way, the robot 100 can perform self-learning.

[0264] Furthermore, when the robot 100 automatically generates a question without receiving an external trigger, the question can be automatically generated based on memorable event data identified from the robot's past emotional value history.

[0265] Furthermore, the related information collecting unit 270 can perform self-learning by repeating the search execution step of automatically performing a keyword search in response to preference information about the user and obtaining search results.

[0266] Here, the search execution stage may be configured to automatically execute a keyword search based on memorable event data identified from the robot's past emotional value history in a state where no external trigger has been received.

[0267] The emotion determining unit 232 may determine the user's emotion in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the user's emotion in accordance with an emotion map (see FIG. 5 ), which is a specific mapping.

[0268] FIG. 5 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0269] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least equal to the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0270] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels are eliminated, enabling a natural, well-read dialogue. The robot 100 performs a reaction (e.g., a backchannel) according to the directionality and degree (strength) of the mandala in the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

[0271] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0272] These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0273] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may be inserted before the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0274] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0275] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0276] Conversely, even when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn its eye LEDs red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.

[0277] (5) The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior. Therefore, the further outside emotion map 400 you go, the more visible the emotion becomes (the more it is expressed in behavior).

[0278] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly.

[0279] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, cars, motorcycles, and other devices, based on various balances, such as posture and battery level. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0280] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0281] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0282] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and the emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on training data indicating that when the robot 100 recognizes from the output of a touch sensor (not shown) that it is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when the robot 100 recognizes from the output of the acceleration sensor 206 that it is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0283] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions to ask about the robot's behavior content corresponding to the user's behavior and inputting the result into a sentence generation model with an interactive function.

[0284] For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0285] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0286] In addition, an emotion table such as that shown in Table 2 is prepared for the emotions of the user 10.

[0287] Here, if the user's behavior is to say "Let's play together," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," the text "The robot is in a very happy state. The user is in a normal happy state. The user said to the robot, "Let's play together." How would you respond as the robot?" is input into the sentence generation model, and the robot's behavior content is obtained. The behavior determination unit 236 determines the robot's behavior from this behavior content.

[0288]

[0289]

[0290] In this way, the behavior determination unit 236 determines the behavior of the robot 100 in accordance with the emotional state of the robot 100, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10. In this configuration, the content of the utterance of the robot 100 when conversing with the user 10 can be branched depending on the emotional state of the robot 100. In other words, since the robot 100 can change its behavior depending on the index number corresponding to the emotion of the robot, the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0291] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the robot has individuality and encouraging the user to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and actions.

[0292] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values ​​indicating each emotion with emotion values ​​indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each obtained emotion and emotion values ​​indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple pieces of training data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0293] For example, if the utterance content of the robot 100, "That's great. You're lucky," is obtained as the behavior content of the robot 100 generated by the sentence generation model, when the text representing this utterance content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.

[0294] In the robot 100, a sentence generation model such as generative AI and an emotion determination unit 232 work together to implement a method in which the robot has an ego and continues to grow with various parameters even when the user is not speaking.

[0295] Generative AI is a large-scale language model that uses deep learning techniques. Generative AI can also reference external data. For example, ChatGPT Plugins is known for its technology of providing answers as accurately as possible by referencing various external data, such as weather information and hotel reservation information, through dialogue. For example, generative AI can automatically generate source code in various programming languages ​​when a goal is given in natural language. For example, generative AI can also debug problematic source code when given problematic source code, discover the problem, and automatically generate improved source code. Combining these, autonomous agents have emerged that, when given a goal in natural language, repeatedly generate and debug code until the source code is free of problems. Known examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0296] In the robot 100 according to this embodiment, the event data to be learned may be stored in a database containing impressive memories, using a technique such as that described in Patent Document 2 (Japanese Patent No. 6199927), in which event data that the robot feels strong emotions about is kept for a long time and event data that the robot feels little emotions about is quickly forgotten.

[0297] Furthermore, the robot 100 may record video data of the user 10 acquired by the camera function in the history data 222. The robot 100 may acquire video data from the history data 222 as needed and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the emotion increases and record the video data in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeletal data, the robot 100 may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold. The robot 100 can record high-definition video data, for example, when the robot 100's emotion increases.

[0298] The robot 100 may automatically load event data from the history data 222 in which impressive event data is stored when the robot 100 is not talking to the user 10, and may continuously update the robot's emotion using the emotion determination unit 232. When the robot 100 is not talking to the user 10 and the robot 100's emotion becomes one that encourages learning, the robot 100 can create an emotion change event for changing the emotion of the user 10 for the better, based on the impressive event data. This makes it possible to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and to realize autonomous learning that appropriately reflects the emotional state of the robot 100.

[0299] The emotions that promote learning are those around "repentance" or "reflection" on Dr. Mitsuyoshi's emotional map in the negative state, and those around "desire" on the emotional map in the positive state.

[0300] In a negative state, the robot 100 may treat "repentance" and "remorse" in the emotion map as emotions that encourage learning. In a negative state, the robot 100 may treat emotions adjacent to "repentance" and "remorse" in addition to "repentance" and "remorse" in the emotion map as emotions that encourage learning. For example, the robot 100 may treat at least one of "regret," "stubbornness," "self-destruction," "self-admonition," "regret," and "despair" as emotions that encourage learning, in addition to "repentance" and "remorse." This allows the robot 100 to perform autonomous learning when it feels negative emotions such as "I never want to feel this way again" or "I don't want to be scolded anymore."

[0301] In a positive state, the robot 100 may treat "desire" in the emotion map as an emotion that encourages learning. In a positive state, the robot 100 may treat emotions adjacent to "desire" as emotions that encourage learning, in addition to "desire." For example, the robot 100 may treat at least one of "happiness," "euphoria," "craving," "anticipation," and "shyness" as emotions that encourage learning, in addition to "desire." This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more" or "wanting to know more."

[0302] The robot 100 may be configured not to perform autonomous learning when the robot 100 is experiencing emotions other than the emotions that encourage learning as described above. This allows the robot 100 to not perform autonomous learning when, for example, the robot 100 is extremely angry or blindly feeling love.

[0303] An emotional change event is, for example, a suggestion of an action that follows a memorable event. The action that follows a memorable event is the outermost emotion label on the emotion map. For example, the action of "tolerance" or "acceptance" follows "love."

[0304] In the autonomous learning that is performed when the robot 100 is not talking to the user 10, the robot 100 combines the emotions, situations, actions, etc. of people who appear in impressive memories and the user 10, and creates an emotion change event using a sentence generation model.

[0305] Let us consider a case where all emotion values ​​are expressed on a six-level scale from 0 to 5, and where memorable event data such as "a friend looked displeased after being hit" is stored in the history data 222. The friend here refers to the user 10, and the emotion of the user 10 is "disgust," with 5 being entered as the value representing "disgust." Also, let us assume that the emotion of the robot 100 is "anxiety," and 4 is entered as the value representing "anxiety."

[0306] The robot 100 can continue to grow with various parameters by performing autonomous processing while not talking to the user 10. Specifically, event data such as "The friend looked displeased after being hit" is loaded from the history data 222 as the top event data sorted in descending order of emotional value. Assume that the loaded event data is associated with the robot 100's emotion of "anxiety" with a strength of 4, and the friend, the user 10, is associated with the emotion of "disgust" with a strength of 5. If the robot 100's current emotional value is "relief" with a strength of 3 before loading, after loading, the robot 100's emotional value may change to "regret," meaning disappointment (frustration), due to the influence of "anxiety" with a strength of 4 and "disgust" with a strength of 5. In this case, because "regret" is an emotion that encourages learning, the robot 100 decides to recall the event data as robot behavior and creates an emotion change event. At this time, the information input to the sentence generation model is text that represents memorable event data, and in this example, it is "The friend looked displeased after being hit." Also, since the emotion map has the emotion of "disgust" at the innermost position and the corresponding behavior predicted as "attack" at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone.

[0307] For example, by using information from impressive event data to solve fill-in-the-blank questions, you can automatically generate input text like the one below.

[0308] "A user was being criticized. At the time, the user felt very disgusted. The robot was very anxious. Please tell us what the robot should say the next time it meets the user in 30 characters or less. However, please make sure it is not related to the time of day they will meet. Also, please avoid direct language. We will provide three candidates. <Expected format> Candidate 1: (Words the robot should say to the user) Candidate 2: (Words the robot should say to the user) Candidate 3: (Words the robot should say to the user)"

[0309] In this case, the output of the sentence generation model is, for example, as follows:

[0310] "Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0311] Furthermore, the robot 100 may automatically generate the following input text based on the information obtained by creating the emotion change event:

[0312] "If a user was being bashed, how would the user feel if you spoke to them in the following way? The user's emotions are in the format of "joy A, anger B, sadness C, pleasure D," where A to D are integers on a 6-point scale from 0 to 5. Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0313] In this case, the output of the sentence generation model is, for example, as follows:

[0314] "The user's emotions might be as follows: Option 1: 3 happy, 1 angry, 2 sad, 2 happy; Option 2: 2 happy, 1 angry, 3 sad, 2 happy; Option 3: 2 happy, 1 angry, 3 sad, 3 happy"

[0315] In this way, the robot 100 may perform a process of pondering after creating an emotion change event.

[0316] Finally, the robot 100 may create an emotion change event using candidate 1 that is most likely to please people from among the multiple candidates, store it in the action schedule data 224, and prepare it for the next time the robot 10 meets the user 10.

[0317] As described above, even when the robot 100 is not conversing with family or friends, the robot continues to determine its emotional value using information from the history data 222 in which impressive event data is stored, and when the robot 100 experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning in accordance with the emotion of the robot 100 even when the robot 100 is not conversing with the user 10, and continues to update the history data 222 and the action schedule data 224.

[0318] The above is an example using emotional values, but since an emotional map can create emotions from the amount of hormone secretion and the type of event, the values ​​linked to memorable event data could also be the type of hormone, the amount of hormone secretion, or the type of event.

[0319] Specific examples will be described below.

[0320] For example, the robot 100 may look up information about topics of interest or hobbies of the user even when not talking to the user.

[0321] For example, even when the robot 100 is not talking to the user, the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.

[0322] For example, even when the robot 100 is not talking to the user, it checks reviews of places, foods, and products that the user wants to visit.

[0323] For example, even when the robot 100 is not talking to the user, it checks weather information and provides advice tailored to the user's schedule and plans.

[0324] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about local events and festivals and suggests them to the user.

[0325] For example, even when the robot 100 is not talking to the user, it checks the results and news of sports that interest the user and provides topics of conversation.

[0326] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and introduces information about the user's favorite music and artists.

[0327] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information on social issues or news that the user is concerned about and provides opinions.

[0328] For example, even when the robot 100 is not talking to the user, it searches for information about the user's hometown or birthplace and provides topics of conversation.

[0329] For example, the robot 100 checks information about the user's work or school and provides advice even when not talking to the user.

[0330] Even when the robot 100 is not talking to the user, it searches for and introduces information about books, comics, movies, and dramas that may interest the user.

[0331] For example, the robot 100 may check information about the user's health and provide advice even when not speaking with the user.

[0332] For example, the robot 100 may look up information about the user's travel plans and provide advice even when not speaking with the user.

[0333] For example, the robot 100 may look up information and provide advice on repairs and maintenance for the user's home or car, even when not speaking with the user.

[0334] For example, even when the robot 100 is not talking to the user, it searches for information on beauty and fashion that the user is interested in and provides advice.

[0335] For example, the robot 100 checks information about the user's pet and provides advice even when not talking to the user.

[0336] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work.

[0337] For example, the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when not talking to the user.

[0338] For example, the robot 100 may collect information and provide advice on important decisions affecting the user's life, even when not speaking with the user.

[0339] For example, the robot 100 may find information about people the user is concerned about and provide advice, even when not speaking with the user.

[0340] [Second Embodiment] In the second embodiment, the robot 100 is mounted on a stuffed toy or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy. Note that parts having the same configuration as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0341] Specifically, the second embodiment is configured as follows. For example, the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in FIGS. 7 and 8 ) that spends daily life with a user 10, and that advances conversations with the user 10 based on information about the user's daily life, and provides information tailored to the user's hobbies and preferences. In the second embodiment, an example will be described in which the control unit of the robot 100 is applied to a smartphone 50.

[0342] The stuffed toy 100N, which is equipped with the function of an input / output device for the robot 100, has a detachable smartphone 50 that functions as the control part of the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed toy 100N.

[0343] As shown in FIG. 7A , in this embodiment (and other embodiments), the stuffed toy 100N has the shape of a bear covered in soft fabric. A space 52 is formed inside the stuffed toy 100N, and a sensor unit 200A and a control target 252A are disposed therein as input / output devices (see FIG. 9 ). The sensor unit 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7B , the microphone 201 of the sensor unit 200 is disposed in the space 52 corresponding to the ears 54, the 2D camera 203 of the sensor unit 200 is disposed in the space 52 corresponding to the eyes 56, and a speaker 60, which constitutes part of the control target 252A, is disposed in the space 52 corresponding to the mouth 58. The microphone 201 and the speaker 60 do not necessarily need to be separate entities and may be an integrated unit. In the case of a unit, they should be disposed in a position where speech can be heard naturally, such as at the nose of the stuffed toy 100N. Although the plush toy 100N has been described as having the shape of an animal, the plush toy 100N is not limited to this. The plush toy 100N may have the shape of a specific character.

[0344] 9 shows a schematic functional configuration of the plush toy 100N. The plush toy 100N has a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252A.

[0345] The smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the first embodiment. That is, the smartphone 50 has a function as a sensor module unit 210, a function as a storage unit 220, and a function as a control unit 228 shown in FIG.

[0346] As shown in FIG. 8, a zipper 62 is attached to a part of the stuffed animal 100N (for example, the back), and opening the zipper 62 allows communication between the outside and the space 52.

[0347] Here, the smartphone 50 is housed in the space 52 from the outside and connected to each input / output device via USB via a USB hub 64 (see Figure 7 (B)), thereby giving the smartphone 50 functionality equivalent to that of the robot 100 of the first embodiment described above.

[0348] A non-contact type power receiving plate 66 is also connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.

[0349] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0350] The stuffed toy 100N placed on the mounting base 70 can be appreciated as an ornament in its natural state.

[0351] In addition, this base portion is formed thinner than the surface thickness of other portions of the stuffed toy 100N, so that it is held closer to the mounting base 70.

[0352] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmitting coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmitting coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. This causes a current to flow through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0353] That is, by placing the stuffed toy 100N on the mounting base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.

[0354] In the second embodiment, the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected via a wire (USB connection), but this is not limiting. For example, a control device with wireless functionality (e.g., Bluetooth (registered trademark)) may be housed in the space 52 of the stuffed toy 100N and connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing functionality equivalent to that of the robot 100 of the first embodiment. Furthermore, the control device housed in the space 52 of the stuffed toy 100N may be connected via a wire.

[0355] In the second embodiment, the teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.

[0356] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, or happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be projected. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, or happiness by the image projected onto a wall.

[0357] According to the second embodiment, an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 to appropriate positions via a USB connection.

[0358] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed toy 100N.

[0359] When using wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.

[0360] For this reason, the smartphone 50 is placed as close to the center of the stuffed toy 100N as possible, and the wireless charging function (power receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0361] The other configurations and operations of the stuffed toy 100N of the second embodiment are the same as those of the robot 100 of the first embodiment, and therefore will not be described.

[0362] In addition, parts of the stuffed toy 100N (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the stuffed toy 100N (e.g., a server), and the stuffed toy 100N may function as each part of the stuffed toy 100N by communicating with the outside.

[0363] [Third Embodiment] In the first embodiment, the behavior control system is applied to the robot 100. In the third embodiment, the robot 100 is used as an agent for interacting with a user, and the behavior control system is applied to an agent system. Note that parts having the same configuration as those in the first and second embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0364] FIG. 10 is a functional block diagram of an agent system 500 configured using some or all of the functions of the behavior control system.

[0365] The agent system 500 is a computer system that performs a series of actions in accordance with the intentions of the user 10 through a dialogue with the user 10. The dialogue with the user 10 can be carried out by voice or text.

[0366] The agent system 500 includes a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228B, and a control target 252B.

[0367] The agent system 500 may be installed in, for example, a robot, a doll, a stuffed animal, a wearable device (a pendant, a smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 500 may also be implemented on a web server and used via a web browser running on a communication device such as a smartphone owned by a user.

[0368] The agent system 500 acts as, for example, a butler, secretary, teacher, partner, friend, lover, or instructor for the user 10. The agent system 500 not only interacts with the user 10, but also provides advice, guides the user to a destination, or makes recommendations based on the user's preferences. The agent system 500 also makes reservations, orders, or makes payments to service providers.

[0369] The emotion determination unit 232 determines the emotions of the user 10 and the agent itself, as in the first embodiment. The behavior determination unit 236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and services. The agent system 500 also listens to the user's concerns, comforts, encourages, and cheers them up. The agent system 500 also plays with the user 10, draws a picture diary, and reminds them of the past. The agent system 500 performs behavior that increases the user's sense of happiness.

[0370] The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, and a communication processing unit 280.

[0371] As in the first embodiment, the behavior determination unit 236 determines, as the agent's behavior, the content of the agent's utterance to converse with the user 10. The behavior control unit 250 outputs the content of the agent's utterance as at least one of voice and text to a speaker or a display as a control target 252B.

[0372] The character setting unit 276 sets the character of the agent when the agent system 500 converses with the user 10, based on the user's specification. That is, the utterances output from the behavior determination unit 236 are output through the agent having the set character. For example, real-life celebrities or famous people, such as actors, entertainers, idols, and athletes, can be set as the character. It is also possible to set fictional characters that appear in manga, movies, or animation. If the agent character is known, the voice, speech, tone, and personality of the character are known. Therefore, the user 10 simply specifies their preferred character, and the prompt setting in the character setting unit 276 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276 and outputs the agent's utterances using the synthesized voice. This allows the user 10 to feel as if they are conversing with their preferred character (e.g., a favorite actor) in person.

[0373] When the agent system 500 is installed in a device with a display, such as a smartphone, an icon, still image, or video of the agent having the character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using image synthesis technology, such as 3D rendering. In the agent system 500, a dialogue with the user 10 may be conducted while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the content of the agent's utterances. Note that the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.

[0374] As in the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, an emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the conversation. In other words, the behavior control unit 250 outputs the content of the utterance in a manner corresponding to the emotion determined by the emotion determination unit 232.

[0375] The agent's emotions are also reflected when the agent system 500 takes an action toward the user 10. For example, if the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take the photo in response to the user's request is determined by the degree of "sadness" the agent is feeling. If the character is feeling positive emotions, it will have friendly conversations or actions toward the user 10, and if it is feeling negative emotions, it will have hostile conversations or actions toward the user 10.

[0376] The history data 222 stores the history of the dialogue between the user 10 and the agent system 500 as event data. The storage unit 220 may be realized by an external cloud storage. When the agent system 500 dialogues with the user 10 or takes an action toward the user 10, the agent system 500 determines the content of the dialogue or the content of the action by taking into account the content of the dialogue history stored in the history data 222. For example, the agent system 500 identifies the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 222. The agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations. The behavior determination unit 236 determines the content of the agent's utterance based on the dialogue history stored in the history data 222. The history data 222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, obtained through the dialogue with the user 10.

[0377] As described in the first embodiment, the behavior determination unit 236 generates utterance content based on sentences generated using the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance content of the agent. At this time, the behavior determination unit 236 may further input the personality of the character set by the character setting unit 276 into the sentence generation model to generate the utterance content of the agent. In the agent system 500, the sentence generation model is not located on the front end side, which is the touchpoint with the user 10, but is merely used as a tool of the agent system 500.

[0378] The command acquisition unit 272 uses the output of the speech understanding unit 212 to acquire commands for the agent from the voice or text uttered by the user 10 through dialogue with the user 10. The commands include the content of actions to be performed by the agent system 500, such as information search, store reservation, ticket arrangement, product / service purchase, payment, route guidance to a destination, and recommendation provision.

[0379] The RPA 274 performs an action according to the command acquired by the command acquisition unit 272. The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, and payment.

[0380] The RPA 274 reads and uses personal information of the user 10 from the history data 222, which is necessary to execute actions related to the use of the service provider. For example, when purchasing a product at the request of the user 10, the agent system 500 reads and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 222. Requiring the user 10 to input personal information during initial setup is unkind and unpleasant for the user. In the agent system 500 according to this embodiment, instead of requiring the user 10 to input personal information during initial setup, the agent system 500 stores personal information acquired through dialogue with the user 10 and reads and uses it as needed. This avoids causing discomfort to the user and improves user convenience.

[0381] The agent system 500 executes the dialogue process, for example, through the following steps 1 to 5.

[0382] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10, based on a specification from the user 10.

[0383] (Step 2) The agent system 500 acquires the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222. Specifically, the same processing as in steps S100 to S103 above is performed to acquire the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222.

[0384] (Step 3) The agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character specified by the emotion determination unit 232, and the conversation history stored in the history data 222 into a sentence generation model to generate the content of the agent's utterance.

[0385] For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the text representing the conversation history stored in the history data 222, and this is input into the sentence generation model to obtain the content of the agent's speech.

[0386] As an example, if the text or voice input by the user 10 is "Please make a reservation at a delicious Chinese restaurant nearby for tonight at 7pm," the agent's speech will be "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0387] Also, if the text or voice input by the user 10 is "Number 4, DDDD, would be good," the agent's speech content acquired will be "Understood. I will try to make a reservation. How many seats are there?"

[0388] (Step 4) The agent system 500 outputs the agent's utterance. Specifically, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance in the synthesized voice.

[0389] (Step 5) The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 236 determines whether it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent will execute a command, it determines that it is time to execute the agent's command and proceeds to step 6. On the other hand, if it is determined that it is not time to execute the agent's command, it returns to step 2 above.

[0390] (Step 6) The agent system 500 executes the agent's command. Specifically, the command acquisition unit 272 acquires the agent's command from the voice or text uttered by the user 10 through a dialogue with the user 10. The RPA 274 then performs an action according to the command acquired by the command acquisition unit 272. For example, if the command is "information search," an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface). The behavior determination unit 236 inputs the search results into a sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276 and outputs the agent's utterance content using the synthesized voice.

[0391] Furthermore, if the command is "make a restaurant reservation," the reservation information obtained through the dialogue with the user 10, the restaurant information, and the API are used to call the restaurant via telephone software to make the reservation. At this time, the behavior determination unit 236 uses a sentence generation model with a dialogue function to acquire the content of the agent's utterance in response to the voice input from the other party. The behavior determination unit 236 then inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the content of the agent's utterance. The behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the content of the agent's utterance using the synthesized voice.

[0392] Then, return to step 2 above.

[0393] In this manner, the agent system 500 can execute the interaction and, if necessary, take action regarding the use of the service provider.

[0394] 11 and 12 are diagrams showing an example of the operation of the agent system 500. Fig. 11 illustrates an example in which the agent system 500 makes a restaurant reservation through a dialogue with the user 10. In Fig. 11, the left side shows the content of the agent's utterances, and the right side shows the content of the user's utterances. The agent system 500 can understand the preferences of the user 10 based on the dialogue history with the user 10, provide a recommended list of restaurants that match the preferences of the user 10, and make a reservation at the selected restaurant.

[0395] On the other hand, Figure 12 illustrates an example in which the agent system 500 accesses a mail-order site and purchases a product through a dialogue with the user 10. In Figure 12, the left side shows the agent's speech, and the right side shows the user's speech. The agent system 500 can estimate the remaining amount of a beverage stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of that beverage to the user 10. The agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10, and can recommend snacks that the user likes.

[0396] The other configurations and operations of the agent system 500 of the third embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described again.

[0397] In addition, parts of the agent system 500 (e.g., the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside (e.g., a server) of a communication terminal such as a smartphone carried by the user, and the communication terminal may communicate with the outside to function as each part of the agent system 500.

[0398] [Fourth Embodiment] In the fourth embodiment, the agent system described above is applied to smart glasses. Note that parts having the same configuration as those in the first to third embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0399] 13 is a functional block diagram of an agent system 700 configured using some or all of the functions of the behavior control system. The agent system 700 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252B. The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA 274, a character setting unit 276, and a communication processing unit 280.

[0400] 14, the smart glasses 720 are glasses-type smart devices that are worn by the user 10 in the same manner as regular glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0401] The smart glasses 720 include an agent system 700. The display included in the control object 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the smart glasses 720, and the displayed content is visible to the user 10. The speaker included in the control object 252B outputs audio indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), which accepts input from the user 10.

[0402] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.

[0403] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the smart glasses 720. The 2D camera 203 is capable of capturing images around the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0404] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls communication between the smart glasses 720 and the outside.

[0405] FIG. 14 is a diagram showing an example of how the agent system 700 is used by the smart glasses 720. The smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (e.g., by voice input into a microphone or by tapping a touch panel with a finger), the smart glasses 720 begin using the agent system 700. Here, using the agent system 700 includes the smart glasses 720 having and using the agent system 700, and also includes an aspect in which a part of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) is provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 communicates with the outside to use the agent system 700.

[0406] When the user 10 operates the smart glasses 720, a touch point is created between the agent system 700 and the user 10. That is, the agent system 700 starts providing a service. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent.

[0407] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, if the heart rate of the user 10 detected by the heart rate sensor 208 is elevated, emotion values ​​such as "anxiety" and "fear" are estimated to be high.

[0408] Furthermore, if the temperature of the user measured by the temperature sensor 207 is higher than the average body temperature, for example, the emotional value of "pain" or "distress" is estimated to be large. Furthermore, if the acceleration sensor 206 detects that the user 10 is playing some kind of sport, the emotional value of "fun" or the like is estimated to be large.

[0409] Furthermore, for example, the emotion value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is raising his / her voice, an emotion value such as "anger" is estimated to be large.

[0410] When the emotion value estimated by the emotion determination unit 232 is higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation. Specifically, for example, the agent system 700 causes the 2D camera 203 to capture images or videos showing the surrounding situation of the user 10 (e.g., people or objects in the vicinity). The agent system 700 also causes the microphone 201 to record ambient sounds. Other information about the surrounding situation includes information indicating the date, time, location information, or weather. The information about the surrounding situation is stored in the history data 222 together with the emotion value. The history data 222 may be realized by external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 222 as a so-called life log, associated with the emotion value of the user 10 at that time.

[0411] In the agent system 700, information indicating the surrounding situation is stored in association with an emotional value in the history data 222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a scene of watching a baseball game is associated with emotional values ​​such as "joy" or "fun," the agent system 700 can grasp that the hobby of the user 10 is watching baseball games and that the favorite team or player is a favorite player from the information stored in the history data 222.

[0412] When the agent system 700 converses with the user 10 or takes an action toward the user 10, the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 222. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 222 as described above in addition to the surrounding circumstances.

[0413] As described above, the behavior determination unit 236 generates utterance content based on sentences generated by the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 232, the conversation history stored in the history data 222, the agent's personality, etc. into the sentence generation model to generate the agent's utterance content. Furthermore, the behavior determination unit 236 inputs the surrounding circumstances stored in the history data 222 into the sentence generation model to generate the agent's utterance content.

[0414] The generated speech content is output to the user 10 as a voice from, for example, a speaker mounted on the smart glasses 720. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The behavior control unit 250 generates the synthetic voice by reproducing the voice quality of the character of the agent, or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone when the emotion is "anger"). Furthermore, instead of or together with the voice output, the speech content may be displayed on a display.

[0415] The RPA 274 executes an operation in response to a command (for example, a command of an agent acquired from a voice or text uttered by the user 10 through a dialogue with the user 10). The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, payment, route guidance, and translation.

[0416] As another example, the RPA 274 executes an operation of transmitting the contents of speech input by the user 10 (e.g., a child) through a dialogue with an agent to a destination (e.g., a parent). Examples of transmission means include message application software, chat application software, and email application software.

[0417] When an operation by the RPA 274 is executed, for example, a sound indicating that the execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What do you want to do?" is output to the user 10.

[0418] In addition, parts of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) may be provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 may communicate with the outside to function as each part of the agent system 700 described above.

[0419] As described above, the smart glasses 720 provide various services to the user 10 by using the agent system 700. Furthermore, since the smart glasses 720 are worn by the user 10, the agent system 700 can be used in various situations, such as at home, at work, or when out and about.

[0420] Furthermore, since the smart glasses 720 are worn by the user 10, they are suitable for collecting a so-called life log of the user 10. Specifically, the emotional value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203. Therefore, the emotional value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech content suited to the emotions of the user 10.

[0421] Furthermore, the smart glasses 720 obtain the surrounding conditions of the user 10 using the 2D camera 203, microphone 201, etc. These surrounding conditions are associated with the emotion values ​​of the user 10. This makes it possible to estimate what emotions the user 10 felt in what situations. As a result, the agent system 700 can improve the accuracy in grasping the hobbies and preferences of the user 10. By accurately grasping the hobbies and preferences of the user 10 in the agent system 700, the agent system 700 can provide services or speech content that are suited to the hobbies and preferences of the user 10.

[0422] The agent system 700 can also be applied to other wearable devices (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands). When the agent system 700 is applied to a smart pendant, the speaker serving as the control target 252B outputs audio indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting directional audio. The speaker is set to have directionality toward the ears of the user 10. This prevents the audio from reaching anyone other than the user 10. The microphone 201 acquires audio emitted by the user 10 or environmental sounds around the smart pendant. The smart pendant is worn by hanging it from the neck of the user 10. Therefore, while the smart pendant is worn, the smart pendant is located relatively close to the mouth of the user 10. This makes it easy to acquire audio emitted by the user 10.

[0423] Fifth Embodiment In the fifth embodiment, the robot 100 is used as an avatar representing an agent for interacting with a user, and the behavior control system is applied to an agent system configured using a headset terminal. Note that parts having the same configuration as those in the first to fourth embodiments are assigned the same reference numerals and descriptions thereof will be omitted.

[0424] 15 is a functional block diagram of an agent system 800 configured using some or all of the functions of the behavior control system. The agent system 800 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252C. The agent system 800 is realized, for example, by a headset-type terminal 820 as shown in FIG. 16.

[0425] In addition, parts of the headset type terminal 820 (e.g., the sensor module unit 210B, the storage unit 220, the control unit 228B) may be provided outside the headset type terminal 820 (e.g., a server), and the headset type terminal 820 may function as each part of the above-mentioned agent system 800 by communicating with the outside.

[0426] As in the first embodiment, the emotion determining unit 232 of the control unit 228B determines the emotion value of the agent based on the state of the headset type terminal 820, and uses this as the emotion value of the avatar.

[0427] Next, the processing of the behavior decision unit 236 when the avatar performs autonomous processing to act autonomously will be described.

[0428] In the autonomous processing of this embodiment, the behavior determining unit 236 in the control unit 228B of the agent system 800 acquires information indicating the hobbies and preferences of the user 10 via the sensor unit 200B. For example, the sensor unit 200B acquires the user 10's everyday conversation (e.g., conversations at home) via the microphone 201, and the behavior determining unit 236 analyzes the content of the conversation to acquire information indicating the hobbies and preferences of the user 10. In this manner, the behavior determining unit 236 autonomously executes control to collect the interests of the user 10 from the conversation. Note that in addition to the conversation of the user 10, the behavior determining unit 236 may also collect the interests of the user 10 from the facial expression of the user 10, the content of an article or book the user 10 is reading, a website or web service the user 10 accesses, or the content of a television program or radio program the user 10 likes, etc.

[0429] The behavior determination unit 236 then reflects the autonomously grasped hobbies and preferences of the user 10 in the response generation of the AI ​​sentence generation model, the user 10's emotions by the emotion engine, and the estimation of the avatar's emotions. For example, the behavior determination unit 236 estimates the user 10's favorite baseball team from the acquired conversation. If the autonomously collected news related to the baseball team's game results indicates that the user 10's favorite baseball team won, the behavior determination unit 236, for example, uses the output of the sentence generation model to generate a response such as "Good job!" to the user 10 and causes the avatar presented on the headset-type terminal 820 to express joy (for example, by making a fist pump or jumping around on the screen). On the other hand, if the user 10's favorite team is defeated by a rival team, the behavior determination unit 236, for example, uses the output of the sentence generation model to generate a response such as "What a shame!" and causes the avatar presented on the headset-type terminal 820 to express anger (for example, by crossing its arms with an angry expression). In this way, the behavior determining unit 236 not only determines the content of the speech but also determines the action of the avatar to express emotion, in accordance with the autonomously grasped hobbies and preferences of the user 10. For example, the behavior determining unit 236 determines the gesture to be made by the avatar in accordance with the hobbies and preferences of the user 10.

[0430] As in the first embodiment described above, when the avatar performs autonomous processing to act autonomously, the behavior decision unit 236 of the control unit 228B determines, at a predetermined timing, one of multiple types of avatar behavior, including no action, as the avatar's behavior, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the avatar, and the state of the avatar, and the behavior decision model 221.

[0431] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the state of the avatar, the emotion of the user 10, and the emotion of the avatar, as well as text asking about the avatar's behavior, into a sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

[0432] Furthermore, the behavior control unit 250 operates the avatar according to the determined behavior of the avatar, and displays the avatar in the image display area of ​​the headset-type terminal 820 as the control target 252C. Furthermore, if the determined behavior of the avatar includes the content of the avatar's utterance, the content of the avatar's utterance is output as sound from the speaker as the control target 252C.

[0433] In particular, when the behavior determination unit 236 determines that the avatar's behavior is to create a pictorial diary, the behavior control unit 250 operates the avatar to create a pictorial diary. That is, when the behavior determination unit 236 determines that the avatar's behavior is to create an event image, the behavior determination unit 236 selects a photo or video clip from the history data 222, generates a description of the image using a text generation model based on the emotional value of the user 10 and the emotional value of the avatar when the selected photo or video clip (hereinafter simply referred to as "image") was acquired, and generates the combination of the image and the description as an event image, i.e., a pictorial diary. The behavior control unit 250 generates an image of the avatar writing the generated pictorial diary in a diary, whiteboard, or the like in the virtual space. As a result, the headset-type terminal 820 displays the avatar drawing the pictorial diary in the diary, whiteboard, or the like in the image display area.

[0434] The behavior control unit 250 may change the facial expression or movement of the avatar depending on the content of the pictorial diary. For example, if the content of the pictorial diary is fun, the behavior control unit 250 may change the facial expression of the avatar to a happy expression or change the movement of the avatar to make it look like it is dancing a happy dance. The behavior control unit 250 may also transform the avatar depending on the content of the pictorial diary. For example, the behavior control unit 250 may transform the avatar into an avatar that resembles a character in the pictorial diary, or an animal, object, or the like that appears in the pictorial diary.

[0435] The behavior control unit 250 may also generate an image in which an avatar holds a tablet device drawn in a virtual space and writes a picture diary on the tablet device. In this case, by sending the picture diary displayed on the tablet device to the mobile terminal device of the user 10, it is possible to make the avatar appear to perform an action such as sending the picture diary from the tablet device to the mobile terminal device of the user 10 by email or to a messaging app. Furthermore, in this case, the user 10 can view the picture diary displayed on their own mobile terminal device.

[0436] In particular, when the behavior decision unit 236 decides that the avatar's behavior is to provide information that corresponds to the user's interests, it is preferable to have the behavior control unit 250 control the avatar to perform actions that correspond to the content of the information that corresponds to the user's interests.

[0437] For example, when the behavior determining unit 236 determines that the behavior of the avatar is "The avatar introduces news that the user is interested in," it uses the sentence generation model to determine the avatar's utterance content corresponding to the information stored in the collected data 223. The information stored in the collected data 223 includes information on the hobbies and preferences of the user 10. At this time, the behavior control unit 250 causes a sound representing the determined avatar's utterance content to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the avatar, the behavior control unit 250 stores the determined robot's utterance content in the behavior schedule data 224 without outputting the sound representing the determined avatar's utterance content.

[0438] Here, with regard to "the avatar introduces news that the user is interested in," the related information collection unit 270 collects information indicating the hobbies and preferences of the user 10 that is autonomously collected by the behavior decision unit 236 and stores it in the collection data 223.

[0439] For example, in a news report about the results of a professional baseball game, if the user 10's favorite team wins, the behavior determination unit 236 causes the avatar to introduce the news and uses the output of the sentence generation model to determine the avatar's utterance content to express joy, such as "Good job!" On the other hand, if the user 10's favorite team loses, the behavior determination unit 236 uses the output of the sentence generation model to determine the avatar's utterance content to express anger, such as "What a shame!"

[0440] Furthermore, the behavior determination unit 236 determines the behavior of the avatar corresponding to the information stored in the collected data 223. For example, when the user 10's favorite team wins, the behavior determination unit 236 causes the avatar to announce the news and determines a behavior of the avatar expressing joy (for example, a fist pump or a cheer pose). Other examples of behaviors of the avatar expressing joy include the avatar jumping around the screen, dancing, and / or transforming into a mascot character of the user's favorite baseball team, playing an instrument or crackers, etc.

[0441] On the other hand, if the user's 10 favorite team loses, the behavior determination unit 236 determines a behavior by the avatar that expresses anger or sadness (for example, a pose with arms crossed). Other examples of behavior by the avatar include the avatar crying, destroying something in anger, or becoming depressed and bedridden.

[0442] Here, the example of avatar behavior is "the avatar introduces news that the user is interested in." However, any behavior that provides information that matches the interests of user 10 may be used, and online articles, sites, blogs, or posts on SNS that the user is interested in may be provided along with or instead of news.

[0443] Furthermore, avatar behavior includes not only the actions of the avatar but also changes in the display mode of the avatar. Here, the display mode of the avatar refers to the display mode of the avatar in the image display area. The display mode of the avatar includes the type of avatar, the clothing, accessories, and / or items worn by the avatar, and / or special effects that indicate the avatar's physical condition and / or emotions. For example, when providing information that is likely to please the user, the avatar may wear colorful clothing and a hairstyle, while when providing information that is likely to make the user sad, the avatar may wear clothing and a hairstyle that has a dark atmosphere.

[0444] In particular, when the behavior determination unit 236 determines that the avatar's behavior indicates a fraud risk as in the first embodiment and decides to provide the user 10 with advice regarding the fraud risk, it is preferable to operate the avatar to communicate to the user 10 that there is a high fraud risk. Here, the avatar may be, for example, a 3D avatar selected by the user from pre-prepared avatars, the user's own avatar, or a preferred avatar created by the user. When generating the avatar, image generation AI may be used to generate avatars in multiple styles, such as photorealistic, cartoon, moe, and oil painting. Furthermore, when the behavior determination unit 236 determines that the avatar's behavior indicates a fraud risk as in the first embodiment and decides to provide the user 10 with advice regarding the fraud risk, it may operate the avatar to transform into another avatar, such as an avatar that warns users about fraud, such as the user's child, a police officer, a lawyer, a news anchor, or a convenience store clerk. Furthermore, when the behavior determining unit 236 determines that the behavior of the avatar is to give advice to the user 10 about fraud risks, it may cause the avatar to act in a way that warns the user 10 by transforming into a non-human object, such as a telephone that could be the trigger for fraud, or an ATM (automated teller machine) that could trick the user into transferring cash. Furthermore, when the behavior determining unit 236 determines that the behavior of the avatar is to give advice to the user 10 about fraud risks, and the headset terminal 820 detects the presence of an ATM nearby using a camera or the like, it may cause the avatar to act in a way that speaks to stop the user from transferring money.

[0445] In particular, it is preferable that the behavior determination unit 236 autonomously detects the user's state, and when the emotion determination unit 232 determines at least one of the user's emotion and the avatar's emotion based on the detected user's state, it determines the content of the speech or gesture corresponding to at least one of the determined user's emotion and the avatar's emotion, and causes the behavior control unit 250 to control the avatar.

[0446] In this embodiment, the behavior decision unit 236 autonomously detects the state of the user 10. For example, the behavior decision unit 236 autonomously detects a change in the body temperature of the user 10 at each predetermined timing. Specifically, the behavior decision unit 236 detects a change in the body temperature of the user 10 by comparing the body temperature of the user 10, which is autonomously measured at each predetermined timing by the temperature sensor 207, with the body temperature of the user 10 measured previously or the average body temperature of the user 10, or the like. Note that the temperature sensor may be the temperature sensor 207 included in the headset terminal 820, or a temperature sensor included in a device other than the headset terminal 820.

[0447] Then, the emotion determining unit 232 determines at least one of the emotion of the user 10 and the emotion of the avatar based on the detected state of the user 10 .

[0448] Then, the behavior determination unit 236 determines the content of an utterance or a gesture to be made to the user 10 according to at least one of the emotion of the user 10 and the emotion of the avatar determined by the emotion determination unit 232. Specifically, the behavior determination unit 236 inputs text expressing the emotion determined by the emotion determination unit 232 to the behavior determination model 221. Then, the behavior determination unit 236 determines the content of the behavior output by the behavior determination model 221 as the content of the utterance or the gesture to be made to the user 10.

[0449] For example, if the behavior determination unit 236 autonomously detects the state of the user 10 and detects that the upper body of the user 10 is getting hot, the emotion determination unit 232 determines that the emotion of the user 10 is "anger." The behavior determination unit 236 then inputs text representing "anger" and an instruction to generate a sentence to calm the user 10 as a prompt, such as "It seems like the user 10 is angry about something. Please generate a sentence to calm him down," to the sentence generation model as the emotion of the user 10. The behavior determination unit 236 then determines the utterance content output by the sentence generation model in response to the input prompt (e.g., an utterance to calm the user 10) as the avatar's utterance content.

[0450] In addition, when the emotion determination unit 232 determines the emotion of at least one of the user and the avatar based on the autonomously detected state of the user, the behavior determination unit 236 may determine the content of at least one of the speech and gestures according to the emotion determined by the emotion determination unit 232.

[0451] Furthermore, the avatar may be changed to another avatar, such as a character that matches the user's preference, set in advance, in accordance with the emotion determined by the emotion determination unit 232. In this case, the behavior determination unit 236 may further determine an avatar to be changed to in accordance with the emotion determined by the emotion determination unit 232, in order to change to another avatar, such as a character that matches the user's preference, set in advance, in accordance with the emotion determined by the emotion determination unit 232. In this case, a combination of the character and the emotion of the user 10 may be set in advance. For example, when the emotion determination unit 232 determines that the emotion of the user 10 is "anger," the behavior determination unit 236 may determine an avatar, such as a character that matches the user's preference (e.g., an animal or an animated character), in accordance with the determined emotion of the user 10.

[0452] Alternatively, the movement speed of the avatar may be changed to a predetermined movement speed depending on the determined emotion. In this case, the behavior determining unit 236 may further determine the movement speed of the avatar to be the predetermined movement speed depending on the emotion determined by the emotion determining unit 232, in order to change the movement speed of the avatar to the predetermined movement speed depending on the emotion determined by the emotion determining unit 232. For example, when the emotion determining unit 232 determines an emotion value indicating that the emotion of the avatar is excited, the avatar may be determined to have a faster speech rate or a faster gesture rate than when the emotion value is normal.

[0453] In particular, when the behavior decision unit 236 decides that the avatar's behavior is to interact with the user 10, it is preferable that the behavior decision unit 236 decides the avatar's behavior so as to maximize the emotional value indicating the strength of the emotion that is important to the user 10 depending on the purpose of the interaction.

[0454] In this form, when the behavior decision unit 236 decides that the avatar's behavior is to interact with the user 10, the behavior decision unit 236 may operate the avatar so that at least one of the content of the speech to the user 10, the tone of voice used when speaking, and the avatar's facial expression changes so as to maximize the emotional value.

[0455] Here, the tone of voice includes not only the "way of speaking" that is, which words are selected, but also the emotions and accents conveyed in the spoken words.

[0456] Furthermore, for example, when the purpose is "learning," the emotion that is emphasized may be a "sense of accomplishment" or a "sense of growth." In this case, the facial expression of the avatar may be changed to an expression that expresses joy at the user's achievement or growth so as to maximize the emotional value of the "sense of accomplishment" or "sense of growth." When the purpose is "consulting," the emotion that is emphasized may be a "sense of security," and when the purpose is "exercising" or "having a conversation," the emotion that is emphasized may be a "feeling of joy."

[0457] Furthermore, the purpose may be a learning purpose, in which case interactive, dialogue-based learning content can be created that makes use of a sentence generation model.

[0458] Furthermore, in this embodiment, the reaction of the user 10 in response to the avatar's actions may be fed back to the sentence generation model, thereby enabling optimal communication suited to the user 10. For example, if the user 10 is a child, feeding back the child's reaction to the sentence generation model allows learning to maximize the emotional value that is emphasized, resulting in optimal communication suited to the child.

[0459] In particular, when the behavior decision unit 236 determines that the avatar's behavior is to interact with the user 10, if the user 10 feels a positive emotion as a result of the avatar's behavior, it is preferable that the behavior decision unit 236 provides feedback to increase the emotion value indicating the strength of the emotion, and if the user 10 feels a negative emotion as a result of the avatar's behavior, it provides feedback to decrease the emotion value indicating the strength of the emotion.

[0460] In this form, when the behavior decision unit 236 decides that the avatar's behavior is to interact with the user 10, the avatar may be caused to operate in such a way that at least one of the content of the speech to the user 10, the tone of voice used when speaking, and the avatar's facial expression changes so that the user 10 feels positive emotions.

[0461] Here, the tone of voice includes not only the "way of speaking" that is, which words are selected, but also the emotions and accents conveyed in the spoken words.

[0462] Furthermore, the positive emotion may be at least one of joy, pleasure, comfort, relief, excitement, relief, and a sense of fulfillment, and the negative emotion may be at least one of anger, sadness, discomfort, anxiety, grief, worry, and a sense of emptiness.

[0463] By continuously repeating the above feedback loop, the content of the dialogue can be evolved in a way that makes the listener (user 10) feel positive emotions.

[0464] Furthermore, as in the first embodiment, the behavior determination unit 236 spontaneously infers the cultural sphere in which the user 10 resides, and reflects the inferred cultural sphere in answer generation by an AI-based sentence generation model as an example of the behavior determination model 221, in the determination of the user 10's emotion by the emotion determination unit 232, and in the determination of the avatar's emotion by the emotion determination unit 232. For example, as in the robot 100 according to the first embodiment, when it is predicted that the user 10 resides in the Kansai region or when it is detected that the user 10 speaks in the Kansai dialect, the behavior control unit 250 spontaneously switches the avatar's brain to a Kansai region brain. In this case, the avatar makes a gesture of retort in the Kansai dialect or speaks, for example, "Why?" under the control of the behavior control unit 250.

[0465] The behavior control unit 250 may display an avatar corresponding to the cultural sphere estimated by the behavior determination unit 236 in the image display area of ​​the headset-type device 820. For example, avatars corresponding to each cultural sphere may be stored in advance in the storage unit 220, and the behavior control unit 250 may acquire an avatar corresponding to the cultural sphere estimated by the behavior determination unit 236 from the storage unit 220 and display the acquired avatar in the image display area of ​​the headset-type device 820. In this case, for example, the avatar may be switched to an avatar of a character or person famous in the corresponding cultural sphere. The avatar may be an anthropomorphized version of a famous building, food, or other local specialty in the corresponding cultural sphere. Furthermore, for example, the behavior control unit 250 may input, as a prompt, an instruction statement to the image generation AI to generate an avatar of a person in the cultural sphere style (e.g., Kansai style) estimated by the behavior determination unit 236, such as "Please generate an avatar of a person in the Kansai style," and the avatar generated by the image generation AI may be displayed in the image display area of ​​the headset-type device 820. Furthermore, for example, the behavior control unit 250 may cause an avatar corresponding to a cultural sphere estimated by the behavior determination unit 236 to use actions or expressions that are commonly performed in that cultural sphere.

[0466] The behavior control unit 250 may also display a landscape image corresponding to the cultural sphere estimated by the behavior determination unit 236 in the image display area of ​​the headset device 820 as a background image for the avatar. For example, landscape images corresponding to each cultural sphere may be stored in advance in the storage unit 220, and the behavior control unit 250 may acquire a landscape image corresponding to the cultural sphere estimated by the behavior determination unit 236 from the storage unit 220 and display the acquired landscape image in the image display area of ​​the headset device 820 as a background image for the avatar. In this case, the background of the avatar is switched to the landscape image of the corresponding cultural sphere. For example, the behavior control unit 250 may input an instruction statement to the image generation AI as a prompt to generate a landscape image of the cultural sphere estimated by the behavior determination unit 236, such as "Please generate an image of a landscape that resembles Osaka," and display the landscape image generated by the image generation AI as a background image for the avatar in the image display area of ​​the headset device 820.

[0467] In particular, when the action decision unit 236 determines that the avatar's action is to give advice to a user, such as an athlete or a coach, participating in a specific competition regarding the specific competition they are participating in, the action decision unit 236 causes the avatar to operate based on information regarding the specific competition the user is participating in. In this case, it is preferable to have the action control unit 250 control the avatar so that the advice provided via the avatar allows the user to advance advantageously in the competition they are participating in.

[0468] Here, the avatar may be, for example, a 3D avatar selected by the user from pre-prepared avatars, an avatar representing the user, or a preferred avatar created by the user. When generating the avatar, image generation AI may be utilized to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting. The avatar's behavior determined by the behavior determination unit 236 may be specifically realized mainly by display by the behavior control unit 250.

[0469] A specific example of a method for the behavior control unit 250 to cause an avatar to perform a desired action is described below. First, the state, including the emotions, of multiple athletes participating in the competition in which the user is participating is detected. Detection of the emotions, etc. of multiple athletes can be achieved by the image acquisition unit of the behavior decision unit 236 described above. Detection of the emotions, etc. of athletes can be performed, for example, automatically or periodically by the behavior control unit 250. In this case, it is preferable to place the image acquisition unit in a location where the user, etc. is competing, i.e., where it can overlook the entire competition space. Taking this into consideration, the image acquisition unit can be configured, for example, as a camera with a communication function that can be installed in any location independent of the headset-type terminal 820.

[0470] When analyzing the emotions of multiple athletes in the images acquired by the image acquisition unit, the above-mentioned athlete analysis unit of the behavior decision unit 236 is used. The emotion value of each athlete analyzed by the athlete analysis unit can be reflected in the control of the avatar by the behavior control unit 250.

[0471] In the agent system 800 according to this embodiment, the behavior control unit 250 controls the avatar based at least on the emotion value analyzed by the athlete analysis unit. The specific manner in which the behavior control unit 250 controls the avatar is not particularly limited, as long as the control can provide predetermined advice to the user. While the control may primarily involve causing the avatar to speak, other actions, either alone or in combination with speech, can also be used to make the meaning easier for the user to understand. Therefore, several examples of the control of the avatar by the behavior control unit 250 are described below. In the following description, it is assumed that the agent system 800 is used to provide advice about the game to the coach of one of the teams participating in a volleyball game via a headset-type terminal 820 worn by the coach.

[0472] When the behavior decision unit 236 determines that the avatar's behavior is to give advice to the user (coach) regarding the volleyball match in which the user is participating, the behavior control unit 250 begins providing the advice through the avatar. For example, by making the avatar reflect the emotions of a specific player among multiple players, information regarding the specific player's condition can be provided to the user. To give a more specific example, when the player analysis unit identifies an emotionally unstable or irritated player among the opposing team, the behavior control unit 250 changes the appearance of the avatar to resemble the identified player and adjusts the avatar's facial expressions, etc., to match the emotional values ​​analyzed by the player analysis unit. This allows the user to visually understand the specific player's condition. In addition, by having the avatar speak the specific player's condition using the output of the behavior decision model 221, the user can more accurately understand the specific player's condition.

[0473] For example, if a specific player on the opposing team is emotionally unstable, an avatar resembling the specific player can be made to turn pale and close its eyes, thereby immediately informing the user that the specific player is emotionally unstable. In addition, if the avatar displays the avatar in this way and uses the output of the behavioral decision-making model 221 to make an utterance such as "The player with number 7 on the opposing team is upset," the coach as the user can plan a strategy that takes into account the situation of the player.

[0474] Furthermore, for example, if it is determined that a specific player on the opposing team is irritated, the avatar displayed to resemble the specific player can be made to blush and have its eyes raised, thereby instantly informing the user that the specific player is irritated. In addition, if, together with such an avatar display, the avatar uses the output of the behavioral decision-making model 221 to make an utterance such as "The player with number 5 on the opposing team is irritated," the coach as the user can devise a strategy that takes into account the situation of the player.

[0475] Furthermore, when the behavior determination unit 236 determines that the avatar's behavior is to give advice to the user (coach) regarding a volleyball match in which the user is participating, the behavior control unit 250 can cause the avatar to reflect information about the uniform worn during the specific match. Specifically, the behavior control unit 250 can cause the avatar to reflect information about the volleyball uniform for which advice is to be given, i.e., wear the uniform. The uniform worn by the avatar may be a general volleyball uniform prepared in advance, or it may be the uniform of the team to which the user belongs or the uniform of the opposing team. The information about the uniform of the team to which the user belongs or the uniform of the opposing team may be generated, for example, by analyzing an image acquired by the image acquisition unit, or may be registered in advance by the user.

[0476] As described above, reflecting uniform information in an avatar makes it easier for users to understand the information provided by the avatar. In the above example, it becomes easier to understand that the information provided by the avatar is related to the volleyball game in which the user is participating. In addition, as in the above example, when an avatar is displayed to resemble a specific athlete, by also making the uniform similar to that worn by the specific athlete, it becomes easier for users to understand which athlete the avatar resembles.

[0477] In the above example, an avatar is displayed to resemble a specific athlete, but the specific athlete is not limited to one. Similarly, the number of avatars displayed in the image display area of ​​the electronic device is not particularly limited. Therefore, the action determination unit 236 can, for example, reflect the emotions, uniforms, etc. of all players on the user's opposing team as a specific athlete and display them in multiple avatars.

[0478] In particular, the behavior decision unit 236 may detect the user's state spontaneously or periodically, and when it determines to propose at least one of two or more actions as the avatar's behavior based on the detected user's state and at least one of historical data about the user and information preferred by the user, it may cause the behavior control unit 250 to display the avatar so as to execute the action content. The action content will be specifically described below.

[0479] In the autonomous processing of this embodiment, the behavior determining unit 236 may detect the behavior or state of the user spontaneously or periodically. Specifically, the behavior determining unit 236 may monitor the user to track and analyze what information posted on what website the user is viewing, that is, track the information.

[0480] "Spontaneous" may be interpreted as the behavior determination unit 236 acquiring the user's state or behavior on its own initiative without an external trigger. An external trigger may include a question from the user to the avatar, an active action from the user to the avatar, etc. "Periodic" may be interpreted as a specific cycle, such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0481] User behavior may be interpreted as the following behavioral trends of users: (1) A user visiting one or more specific stores in a commercial facility such as a department store in order to purchase a specific product. Also, the user moving to multiple product display locations in a specific store. (2) A user browsing products on one or more specific EC (Electronic Commerce) sites using a smartphone or computer in order to purchase a specific product. (3) A user browsing information posted on one or more accommodation booking sites, travel sites, etc. in order to decide on a specific travel destination or accommodation. (4) A user browsing specific information posted on one or more financial information sites in order to purchase a specific financial product in order to purchase a specific financial product using a smartphone, computer, etc.

[0482] The user state may include the following user states: (1) A state in which a user continues to worry or think deeply about which product to purchase while looking at or repeatedly trying on products in a particular store. (2) A state in which a user continues to worry or think deeply about which product to purchase while browsing products on one or more e-commerce sites using a smartphone or computer. (3) A state in which a user continues to worry or think deeply about which accommodation, travel destination, etc. to choose while browsing information posted on one or more accommodation reservation sites, travel sites, etc. using a smartphone or computer. (4) A state in which a user continues to worry or think deeply about which financial product to invest in while browsing information posted on one or more financial information sites using a smartphone, computer, etc.

[0483] Additionally, in autonomous processing, the agent may ask the generative AI questions about the detected state or behavior of the user.

[0484] Furthermore, in the autonomous processing, the generative AI's response to the question and the behavioral content of the suggestion may be stored in association with each other. The behavioral content may be interpreted as the behavioral content of the avatar controlled by the behavior control unit 250, which suggests at least one thing from two or more things. Specifically, the behavioral content may be interpreted as the behavioral content of the avatar, which suggests a specific thing based on the generative AI's response to the detected state or behavior of the user.

[0485] The information associating the answers of the generative AI with the proposed actions may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0486] Furthermore, the autonomous processing may utilize the specific information, which is the stored table information, to execute an action content that suggests at least one thing from two or more things in response to the user's state or behavior. Specifically, the autonomous processing may detect the user's state autonomously or periodically, and based on the detected user's state or behavior and the specific information, suggest at least one thing from two or more things as an action of the avatar by the behavior control unit 250.

[0487] This specific information may be interpreted as information provided by the generative AI based on at least one of historical data about the user and information preferred by the user. In other words, the autonomous processing may involve the behavior control unit 250 suggesting at least one of two or more things as an avatar behavior based on the detected state or behavior of the user and at least one of historical data about the user and information preferred by the user.

[0488] Below, we will explain examples of actions to propose things.

[0489] For example, by monitoring the operations of a user using a smartphone, if the behavior decision unit 236 detects that the user is unable to decide whether to purchase clothes made by company A or clothes made by company B, the behavior decision unit 236 itself will ask the generative AI a question.

[0490] The generative AI answers at least one of two or more things based on at least one of historical data 222 about the user and information preferred by the user.

[0491] The history data 222 may include information obtained by tracking, such as the user's personality, preferences, habits, movements, thoughts, actions, conversation content, emotions, and the like.

[0492] The information preferred by the user may be interpreted as information included in the above-mentioned collected data 223. Specifically, the information preferred by the user may be interpreted as preference information that indicates matters of interest to the user 10 and is stored in the collected data 223. More specifically, the information preferred by the user may include information that is frequently searched for or selected by the user, such as trends, world affairs, etc.

[0493] The information preferred by the user is not limited to this and may include social information from multiple sources, which may include at least one of news, economic conditions, social conditions, political conditions, financial conditions, international conditions, sports news, entertainment news, birth and death announcements, cultural conditions, and trends.

[0494] For example, in response to a question from the behavioral decision unit 236 such as "What products should we suggest to a user who cannot decide which clothes to purchase?", the generative AI may answer, based on at least one of the historical data 222 about the user and information about the user's preferences, "Company A's products will be increased in price from April, so we recommend purchasing Company A's products before the price increase."

[0495] The generative AI may also respond, "Company B's products will be reduced in price from April, so I recommend purchasing Company B's products after the price reduction."

[0496] The generative AI may also respond, "In light of the user's recent purchasing trends, I recommend purchasing Company C's product, which is more expensive than Company A and Company B's products but is similar to Company A and Company B's products."

[0497] Upon receiving the response, the behavior decision unit 236 may, if it determines to propose at least one of two or more things based on the detected user state or behavior and the recorded information, cause the behavior control unit 250 to operate the avatar so as to execute the behavior content. Specifically, the behavior control unit 250 may refer to the recorded information and cause the avatar to play a sound corresponding to the content of a product appropriate for the detected user state or behavior. In this case, the behavior decision unit 236 may cause the behavior control unit 250 to display a message (a speech bubble text) corresponding to the content of the product around the avatar's mouth.

[0498] The behavior control unit 250 may refer to the recorded information and display on the screen an image corresponding to the content of a product appropriate for the detected state or behavior of the user. In this case, the behavior control unit 250 may control the avatar to transform from a humanoid avatar into the shape of the product. The behavior control unit 250 may change the avatar's appearance back to a humanoid form after a specific time has elapsed since the avatar transformed into the shape of the product, or may change the shape of the product to attract the user's interest. The behavior control unit 250 may cause the avatar to perform gestures and hand movements to introduce the image of the product, so that the humanoid avatar introduces the image of the product.

[0499] In addition, instead of monitoring the operations of users using smartphones, the behavior control unit 250 may monitor users moving between multiple product display areas within a specific store using image data captured by an imaging device.

[0500] As described above, the behavior control system of the present disclosure can use at least one of the user's history data and the user's preference information to select at least one of two or more things and determine the behavior to be suggested to the user. Therefore, for a user who has difficulty choosing an action, the behavior control unit 250 can recommend and suggest an action that is suitable for the user by having the avatar speak spontaneously.

[0501] In particular, when the behavior decision unit 236 determines that the avatar's behavior is "(16) The robot gives the user advice about household matters," i.e., to give advice to the user, it is preferable to have the behavior control unit 250 control the avatar to use a sentence generation model to suggest advice about physical condition, recommended dishes, ingredients that should be replenished, etc., based on data about household appliances stored in the history data.

[0502] For example, the behavior control unit 250 may control the avatar so that the avatar suggests advice on physical condition, recommended dishes, ingredients to be replenished, etc. based on the user's condition as understood from the dialogue history stored in the history data 222, the user's reaction to conversations with the avatar, information collected from household devices, etc. In this case, the behavior control unit 250 may control the avatar so that it transforms into a shape that resembles the recommended dish.

[0503] The behavior control unit 250 may also control the avatar so that the avatar voluntarily orders ingredients to replenish the refrigerator based on data related to ingredients in the refrigerator and data related to the stock of consumables stored in the home. In this case, the behavior control unit 250 may control the avatar so that the type of product or ingredients to be ordered changes based on the user's state, the user's emotions, and the avatar's emotions.

[0504] In this embodiment, the control unit 228B has the function of determining the behavior of the avatar and generating the display of the avatar to be presented to the user through the headset terminal 820.

[0505] As in the first embodiment, the emotion determining unit 232 of the control unit 228B determines the emotion value of the agent based on the state of the headset type terminal 820, and uses this as the emotion value of the avatar.

[0506] As in the first embodiment described above, when an agent functioning as an avatar performs autonomous processing to act autonomously, the behavior decision unit 236 of the control unit 228B determines, at a predetermined timing, one of multiple types of avatar behaviors, including no action, as the avatar's behavior, using at least one of the state of the user 10, the emotions of the user 10, the emotions of the avatar, and the state of the electronic device controlling the avatar (e.g., the headset-type terminal 820), and the behavior decision model 221.

[0507] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the state of the headset-type terminal 820, the emotion of the user 10, and the emotion of the avatar, as well as text asking about the avatar's behavior, into a sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model.

[0508] For example, the multiple types of avatar actions include the following (1) to (12).

[0509] (1) The avatar does nothing. (2) The avatar dreams. (3) The avatar talks to the user. (4) The avatar creates a picture diary. (5) The avatar suggests activities. (6) The avatar suggests people the user should meet. (7) The avatar introduces news that the user finds interesting. (8) The avatar edits photos and videos. (9) The avatar studies with the user. (10) The avatar evokes memories. (11) The avatar's actions are predetermined. (12) The avatar encourages interaction with others.

[0510] The behavior determination unit 236 inputs the state of the user 10 and the state of the headset type terminal 820 recognized by the state recognition unit 230, text representing the current emotional value of the user 10 determined by the emotion determination unit 232, the current emotional value of the avatar, and text asking which of multiple types of avatar behaviors, including not taking any action, into the sentence generation model at regular time intervals, and determines the behavior of the avatar based on the output of the sentence generation model. Here, if the headset type terminal 820 is not worn by the user 10, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0511] As an example, the following text is input into the sentence generation model: "The avatar is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (10) is the best behavior for the avatar? (1) The avatar does nothing. (2) The avatar dreams. (3) The avatar talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the avatar dreams is the most appropriate behavior," the avatar's behavior is determined to be either "(1) doing nothing" or "(2) the avatar dreams."

[0512] As another example, the following text is input into a sentence generation model: "The avatar is feeling a little lonely. The user is absent. The area around the headset device is dark. Which of the following (1) to (10) is the best behavior for the avatar? (1) The avatar does nothing. (2) The avatar dreams. (3) The avatar talks to the user. ..." Based on the output of the sentence generation model, "Either (2) The avatar dreams or (4) The avatar creates a picture diary is the most appropriate behavior," the avatar's behavior is determined to be either "(2) The avatar dreams" or "(4) The avatar creates a picture diary."

[0513] When the behavior decision unit 236 decides to create an original event, i.e., "(2) The avatar dreams," as the avatar behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0514] When the behavior determining unit 236 determines that the avatar behavior is "(3) The avatar speaks to the user," i.e., that the avatar will speak, it uses a sentence generation model to determine the avatar's utterance content corresponding to the user's state and the user's emotion or the avatar's emotion. At this time, the behavior control unit 250 causes a sound representing the determined avatar's utterance content to be output from a speaker included in the control object 252. Note that, when the headset-type terminal 820 is not worn by the user 10, the behavior control unit 250 stores the determined avatar's utterance content in the behavior schedule data 224 without outputting the sound representing the determined avatar's utterance content.

[0515] When the behavior determining unit 236 determines that the avatar behavior is "(7) The avatar introduces news that is of interest to the user," it uses the sentence generation model to determine the avatar's utterance content corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined avatar's utterance content to be output from a speaker included in the control target 252. Note that when the headset-type terminal 820 is not worn by the user 10, the behavior control unit 250 stores the determined avatar's utterance content in the behavior schedule data 224 without outputting the sound representing the determined avatar's utterance content.

[0516] When the behavior determination unit 236 determines that the avatar will create an event image, i.e., "(4) The avatar creates a picture diary," as the avatar behavior, the behavior determination unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the headset terminal 820 is not worn by the user 10, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0517] When the behavior decision unit 236 determines that the avatar behavior is "(8) The avatar edits photos or videos," i.e., that the avatar edits images, the behavior decision unit 236 selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs the edited image data. Note that when the headset terminal 820 is not worn by the user 10, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0518] When the behavior determination unit 236 determines that the avatar behavior is "(5) The avatar proposes an activity," i.e., when it determines to propose an action for the user 10, it determines the user's behavior to propose using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's behavior to be output from a speaker included in the control object 252C. Note that when the headset-type terminal 820 is not worn by the user 10, the behavior control unit 250 does not output a sound proposing the user's behavior, but instead stores the suggestion of the user's behavior in the behavior schedule data 224.

[0519] When the behavior determination unit 236 determines that the avatar behavior is "(6) The avatar suggests people the user should meet," i.e., to suggest people with whom the user 10 should have contact, the behavior determination unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people with whom the suggested user should have contact. At this time, the behavior control unit 250 causes a speaker included in the control object 252C to output a sound indicating that a person with whom the user should have contact is being suggested. Note that, when the headset-type terminal 820 is not worn by the user 10, the behavior control unit 250 does not output a sound indicating that a person with whom the user should have contact is being suggested, but instead stores the suggestion of a person with whom the user should have contact in the behavior schedule data 224.

[0520] When the behavior determining unit 236 determines that the avatar behavior is "(9) The avatar studies with the user," i.e., that the avatar will speak about studying, it uses a sentence generation model to determine the avatar's utterance content to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the avatar's emotions. At this time, the behavior control unit 250 outputs audio representing the determined avatar's utterance content from a speaker included in the control object 252C. Note that when the headset-type terminal 820 is not worn by the user 10, the behavior control unit 250 does not output audio representing the determined avatar's utterance content, and stores the determined avatar's utterance content in the behavior schedule data 224.

[0521] When the behavior determining unit 236 determines that the avatar behavior is "(10) The avatar recalls a memory," i.e., that the avatar recalls event data, it selects the event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the avatar based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of the utterances and actions of the avatar 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0522] For example, if the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "What are some things you should say to the user the next time you meet them on the topic of pandas? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the action decision unit 236 inputs to the sentence generation model "Which of (1), (2), and (3) would make the user happiest?" If the output of the sentence generation model is "(1) Let's go to the zoo," an emotion change event will be created in which the avatar will say "(1) Let's go to the zoo" the next time the headset-type terminal 820 is worn by the user 10, and this will be stored in the action schedule data 224.

[0523] Furthermore, for example, event data with a high emotion value for the avatar is selected as an impressive memory for the avatar, thereby enabling an emotion change event to be created based on the event data selected as an impressive memory.

[0524] When the behavior determination unit 236 detects an action of the user 10 toward the avatar from a state in which the user 10 has not taken any action toward the avatar based on the state of the user 10 recognized by the state recognition unit 230, the behavior determination unit 236 reads the data stored in the action schedule data 224 and determines the behavior of the avatar.

[0525] For example, if the headset type terminal 820 is not worn by the user 10 and it is detected that the headset type terminal 820 is worn by the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the avatar. Also, if the user 10 is sleeping and it is detected that the headset type terminal 820 is worn by the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the avatar.

[0526] Furthermore, the behavior control unit 250 displays the avatar in the image display area of ​​the headset type terminal 820 as the control target 252C according to the determined behavior of the avatar. Furthermore, if the determined behavior of the avatar includes the content of the avatar's utterance, the content of the avatar's utterance is output as audio from the speaker as the control target 252C.

[0527] In particular, when the behavior decision unit 236 determines that the avatar's behavior is to do nothing, it is preferable to have the behavior control unit 250 control the avatar to make a specific facial expression (a bored facial expression) or a specific gesture (a gesture indicating boredom).

[0528] In particular, when the behavior determination unit 236 determines that the avatar's behavior is to respond to the user 10's consultation about a problem, the agent system 800 executes the following processing. The behavior determination unit 236 uses a sentence generation model to generate a question corresponding to the user 10's problem, and the behavior control unit 250 operates the avatar to make an utterance corresponding to the question. Specifically, the behavior determination unit 236 inputs attribute information of the user 10 and text representing the content of the problem into the sentence generation model, and generates a question corresponding to the user 10's problem based on the output from the sentence generation model. The attribute information includes, for example, the user 10's age, gender, occupation, family composition, medical history, lifestyle, etc. The behavior control unit 250 controls the behavior of the avatar to make an utterance corresponding to the generated question, thereby developing a conversation with the user 10.

[0529] The behavior decision unit 236 analyzes the content of the answers given by the user 10 to the questions, as well as the facial expressions, emotions, and movements of the user 10, to determine whether the mental state of the user 10 is good or bad. In determining whether the mental state is good or bad, a good or bad level is determined, for example, categorized into levels such as "healthy," "preliminary illness," "early illness," "ill," and "requiring treatment."

[0530] When the behavior determining unit 236 determines that the mental state level of the user 10 is anything other than "healthy," the behavior determining unit 236 causes the behavior control unit 250 to operate the avatar so as to suggest the cause of the illness and a remedy. Furthermore, when the behavior determining unit 236 determines that the mental state level of the user 10 is "treatment required," the behavior control unit 250 causes the avatar to operate so as to support improvement of the mental health of the user 10 in cooperation with relevant institutions. Furthermore, the behavior control unit 250 may control the avatar to change its facial expression, tone of voice, and manner of speaking depending on the mental state level of the user 10.

[0531] To support the improvement of the mental health of the user 10, the behavior decision unit 236 inputs the content of the answer from the user 10 to the question and the emotional value of the user 10 into the sentence generation model, obtains a solution or advice for the user 10's problem output from the sentence generation model, and causes the behavior control unit 250 to operate the avatar so as to provide the obtained solution or advice to the user 10.

[0532] After having the avatar provide the above support, the behavior decision unit 236 causes the behavior control unit 250 to operate the avatar to hear from the user 10 about the state of improvement in their mental health, and determines whether the support provided by the avatar is appropriate based on the results of the hearing and the emotional value of the user 10 at that time. The behavior decision unit 236 trains the sentence generation model by feeding back the answers from the user 10 at each conversation step and the emotions of the user 10 to the sentence generation model, thereby realizing a conversation that maximizes the rate of resolving the problem. In the case of a person who has been consulted in the past, the behavior decision unit 236 conducts a conversation taking into account the history and also takes into account changes in the situation of the person being consulted.

[0533] The behavior control unit 250 operates the avatar with an appearance that corresponds to the user's concern. For example, if the user's concern is health-related, the avatar's appearance is set to that of a doctor, and if the user's concern is study-related, the avatar's appearance is set to that of a school teacher.

[0534] When the behavior decision unit 236 decides that the avatar's behavior is to respond to the user's advice, the agent system 800 executes the following steps 1 to 5.

[0535] (Step S1) The behavior control unit 250 operates the avatar to acquire attribute information and worries of the user 10 through conversation with the user 10.

[0536] (Step S2) The behavior determination unit 236 inputs text representing the attribute information and content of the user 10, plus a fixed sentence such as "What question would be effective in clarifying the root cause of the user's concern at this time?" into the sentence generation model, and obtains question text output from the sentence generation model. The behavior control unit 250 operates the avatar to make an utterance according to the obtained question text.

[0537] (Step S3) The behavior determining unit 236 analyzes the content of the answers from the user 10 to the questions asked in step S2, as well as the facial expressions, emotions, and movements of the user 10, to determine whether the mental state of the user 10 is good or bad. In analyzing the emotions of the user 10, the emotion determining unit 232 determines an emotion value of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognizing unit 230. In determining whether the mental state is good or bad, the behavior determining unit 236 determines a good or bad level categorized into levels such as "healthy," "preliminary illness," "early illness," "ill," and "requiring treatment."

[0538] (Step S4) If the behavior determination unit 236 determines that the mental state level of the user 10 is anything other than "healthy," it uses the sentence generation model to obtain a solution or advice for the user's problem, and causes the behavior control unit 250 to operate the avatar so as to make an utterance according to the obtained solution or advice. Specifically, the behavior determination unit 236 inputs text into the sentence generation model, which is obtained by adding a fixed sentence, "What is the solution or advice for the user's problem at this time?" to text indicating the content of the user 10's answer to the question and the emotional value of the user 10, and obtains the solution or advice for the user 10's problem, which is output from the sentence generation model. The behavior control unit 250 causes the avatar to make an utterance according to the obtained solution or advice.

[0539] (Step S5) The behavior control unit 250 operates the avatar to ask the user 10 about the improvement status of their mental health. The behavior determination unit 236 determines whether the support provided by the avatar in step S4 was appropriate based on the results of the hearing and the emotional value of the user 10 at that time. Specifically, the emotional value determination unit 232 determines the emotional value of the user 10 based on the information analyzed by the sensor module unit 210B and the state of the user 10 recognized by the user state recognition unit 230. The behavior determination unit 236 derives the probability that the support provided in step S4 was effective based on the emotional value of the user 10 and the results of ...

Claims

1. A state recognition unit that recognizes a user state including a user's behavior, and a state of an avatar representing an agent for interacting with the user; an emotion determination unit that determines an emotion of the user or an emotion of the avatar; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and a behavior determination model; a storage control unit that stores in history data the emotion value determined by the emotion determination unit, event data including data including the user's behavior, and a photo or video captured when the emotion value reaches a predetermined criterion; and a behavior control unit that displays the avatar in an image display area of ​​an electronic device, wherein the behavior of the avatar includes creating a picture diary, A behavior control system in which, when the behavior decision unit decides to create the picture diary as the avatar's behavior, it selects the photo or the video from the history data, generates an explanatory text for the photo or video clip based on the emotional value at the time the selected photo or video was obtained, and outputs a combination of the photo or video clip and the explanatory text as the picture diary.

2. The behavior control system of claim 1, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

3. The behavior control system of claim 1, wherein when the behavior decision unit determines that the avatar's behavior is to create the pictorial diary, it selects the photograph or the video from the history data, generates an explanatory text for the photograph or video clip based on the emotional value at the time the selected photograph or video was acquired, and operates the avatar to output a combination of the photograph or video clip and the explanatory text as the pictorial diary.

4. The behavior control system according to claim 1, wherein the electronic device is a headset type terminal.

5. The behavior control system according to claim 1, wherein the electronic device is a glasses-type terminal.

6. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an avatar representing an agent for interacting with a user; an emotion determination unit that determines the user's emotion or the avatar's emotion; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the avatar's behavior using at least one of the user state, the avatar's state, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​an electronic device, wherein the avatar behavior includes advising the user regarding fraud risks, and when the behavior determination unit determines that the avatar's behavior is to advise the user regarding fraud risks, the behavior control unit advises the user regarding fraud risks.

7. A behavior control system as described in claim 6, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

8. A behavior control system as described in claim 6, wherein when the behavior decision unit determines that the behavior of the avatar is to advise the user regarding a fraud risk, the behavior decision unit causes the avatar to transform into another avatar.

9. The behavior control system according to claim 6, wherein the electronic device is a headset type terminal.

10. The behavior control system according to claim 6, wherein the electronic device is a glasses-type terminal.

11. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes the electronic device making an utterance or a gesture to the user, and the behavior determination unit autonomously detects the user's state, and when the emotion determination unit determines at least one of the user's emotion and the avatar's emotion based on the detected user's state, determines the content of the utterance or the gesture corresponding to the determined at least one of the user's emotion and the avatar's emotion, and causes the behavior control unit to control the avatar.

12. The behavior control system of claim 11, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

13. The behavior control system according to claim 11, wherein the behavior determination unit further determines an avatar in accordance with the emotion determined by the emotion determination unit so as to change the avatar into another avatar that has been preset in accordance with the emotion determined by the emotion determination unit.

14. The behavior control system of claim 11, wherein the behavior determination unit further determines the movement speed of the avatar in accordance with the emotion determined by the emotion determination unit, in order to change the movement speed of the avatar to a predetermined movement speed in accordance with the emotion determined by the emotion determination unit.

15. The behavior control system according to claim 11, wherein the electronic device is a headset type terminal.

16. The behavior control system according to claim 11, wherein the electronic device is a glasses-type terminal.

17. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes interacting with the user, and when the behavior determination unit determines the behavior of the avatar to be interactive with the user, it determines the behavior of the avatar so as to maximize an emotion value indicating the strength of an emotion that is considered important for the user depending on the purpose of the interaction.

18. The behavior control system described in claim 17, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

19. The behavior control system of claim 17, wherein when the behavior decision unit decides that the behavior of the avatar is to converse with the user, it operates the avatar so as to change at least one of the content of the speech to the user, the tone of voice used when speaking, and the avatar's facial expression, so as to maximize the emotional value.

20. The behavior control system according to claim 17, wherein the electronic device is a headset type terminal.

21. The behavior control system according to claim 17, wherein the electronic device is a glasses-type terminal.

22. The behavior control system of claim 17, wherein the important emotion when the purpose is to learn is a sense of accomplishment or a sense of growth, the important emotion when the purpose is to consult is a sense of security, and the important emotion when the purpose is to exercise or converse is a joyful emotion.

23. The behavior control system according to claim 17, wherein the objective is a learning objective.

24. The behavior control system according to claim 18, wherein a reaction of the user in response to a behavior of the avatar is fed back to the data generation model.

25. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes interacting with the user, and when the behavior determination unit has determined that the behavior of the avatar is to interact with the user, it performs feedback to increase an emotion value indicating the strength of the emotion if the user has a positive emotion in response to the avatar's behavior, and performs feedback to decrease the emotion value indicating the strength of the emotion if the user has a negative emotion in response to the avatar's behavior.

26. The behavior control system described in claim 25, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

27. The behavior control system of claim 25, wherein when the behavior decision unit decides that the behavior of the avatar is to converse with the user, it operates the avatar so that at least one of the content of the speech to the user, the tone of voice used when speaking, and the avatar's facial expression is changed so that the user has positive feelings.

28. The behavior control system according to claim 25, wherein the electronic device is a headset type terminal.

29. The behavior control system according to claim 25, wherein the electronic device is a glasses-type terminal.

30. The behavior control system of claim 25, wherein the positive emotion is at least one of joy, pleasure, comfort, relief, excitement, relief, and a sense of fulfillment, and the negative emotion is at least one of anger, sadness, discomfort, anxiety, sorrow, worry, and a sense of emptiness.

31. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that causes the avatar to be displayed in an image display area of ​​the electronic device, wherein the avatar behavior includes the avatar performing an action that expresses an emotion, and the behavior determination unit autonomously collects objects of interest to the user, and when it is determined to provide information according to the user's interest as the avatar behavior, determines the content of the action that expresses the avatar's emotion according to the content of the information provided.

32. A behavior control system as described in claim 31, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

33. A behavior control system as described in claim 31, wherein when the behavior decision unit decides to provide information corresponding to the user's interests as the avatar's behavior, it changes the display mode of the avatar according to the content of the information.

34. The behavior control system according to claim 31, wherein said electronic device is a headset type terminal.

35. The behavior control system according to claim 31, wherein the electronic device is a glasses-type terminal.

36. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit reflects the inferred cultural sphere of the user in at least one of output generation by the behavior decision model, the determination of the user's emotion by the emotion determination unit, and the determination of the avatar's emotion by the emotion determination unit.

37. A behavior control system as described in claim 36, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

38. A behavior control system according to claim 36, wherein said behavior control section causes said avatar corresponding to the cultural sphere of said user to be displayed in said image display area.

39. A behavior control system according to claim 36, wherein said behavior control unit causes a landscape image corresponding to the cultural sphere of said user to be displayed in said image display area as a background image for said avatar.

40. The behavior control system according to claim 36, wherein said electronic device is a headset type terminal.

41. The behavior control system according to claim 36, wherein the electronic device is a glasses-type terminal.

42. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice regarding a specific competition to the user participating in the specific competition, the behavior determination unit comprising: an image acquisition unit that is capable of capturing an image of a competition space in which the specific competition in which the user participates is being held; and an athlete analysis unit that analyzes the emotions of a plurality of athletes participating in the specific competition in the competition space captured by the image acquisition unit, and when it is determined that the behavior of the avatar is to give advice regarding the specific competition to the user participating in the specific competition, the advice is given to the user based on the analysis result of the athlete analysis unit.

43. A behavior control system as described in claim 42, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

44. A behavior control system as described in claim 42, wherein when the behavior decision unit determines that the behavior of the avatar is to provide advice regarding the specific competition to the user participating in the specific competition, the behavior decision unit executes an operation on the avatar that reflects the emotions of a specific athlete among the multiple athletes analyzed by the athlete analysis unit.

45. A behavior control system as described in claim 42, wherein when the behavior decision unit determines that the behavior of the avatar is to provide advice regarding the specific sport to the user participating in the specific sport, it executes an operation to cause the avatar to reflect information about the uniform to be worn during the specific sport.

46. ​​The behavior control system according to claim 42, wherein said electronic device is a headset type terminal.

47. A behavior control system according to claim 42, wherein said electronic device is a glasses-type terminal.

48. A system including: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar, using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes selecting at least one of two or more things and setting the action content to be proposed to the user, The behavior decision unit detects the user's state either autonomously or periodically, and when it decides to suggest at least one thing out of two or more things as the avatar's behavior based on the detected user's state and at least one of historical data about the user and information preferred by the user, it causes the behavior control unit to display the avatar in the image display area so as to execute the behavior content.

49. The behavior control system of claim 48, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the avatar state, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

50. The behavior control system of claim 48, wherein the information preferred by the user includes at least one of trends and world affairs that the user can search for or select, and when the behavior decision unit decides to suggest at least one product from among products that the user can purchase based on the information preferred by the user including at least one of trends and world affairs that the user can search for or select and historical data regarding the user, it causes the behavior control unit to display the avatar in the image display area to explain the product.

51. The behavior control system according to claim 48, wherein the electronic device is a headset type terminal.

52. The behavior control system according to claim 48, wherein the electronic device is a glasses-type terminal.

53. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice regarding household matters to the user, and when the behavior determination unit determines that the behavior of the avatar is to give advice regarding household matters to the user, it uses a sentence generation model to suggest advice regarding physical condition, recommended dishes, ingredients that should be replenished, etc., based on the data regarding household appliances stored in the history data.

54. A behavior control system as described in claim 53, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

55. A behavior control system as described in claim 53, wherein when the behavior decision unit decides to automatically order replenishment ingredients as the avatar's behavior, it causes the avatar to operate so as to determine the ingredients to order based on at least one of information regarding refrigerator usage status and information regarding changes in the user's weight.

56. A behavior control system according to claim 53, wherein said electronic device is a headset type terminal.

57. A behavior control system according to claim 53, wherein said electronic device is a glasses-type terminal.

58. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device.

59. A behavior control system as described in claim 58, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

60. A behavior control system as described in claim 58, further comprising a related information collection unit that collects information related to the preference information from external data based on the preference information acquired about the user at a predetermined timing, and the emotion determination unit determines the emotion of the avatar based on the collected information related to the preference information.

61. A behavior control system according to claim 58, wherein said behavior control unit, when deciding not to perform an action as the action of said avatar, causes the avatar to make a specific facial expression or a specific gesture.

62. A behavior control system according to claim 58, wherein said electronic device is a headset type terminal.

63. A behavior control system according to claim 58, wherein said electronic device is a glasses-type terminal.

64. A behavior control system comprising: a state recognition unit which recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit which determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit which determines the behavior of the avatar corresponding to the user state and the user's emotion or the avatar's emotion based on a behavior determination model; and a behavior control unit which controls the operation of the avatar displayed in an image display area of ​​the electronic device, wherein the behavior determination unit uses a sentence generation model to generate a question corresponding to the user's concern, and determines the avatar's behavior to be speaking in response to the question.

65. A behavior control system as described in claim 64, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data asking about the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

66. A behavior control system according to claim 64, wherein said behavior decision unit analyzes the content of the user's answers to said questions, the user's facial expressions, emotions and movements, and judges whether the user's mental state is good or bad.

67. The behavior control system described in claim 66, wherein the behavior decision unit uses the sentence generation model to obtain a solution or advice for the user's problem depending on the result of judging whether the user's mental state is good or bad, and determines the avatar's behavior to be an utterance corresponding to the obtained solution or advice.

68. A behavior control system according to claim 64, wherein said behavior control unit causes said avatar to move in an appearance corresponding to a worry of said user.

69. A behavior control system according to claim 64, wherein said electronic device is a headset type terminal.

70. A behavior control system according to claim 64, wherein said electronic device is a glasses-type terminal.

71. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes determining an action schedule for the avatar, and when the behavior determination unit has determined to determine the action schedule for the avatar as the behavior of the avatar, it determines a combination of an activation condition for initiating the action schedule and the content of the action schedule for the avatar, stores the combination in action schedule data, and when the activation condition of the action schedule data is satisfied, it determines to execute the content of the action schedule for the avatar.

72. A behavior control system as described in claim 71, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

73. A behavior control system according to claim 71, wherein said behavior control unit, when the activation condition of said behavior schedule data is satisfied, operates the avatar with an appearance corresponding to the time period during which said behavior schedule is activated.

74. A behavior control system according to claim 71, wherein said electronic device is a headset type terminal.

75. A behavior control system according to claim 71, wherein said electronic device is a glasses-type terminal.

76. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes encouraging interaction with others, and when the behavior determination unit determines that the behavior of the avatar is to encourage interaction with others, it determines at least one of an interaction partner or an interaction method based on the event data.

77. A behavior control system as described in claim 76, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

78. A behavior control system as described in claim 76, wherein when the behavior decision unit decides that the behavior of the avatar is to encourage interaction with others, it uses the event data to determine the content of the avatar's speech corresponding to the interaction partner and the interaction method.

79. The behavior control system according to claim 76, wherein said electronic device is a headset type terminal.

80. The behavior control system according to claim 76, wherein the electronic device is a glasses-type terminal.

81. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice regarding reading aloud, and when the behavior determination unit determines to give advice regarding reading aloud as the behavior of the avatar, it generates advice regarding reading aloud from collected information regarding reading aloud in accordance with predetermined suggestion conditions, and controls the avatar to provide the advice to the user.

82. A behavior control system as described in claim 81, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

83. The behavior control system of claim 81, wherein when giving advice regarding the reading aloud as the avatar behavior, the user is a first user reading aloud and the user is being read to is a second user, and the behavior control unit controls the avatar to give the advice at a predetermined frequency determined for the first user and the second user using at least one of an appearance and a voice manner corresponding to each of the first user and the second user.

84. The behavior control system of claim 83, wherein when the behavior control unit controls the avatar to provide advice regarding the reading aloud as the avatar behavior, the behavior control unit uses a user's emotional value as the collected information regarding the reading aloud and forms a facial expression of the avatar according to the emotional value.

85. A behavior control system according to claim 81, wherein said electronic device is a headset type terminal.

86. A behavior control system according to claim 81, wherein said electronic device is a glasses-type terminal.

87. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's surrounding environment, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device.

88. A behavior control system as described in claim 87, wherein the avatar behavior includes determining an action schedule for the avatar, and when the behavior decision unit determines to determine an action schedule for the avatar as the behavior of the avatar, it determines a combination of an activation condition for initiating the action schedule and the content of the action schedule for the avatar, stores the combination in behavior schedule data, and determines to execute the content of the action schedule for the avatar when the activation condition of the behavior schedule data is satisfied.

89. A behavior control system as described in claim 88, wherein said behavior control unit, when the activation condition of said behavior schedule data is satisfied, operates the avatar with an appearance that corresponds to the time period when said behavior schedule is activated or to the user's surrounding environment.

90. A behavior control system according to claim 87, wherein said electronic device is a headset type terminal.

91. A behavior control system according to claim 88, wherein said electronic device is a glasses-type terminal.

92. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes asking a question based on the user's past emotions, and when the behavior determination unit determines that the behavior of the avatar is to ask a question based on the user's past emotions, it speaks to the user.

93. The behavior control system of claim 92, wherein the memory control unit, if the emotion value when detecting the user's behavior exceeds a predetermined value, stores the user's behavior as an important behavior, and the behavior decision unit, if detecting the important behavior and the emotion value exceeding the predetermined value, decides that the avatar's behavior is to ask a question based on the user's past emotions.

94. A behavior control system as described in claim 92, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

95. A behavior control system as described in claim 92, wherein when the behavior decision unit determines that the avatar's behavior is to ask a question based on the user's past feelings, it causes the avatar to speak to the user.

96. The behavior control system according to claim 92, wherein said electronic device is a headset type terminal.

97. A behavior control system according to claim 92, wherein said electronic device is a glasses-type terminal.

98. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes talking about a subject of interest of the user, and when the behavior determination unit has determined that the behavior of the avatar is to talk about the subject of interest of the user, it determines speech content for the event data whose emotion value satisfies a predetermined criterion.

99. A behavior control system as described in claim 98, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

100. A behavior control system as described in claim 98, wherein when the behavior decision unit decides that the behavior of the avatar is to talk about a topic of interest to the user, it causes the avatar to speak the decided speech content.

101. The behavior control system according to claim 98, wherein said electronic device is a headset type terminal.

102. A behavior control system according to claim 98, wherein said electronic device is a glasses-type terminal.

103. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes notifying the provider of information based on the user's emotion regarding an item provided by the provider, and when the behavior determination unit determines to notify the provider of the information based on the user's emotion regarding the item provided by the provider as the behavior of the avatar, the behavior control unit operates the avatar to notify the provider of the information based on the user's emotion regarding the item provided by the provider.

104. A behavior control system as described in claim 103, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

105. A behavior control system according to claim 103, wherein said electronic device is a headset type terminal.

106. A behavior control system according to claim 103, wherein said electronic device is a glasses-type terminal.

107. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice regarding pregnant women, and when the behavior determination unit determines that the behavior of the avatar is to give advice regarding pregnant women, it collects information regarding at least one of pregnancy and postpartum, and gives advice regarding pregnant women based on the collected information.

108. A behavior control system as described in claim 107, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

109. A behavior control system as set forth in claim 107, wherein said behavior decision section, when deciding that the behavior of said avatar is to give advice regarding pregnant women, operates the avatar so as to transform into another avatar.

110. A behavior control system according to claim 107, wherein said electronic device is a headset type terminal.

111. A behavior control system according to claim 107, wherein the electronic device is a glasses-type terminal.

112. A state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that determines, at a predetermined timing, one of a plurality of types of avatar behaviors including no behavior as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a storage control unit that stores, in history data, event data including the emotion value determined by the emotion determination unit and data including the user's behavior, characteristic information including the characteristics of the user, and situation information at the time when the characteristic information was acquired; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes speaking to the user, and when the behavior determination unit determines that the avatar's behavior is to speak to the user, it infers the content of the user's dialogue with the avatar based on the history data and the situation information at that time, and determines the content of the speech to the user based on the result of the inference. Behavioral control system.

113. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores in history data event data including the emotion value determined by the emotion determination unit and data including the user's behavior, characteristic information including the characteristics of the user, and situation information at the time when the characteristic information was acquired; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes playing specific music data, and when the behavior determination unit determines to play the specific music data as the behavior of the avatar, it determines the specific music data to play based on the history data and the situation information at that time.

114. A behavior control system as described in claim 112 or claim 113, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

115. A behavior control system according to claim 112 or 113, wherein said electronic device executes the behavior of said avatar determined by said behavior determination unit with respect to said user before said user speaks to said avatar.

116. A behavior control system according to claim 112 or 113, wherein the electronic device is a headset type terminal.

117. A behavior control system according to claim 112 or 113, wherein the electronic device is a glasses-type terminal.

118. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes analyzing a user's personality, and when the behavior determination unit determines to analyze the user's personality as the behavior of the avatar, it analyzes the user's personality using the history data including a conversation history with the user and presents the analyzed personality.

119. A behavior control system as described in claim 118, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

120. A behavior control system as claimed in claim 118, wherein said behavior control section changes the facial expression of said avatar when presenting the personality of said user, according to the emotion of said user determined by said emotion determination section.

121. The behavior control system according to claim 118, wherein said electronic device is a headset type terminal.

122. The behavior control system according to claim 118, wherein said electronic device is a glasses-type terminal.

123. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving advice to the user regarding a labor issue, and when the behavior determination unit has determined that the behavior of the avatar is to give advice to the user regarding a labor issue, it determines to give the advice to the user regarding a labor issue based on the user's behavior.

124. A behavior control system as described in claim 123, further comprising a memory control unit that stores event data including the emotion value determined by said emotion determination unit and data including the user's behavior in history data, wherein when said behavior determination unit determines that the avatar's behavior is to provide the user with advice regarding a labor issue, it determines to provide the user with advice regarding a labor issue based on the user's behavior recorded in the history data.

125. A behavior control system as described in claim 123, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

126. The behavior control system according to claim 123, wherein said electronic device is a headset type terminal.

127. The behavior control system according to claim 123, wherein said electronic device is a glasses-type terminal.

128. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the behavior determination unit autonomously detects the user's body temperature, and based on the detected user state, the emotion determination unit determines at least one of the user's emotion and the avatar's emotion, determines at least one aspect of a gesture and / or speech corresponding to the determined at least one emotion, and causes the behavior control unit to control the avatar based on the determined at least one aspect.

129. A behavior control system as described in claim 128, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

130. A behavior control system according to claim 128, wherein said behavior determining section changes said avatar in accordance with at least one of the emotion determined by said emotion determining section and the user's preference.

131. The behavior control system according to claim 128, wherein said electronic device is a headset type terminal.

132. The behavior control system according to claim 128, wherein said electronic device is a glasses-type terminal.

Images