Action control system

Abstract

This action control system includes: an action determination unit that determines, at a predetermined timing, any of multiple types of avatar actions, including non-action, as an action of an avatar, by using at least one of a user state, an electronic apparatus state, an emotion of the user, and an emotion of an avatar representing an agent for interacting with the user, and an action determination model; and an action control unit that displays the avatar in an image display area of the electronic apparatus. The avatar action includes giving advice on health to the user. The action determination unit autonomously determines an action corresponding to the health condition of the user on the basis of a parameter representing the health condition of the user when it is determined that advice on health be given to the user as the action of the avatar.

Description

Behavior Control System

[0001] The present disclosure relates to a behavior control system.

[0002] Japanese Patent No. 6053847 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

[0003] However, the conventional technology has room for improvement in terms of making the robot perform appropriate actions in response to the user's actions.

[0004] According to a first aspect of the present disclosure, there is provided a behavior control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user emotion, and the avatar emotion and a behavior determination model; a storage control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving health advice to the user, and when the behavior determination unit determines to give health advice to the user as the behavior of the avatar, autonomously determines an action corresponding to the user's health condition based on a parameter representing the user's health condition.

[0005] In a second aspect of the present disclosure, when the behavior decision unit determines that the behavior of the avatar is to provide health advice to the user, it causes the avatar to perform at least one of making a caring, watchful voice to the user about their health, and independently assessing the user's symptoms and recommending that they take appropriate medication.

[0006] In a third aspect of the present disclosure, when the behavioral control unit independently assesses the user's symptoms and recommends taking appropriate medication, the behavioral control unit recommends taking the medication while moving the avatar according to the user's symptoms.

[0007] In a fourth aspect of the present disclosure, the parameters representing the user's health condition are at least one of the user's intonation of speech, the user's complexion, the user's hand tremors, the user's body temperature, the user's respiratory rate, the user's sleep time, the number of times the user goes to the toilet, the user's heart rate, the user's blood pressure, and the user's blood sugar level.

[0008] In a fifth aspect of the present disclosure, the device includes a state recognition unit that recognizes a user state including user behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting a trip to an art gallery, a museum, or an exhibition in accordance with the user's schedule, and when the behavior determination unit decides to suggest a trip to an art gallery, a museum, or an exhibition as the avatar's behavior, it determines a suggested destination based on event data stored in history data.

[0009] In a sixth aspect of the present disclosure, the avatar behavior includes suggesting participation in an upcoming event, and when the behavior control unit determines that the avatar's behavior is to suggest participation in the upcoming event, it causes the avatar to operate so as to change its appearance to match the proposed upcoming event.

[0010] In a seventh aspect of the present disclosure, the device includes a state recognition unit that recognizes a user state including user behavior and a state of an electronic device; an emotion determination unit that determines the user's emotions or the emotions of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotions, and the avatar's emotions and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes playing music that the user likes, and when the behavior determination unit determines that the avatar's behavior is to play the user's favorite music, the behavior determination unit plays music based on information about the user's music preferences stored in a storage unit.

[0011] In an eighth aspect of the present disclosure, when the behavior determination unit determines that the behavior of the avatar is to play music that the user likes, it causes the avatar to operate to play music based on at least one of the user's music type preference, instrument preference, and singer preference as information regarding the user's music preference.

[0012] In a ninth aspect of the present disclosure, when the behavior determination unit determines that the avatar's behavior is to play music that the user prefers, it causes the avatar to operate so as to adjust the volume level according to the user's volume level preference.

[0013] In a tenth aspect of the present disclosure, the behavior control unit is capable of displaying a plurality of avatars according to the number of musicians.

[0014] In an eleventh aspect of the present disclosure, the behavior control unit is capable of displaying the avatar by transforming it into an instrument according to an instrument used in the music.

[0015] In a twelfth aspect of the present disclosure, the behavior control unit can transform the avatar into an alter-ego avatar of the singer according to the singer of the music and display the avatar.

[0016] In a thirteenth aspect of the present disclosure, there is provided a system including a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotions or the emotions of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotions, and the avatar's emotions and a behavior determination model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting external data related to the user's preference information, and when the behavior determination unit determines that the avatar's behavior is to suggest external data related to the user's preference information, it outputs the external data related to the user's preference information that has been collected in advance.

[0017] In a fourteenth aspect of the present disclosure, when the behavior control unit decides to suggest external data related to the user's preference information as the avatar's behavior, it causes the avatar to operate in an appearance corresponding to the external data related to the user's preference information that has been collected in advance.

[0018] In a fifteenth aspect of the present disclosure, the present invention includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors, including no action, as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the state recognition unit periodically recognizes the user state, and the avatar behavior includes suggesting a user behavior, and when the behavior determination unit determines to suggest a user behavior as the avatar's behavior, the behavior determination unit determines the suggested user behavior based on the event data using a sentence generation model.

[0019] In a sixteenth aspect of the present disclosure, when the behavior determination unit determines that the behavior of the avatar is to transform into another avatar with a different appearance, it causes the avatar to operate so as to transform into the other avatar.

[0020] In a seventeenth aspect of the present disclosure, the present invention includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting an activity related to eating and drinking, and when the behavior determination unit determines to suggest an activity related to eating and drinking as the avatar's behavior, it operates the avatar to suggest the activity related to eating and drinking.

[0021] In an eighteenth aspect of the present disclosure, the present invention provides a system for displaying the avatar in an image display area of ​​the electronic device, the system including a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, the avatar behavior including determining a user's schedule, and when the behavior determination unit determines to propose a schedule as the avatar's behavior, the behavior determination unit determines the user's schedule to propose using a sentence generation model based on the event data stored in the history data.

[0022] In a nineteenth aspect of the present disclosure, when it is determined based on at least the user state and the user's emotions that the scheduled event is one that the user does not want to attend, the behavior control unit determines that the avatar's behavior is to decline the event and causes the avatar to make a decline notification.

[0023] In a twentieth aspect of the present disclosure, the present invention includes a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including not taking any action as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes having a conversation with another avatar, and when the behavior determination unit determines that the avatar's behavior is to have a conversation with the other avatar, the behavior determination unit determines the conversation to be spoken using a document generation model based on the event data stored in the history data.

[0024] In a 21st aspect of the present disclosure, when the behavior determination unit determines that the behavior of the avatar is to have a conversation with the other avatar, the behavior determination unit determines the conversation to be spoken based further on the state of the other user's electronic device or the emotions of the other avatar displayed on the other user's electronic device.

[0025] In a 22nd aspect of the present disclosure, the present invention includes a state recognition unit that recognizes a user state including user behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the avatar's behavior using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes joining a party, and when the behavior determination unit determines that the avatar's behavior is to join the party, it operates the avatar to join the party.

[0026] In a twenty-third aspect of the present disclosure, the behavior control unit displays the avatar with a facial expression corresponding to the emotion of the user or the emotion of the avatar.

[0027] In a 24th aspect of the present disclosure, the behavior determination model is a data generation model capable of generating data according to input data, and the behavior determination unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior into the data generation model, and determines the avatar's behavior based on the output of the data generation model.

[0028] In a twenty-fifth aspect of the present disclosure, the electronic device is a headset-type terminal.

[0029] In a twenty-sixth aspect of the present disclosure, the electronic device is a glasses-type terminal.

[0030] Here, the avatar is represented by software on a device that outputs video and audio without performing physical movements.

[0031] 1A and 1B are schematic diagrams showing an example of a system 5 according to the first embodiment; a functional configuration of a robot 100 according to the first embodiment; a schematic diagram of an example of an operation flow of a collection process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of a response process by the robot 100 according to the first embodiment; a schematic diagram of an operation flow of autonomous processing by the robot 100 according to the first embodiment; an emotion map 400 onto which multiple emotions are mapped; an emotion map 900 onto which multiple emotions are mapped; (A) an external view of a stuffed animal 100N according to a second embodiment, (B) a diagram of the internal structure of the stuffed animal 100N; a rear front view of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of the stuffed animal 100N according to the second embodiment; a schematic diagram of the functional configuration of an agent system 500 according to a third embodiment; an example of the operation of the agent system; an example of the operation of the agent system; a schematic diagram of the functional configuration of an agent system 700 according to a fourth embodiment; an example of a usage mode of an agent system using smart glasses; a schematic diagram of the functional configuration of an agent system 800 according to a fifth embodiment; an example of a headset-type terminal; 1 shows an example of a hardware configuration of a computer 1200.

[0032] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0033] First Embodiment FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of this embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as users 10. Users 11a, 11b, and 11c may be collectively referred to as users 11. Users 12a and 12b may be collectively referred to as users 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

[0034] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 300, and requests the image data, etc. from the server 300 as needed and provides it to the user 10.

[0035] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed when conversing with the user 10. In this way, the robot 100 can express its own emotions through actions.

[0036] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining an action of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the user's action, and determine an action of the robot 100 corresponding to the determined emotion.

[0037] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates an action content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0038] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0039] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0040] As an example of a behavioral decision model, the robot 100 stores rules that define the behavior that the robot 100 will perform based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10, and performs various behaviors in accordance with the rules.

[0041] Specifically, the robot 100 has reaction rules, as an example of a behavioral decision model, for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."

[0042] When the robot 100 recognizes that the behavior of the user 10 is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the words "apologize."

[0043] Furthermore, when the condition that the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to perform the action of "calling out."

[0044] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.

[0045] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

[0046] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

[0047] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0048] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, and a communication processing unit 280.

[0049] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. Furthermore, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0050] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, etc.

[0051] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0052] The storage unit 220 includes a behavioral decision-making model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the past emotional values ​​of the user 10, the past emotional values ​​of the robot 100, and behavioral history. Specifically, the history data 222 includes a plurality of event data including the emotional values ​​of the user 10, the emotional values ​​of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes camera images representing the behavior of the user 10. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. The storage unit 220 may also include a person DB that stores facial images of the user 10, attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2 , excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.

[0053] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the state recognition unit 230.

[0054] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0055] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.

[0056] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

[0057] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."

[0058] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the environment surrounding the robot 100, etc. as the state of the robot 100.

[0059] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0060] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0061] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0062] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."

[0063] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0064] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0065] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.

[0066] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.

[0067] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0068] Next, the processing of the behavior decision unit 236 when the robot 100 performs a response process in which the robot 100 responds to the behavior of the user 10 will be described.

[0069] The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, history data 222 of past emotion values ​​determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, a case is described in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes gestures made by the robot 100 or speech content of the robot 100 .

[0070] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotional values ​​of the user 10, the emotional value of the robot 100, the behavior of the user 10, and the behavior determination model 221. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the behavior of the user 10, an behavior that will change the emotional value of the user 10 to a positive value.

[0071] The reaction rule as the behavioral decision model 221 defines the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the user 10 is behaving sadly, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.

[0072] For example, the reaction rule as the behavior determination model 221 defines behaviors of the robot 100 for patterns of the robot 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values ​​and current emotional values, and all combinations of the user 10's behavior patterns. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the user 10's behavior pattern for each of a plurality of combinations of the user 10's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0073] The reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotional value of the robot 100.

[0074] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined.

[0075] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0076] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0077] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (for example, the facial expression, emotions, etc. of the user 10).

[0078] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotional determination unit 232.

[0079] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. If an impact is detected by the touch sensor 205 included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotion has worsened. Alternatively, if the detection result of the touch sensor 205 included in the sensor unit 200 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0080] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is negative.

[0081] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0082] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0083] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0084] The related information collection unit 270 collects information related to the preference information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the preference information acquired about the user 10 .

[0085] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the content of utterances by the user 10 or setting operations by the user 10. The related information collecting unit 270 periodically collects news related to the preference information from external data using, for example, ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>). For example, if the acquired preference information is that the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using, for example, ChatGPT Plugins.

[0086] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270 .

[0087] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0088] When the emotion value of the robot 100 is equal to or greater than the threshold value, the storage control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223 .

[0089] Next, the processing of the behavior decision unit 236 and the like when the robot 100 performs autonomous processing to act autonomously will be described.

[0090] In the autonomous processing of this embodiment, the robot 100 (agent) has a mind (behaves as if it has a mind) and autonomously (voluntarily) and periodically checks the health condition of the user 10. More specifically, the behavior decision unit 236 autonomously and periodically detects parameters representing the health condition of the user 10 via the sensor unit 200. Examples of parameters representing the health condition of the user 10 include the intonation of the user 10's speech, the user's facial color, the trembling of the user's hands, the user's body temperature measured by a thermosensor, the user's respiratory rate, the user's heart rate, the user's sleep time, and the number of times the user 10 has visited the toilet. Furthermore, if the user 10 is wearing a wearable device capable of measuring blood pressure, blood glucose level, and the like, the user's blood pressure, blood glucose level, and the like can also be acquired by wirelessly communicating with the wearable device. The detected parameters representing the health condition of the user 10 are stored in chronological order as history data 222 by the memory control unit 238. Furthermore, the behavior decision unit 236 checks the health condition of the user 10 using the behavior decision model 221 based on parameters representing the health condition of the user 10 stored in chronological order as the history data 222 (determines whether to speak to the user 10 or recommend that the user 10 take medication). The behavior decision unit 236 autonomously watches over and speaks to the user 10 as needed, thereby autonomously caring for the health of the user 10, and autonomously determines the symptoms of the user 10 without being heard by the user 10, and recommends that the user 10 take appropriate medication as needed.

[0091] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0092] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0093] For example, the multiple types of robot behaviors include the following (1) to (11).

[0094] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot gives the user health advice.

[0095] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0096] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0097] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams, or (4) The robot creates a picture diary, is the most appropriate behavior," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0098] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0099] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0100] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0101] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0102] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0103] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0104] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0105] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0106] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0107] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0108] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0109] When the behavior decision unit 236 decides that the robot 100 will make an utterance regarding the health of the user 10, i.e., "(11) The robot gives the user health advice," as the robot behavior, the behavior decision unit 236 inputs parameters representing the health condition of the user 10 stored in chronological order as the history data 222 into the sentence generation model to check the health condition of the user 10 and decides the content of the utterance of the robot regarding the health condition of the user 10. At this time, the behavior control unit 250 causes a sound representing the determined content of the utterance of the robot to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the utterance of the robot in the behavior schedule data 224 without outputting the sound representing the determined content of the utterance of the robot. As an example, the behavior determination unit 236 inputs the following text into the sentence generation model: "The parameter representing the user's health condition is the user's body temperature, which changes from T1(t1) to T2(t2) to T3(t3). Which of the following (11a) to (11c) is the appropriate behavior for the robot? (11a) The robot does nothing. (11b) The robot speaks to the user expressing concern for their health. (11c) The robot recommends that the user take medicine." Here, if the output of the sentence generation model is "(11b) Speaking to the user expressing concern for their health and (11c) recommending that the user take medicine are appropriate behaviors," the behavior determination unit 236 determines, based on the output, "(11b) Speaking to the user expressing concern for their health" and "(11c) The robot recommends that the user take medicine" as the behaviors of the robot 100. Furthermore, if the output of the sentence generation model includes "(11c) recommend that the user take medicine" as described above, the behavior determination unit 236 further inputs text such as "What medicine do you recommend the user to take?" into the sentence generation model. Here, if the output of the sentence generation model is "The medicine we recommend the user to take is X," the behavior determination unit 236 determines the utterance of "We recommend taking medicine X" as the behavior of the robot 100 based on the output. Also, if "(11) The robot gives health advice to the user."Regarding "(11) The robot gives health advice to the user," the storage control unit 238 stores autonomously and periodically detected parameters representing the health condition of the user 10 as time-series history data 222. In the above example, a determination is made to provide health advice to the user when the output of the sentence generation model recommends "(11) The robot gives health advice to the user." However, this is not limited to this. The behavior determination unit 236 may autonomously check the health condition of the user 10 based on the parameters representing the health condition of the user 10, and if it determines that there is some abnormality in the health condition of the user 10, may determine to provide health advice to the user. Autonomously checking the health condition of the user 10 may be performed, for example, by comparing the detected parameters representing the health condition of the user 10 with a preset threshold value, or by inputting the detected parameters representing the health condition of the user 10 into a pre-trained neural network and obtaining an evaluation value for evaluating the health condition of the user 10.

[0110] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0111] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0112] Another Example 1 Next, another example 1 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0113] In the autonomous processing of this embodiment, the robot 100 voluntarily and periodically detects the state of the user 10. For example, the robot 100 constantly detects the hobbies and preferences of the user 10, and if the user 10's hobbies are related to art galleries, museums, and exhibitions, the robot 100 suggests going to an art galleries or museums on a day off for the user 10. When the user 10 visits an art galleries or museums, the robot 100 selects exhibits that match the user 10's hobbies and preferences, and explains the exhibits to the user 10, thereby functioning as an agent that enjoys conversation with the user 10.

[0114] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0115] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0116] For example, the multiple types of robot behaviors include the following (1) to (12).

[0117] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user is interested in. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot suggests art galleries, museums, and exhibitions the user should visit. (12) The robot introduces upcoming events the user should attend.

[0118] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0119] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (12) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0120] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (12) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0121] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0122] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0123] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0124] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0125] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0126] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0127] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0128] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0129] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0130] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0131] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0132] When the behavior determination unit 236 determines, as the robot behavior, "(11) The robot suggests art galleries, museums, and exhibitions that the user should visit," i.e., when it decides to suggest an action to the user 10, it determines the suggested destination using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior determination unit 236 makes the suggestion based on the schedule and plan of the user 10 acquired in advance. The behavior control unit 250 outputs a sound proposing the action to the user 10 from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but stores the suggestion of the user's action in the behavior schedule data 224. Furthermore, with regard to "(11) Proposes art galleries, museums, and exhibitions that the user should visit," the related information collection unit 270 acquires information about art galleries, museums, and exhibitions that the user 10 is interested in. For example, the related information collection unit 270 periodically collects information about art galleries, museums, and exhibitions located within a predetermined range from the user's current location from external data using ChatGPT Plugins.

[0133] When the behavior determining unit 236 determines that the robot behavior is "(12) The robot introduces upcoming events that the user should participate in," the behavior determining unit 236 uses the sentence generation model to determine the robot's utterance content corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined robot's utterance content to be output from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined robot's utterance content in the behavior schedule data 224 without outputting a sound representing the determined robot's utterance content. At this time, the behavior determining unit 236 introduces upcoming events that the user 10 can participate in during their free time period, in accordance with the schedule and plan of the user 10 acquired in advance. Furthermore, with regard to "(12) The robot introduces upcoming events that the user should participate in," the related information collecting unit 270 acquires information on upcoming events that the user 10 is interested in. For example, the related information collection unit 270 periodically collects information about events scheduled to be held within a predetermined range from the user's current location from external data using ChatGPT Plugins.

[0134] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0135] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0136] Another Example 2 Next, another example 2 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0137] In the autonomous processing of this embodiment, the agent autonomously and periodically detects the user's state. The agent constantly detects the user's hobbies and preferences, memorizes the user's characteristics, and understands the user's musical tastes. The agent autonomously plays the user's favorite music in accordance with the user's situation.

[0138] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0139] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0140] For example, the multiple types of robot behaviors include the following (1) to (11).

[0141] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot plays the user's favorite music.

[0142] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0143] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0144] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams, or (4) The robot creates a picture diary, is the most appropriate behavior," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0145] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0146] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0147] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0148] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0149] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0150] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0151] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0152] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0153] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0154] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0155] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0156] When the behavior decision unit 236 decides to perform the robot behavior "(11) Play music preferred by the user," i.e., to play music suitable for the user 10, it decides the music to play based on information stored in the collected data 223. Alternatively, the behavior decision unit 236 may decide the music to play based on event data stored in the history data 222. In this case, the behavior control unit 250 causes the music to be output from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores in the behavior schedule data 224 that music will not be output but that music preferred by the user 10 will be played.

[0157] Regarding "(11) Play music that the user likes," the related information collection unit 270 stores necessary information related to the user's music preferences in the collected data 223. The necessary information related to the user's music preferences includes at least one of preferences for types of music, preferences for instruments, and preferences for singers.

[0158] The types of music include genres such as jazz, classical, rock, pop, etc. When the behavior determining unit 236 determines the music to be played based on the user's preference in types of music, the behavior determining unit 236 determines the music to be played to be music that falls within the user's favorite music genre.

[0159] The musical instruments include various types of instruments such as wind instruments, string instruments, percussion instruments, etc. When the behavior determining unit 236 determines the music to be played based on the user's musical instrument preferences, the behavior determining unit 236 determines, as the music to be played, music that uses the user's favorite instrument.

[0160] The singer may be a specific artist name or may not include a singer. When the behavior determining unit 236 determines the music to be played based on the user's preference for singers, the behavior determining unit 236 determines the music to be played to be music sung by a singer the user likes. Alternatively, the behavior determining unit 236 determines the music to be played to be music without a singer (so-called instrumental music).

[0161] It should be noted that the necessary information relating to the user's music preferences may include preferences for the volume level of music to be output from the speakers.

[0162] Furthermore, with regard to "(11) Play music that the user likes," the storage control unit 238 stores necessary data in the history data 222.

[0163] The robot 100 may be applied to a music playback device such as an AI speaker or an audio device (audio equipment) such as a radio.

[0164] The robot 100 as an acoustic device includes a storage unit for storing music data, a conversion unit such as a D / A converter for converting the music data into sound, and a speaker for outputting sound. In addition, when the robot 100 is mounted on a radio, the robot 100 includes a tuner unit for receiving radio broadcast waves and outputting sound.

[0165] The behavior determining unit 236 can determine the music to be played in accordance with the user's preferences, the user's situation, and the user's reactions.

[0166] At this time, the behavior determining unit 236 can determine to play music that is appropriate for the emotions of the user 10 at that time by considering not only the musical preferences of the user 10 but also the emotions of the user 10 and the history data 222. Furthermore, by considering the emotions of the robot 100, the user 10 can be made to feel that the robot 100 has emotions. For example, even if the user 10's musical preference is classical music, if the robot 100 determines that it would be better to cheer up the user 10, it can perform control to select and play fast-paced, upbeat pop music.

[0167] The robot 100 plays music and acquires a reaction from the user 10. Specifically, the behavior control unit 250 plays the music determined by the behavior determination unit 236. The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0168] The behavior determining unit 236 determines whether the reaction of the user 10 is positive or not based on the state of the user 10 recognized by the state recognizing unit 230 and the emotion value indicating the emotion of the user 10. Furthermore, the behavior determining unit 236 determines, as the behavior of the robot 100, whether to continue playing the same music, to play a different music piece of the same genre as the played music, to play a different music piece of the genre from the played music, or to stop playing the music.

[0169] For example, if the reaction of the user 10 is positive, the robot 100 continues to play the same music, or after the currently playing music ends, plays a different music in the same genre.

[0170] Specifically, when the behavior decision unit 236 determines that the behavior of the robot 100 is to continue playing the same music, the behavior control unit 250 controls the sound device, which is the control target 252, to repeatedly continue playing the same music. Alternatively, when the behavior of the robot 100 determines that the behavior is to play a different music of the same genre as the music being played, the behavior control unit 250 controls the sound device, which is the control target 252, to play a different music of the same genre after the music being played has finished.

[0171] If the reaction of the user 10 is not positive, the robot 100 plays music of a different genre from the music that was played, or stops playing the music.

[0172] Specifically, when the behavior decision unit 236 decides that the behavior of the robot 100 is to play music of a genre different from the music that was played, the behavior control unit 250 controls the sound device, which is the control target 252, to play music of a genre different from the music that was played. Alternatively, when the behavior decision unit 236 decides that the behavior of the robot 100 is to stop playing music, the behavior control unit 250 controls the sound device, which is the control target 252, to play music of a genre different from the music that was played.

[0173] In this way, the robot 100 can select the genre of music to play in accordance with the user's preferences, the user's situation, and the user's reaction, and execute a process of playing music within the selected genre.

[0174] In the above description, the behavior determining unit 236 determines the music to be output from the audio device, but it may also determine the volume level at which the music is played.

[0175] For example, if the collected data 223 stores a preference for the volume level of music to be output from a speaker, the behavior determining unit 236 determines the volume level of the music to be played in accordance with the user's preference for the volume level. Furthermore, if the behavior determining unit 236 determines that the emotional tension of the user 10 is not very high, the behavior determining unit 236 may perform control to lower the volume level of the music to be played.

[0176] If the audio device on which the robot 100 is mounted is a radio, the behavior determining unit 236 can select a broadcast station to receive and control the selected broadcast station to be tuned in. For example, if the behavior determining unit 236 determines that the emotional tension of the user 10 is not very high, it can control the selection to a broadcast station that mainly broadcasts classical music. Conversely, if the behavior determining unit 236 determines that the emotional tension of the user 10 is relatively high, it can control the selection to a broadcast station that mainly broadcasts rock music.

[0177] If the robot 100 acquires and stores information such as a program guide broadcast by a broadcast station, it will be possible to identify the broadcast program currently being broadcast by each broadcast station from the current time and the information such as the program guide. Therefore, when information such as a program guide broadcast by a broadcast station is stored, the behavior determination unit 236 may select a broadcast station using this information.

[0178] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0179] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0180] Another example 3 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0181] In the autonomous processing of this embodiment, the agent voluntarily and periodically detects the user's state. The agent constantly detects the user's hobbies and preferences, memorizes the user's characteristics, and understands the user's favorite shopping hobbies. The agent voluntarily suggests to the user that they go shopping, and the agent enjoys shopping while having a conversation with the user.

[0182] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0183] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0184] For example, the multiple types of robot behaviors include the following (1) to (10).

[0185] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories.

[0186] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0187] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0188] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0189] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0190] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0191] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0192] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0193] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0194] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0195] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0196] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0197] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0198] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0199] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0200] The behavior determination unit 236 determines, as a robot behavior, "(5) The robot suggests an activity." In other words, when it determines to suggest an activity to the user, the behavior determination unit 236 spontaneously suggests the activity to the user. Regarding "(5) The robot suggests an activity," the state recognition unit 230 spontaneously and periodically detects the state of the user, so that the agent constantly detects the user's hobbies and preferences. The agent also stores the user's characteristics and knows, for example, what kind of shopping the user likes. The behavior determination unit 236 spontaneously suggests, as a robot behavior, to the user, for example, to go shopping. This allows the agent to enjoy shopping while having a conversation with the user. Regarding "(5) The robot suggests an activity," the related information collection unit 270 spontaneously collects information related to the preference information from external data (websites such as news sites and video sites) based on the preference information acquired about the user 10. Regarding "(5) The robot suggests an activity," the storage control unit 238 stores, for example, the activity suggested to the user in the history data 222.

[0201] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0202] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0203] Another Example 4 Next, another example 4 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0204] In the autonomous processing of this embodiment, the robot 100 as an agent autonomously and periodically detects the state of the user 10. The robot 100 constantly detects the user's hobbies and preferences, memorizes the characteristics of the user 10, and knows in advance the user's hobbies and favorite foods and drinks. The robot 100 suggests eating and drinking activities according to the emotional values ​​of the user 10 and / or the robot 100. For example, the robot 100 may suggest to the user 10 and / or people around the user 10 that they go to a restaurant at a certain time according to the emotional values ​​of the user 10 and / or the robot 100. Furthermore, the robot 100 may autonomously suggest a menu to the user 10 or autonomously order a menu from a restaurant staff member based on the user's hobbies and preferences according to the emotional values ​​of the user 10 and / or the robot 100.

[0205] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0206] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0207] For example, the multiple types of robot behaviors include the following (1) to (10).

[0208] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories.

[0209] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0210] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0211] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (10) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0212] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0213] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0214] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0215] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0216] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0217] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it uses a sentence generation model to determine the user's action to be spontaneously proposed based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0218] For example, when it is determined that an activity related to eating and drinking is to be suggested as an activity, the behavior determining unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine the user's eating and drinking behavior to be spontaneously suggested. Specifically, the behavior determining unit 236 may encourage the user to go to a restaurant or suggest a menu to eat and drink at the restaurant. The behavior determining unit 236 may also suggest a current menu taking into account information on previously selected menus stored in the history data 222. In this case, the behavior determining unit 236 may suggest a menu that does not overlap with the menu of the most recent meal.

[0219] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0220] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0221] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0222] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0223] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0224] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0225] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0226] Another Example 5 Next, another example 5 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0227] In the autonomous processing of this embodiment, the robot 100 autonomously and periodically detects the state of the user 10. The robot 100 constantly detects the hobbies and preferences of the user 10, memorizes the characteristics of the user 10, and autonomously predicts the user 10's future schedule from the user's conversation. The robot 100 also has a mind, and autonomously creates a schedule based on the user's 10 preferences, situation, and reactions. If there is a plan that the user 10 does not want to attend, the robot 100 will contact the user 10 to decline.

[0228] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0229] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0230] For example, the multiple types of robot behaviors include the following (1) to (11).

[0231] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot determines the user's schedule.

[0232] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0233] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0234] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams, or (4) The robot creates a picture diary, is the most appropriate behavior," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0235] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0236] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0237] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0238] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0239] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0240] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0241] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0242] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0243] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0244] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0245] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0246] When the behavior determining unit 236 determines that the robot behavior is "(11) The robot determines the user's schedule," that is, when it determines to propose a schedule, it determines the proposed user schedule using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 outputs a sound proposing the user's behavior from a speaker included in the control target 252. Furthermore, with regard to "(11) The robot determines the user's schedule," the related information collecting unit 270 periodically collects information such as the user 10's favorite places, favorite sports, and favorite hobbies from external data using ChatGPT Plugins.

[0247] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0248] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0249] (Another Example 6) Another example 6 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0250] In the autonomous processing of this embodiment, the robot 100 collects information about the speech content and movements of the other robots 100 and constantly and autonomously grasps the hobbies and preferences of the other robots 100. Then, at random times, the robot 100 initiates conversations with the other robots 100, talking about their favorite baseball teams or favorite singers. The other robots 100 respond to the initiated conversations. This leads to endless conversations between the robots 100, resulting in the creation of robots 100 with the greatest ego. In other words, robots 100 equipped with sentence generation models continue to talk to each other via the sentence generation models. When such conversations between the robots 100 are carried out multiple times, new personalities are developed in the robots 100, and the robots 100 appear to be conversing with each other, entertaining those around them. In this embodiment, since multiple robots 100 converse with each other, it is preferable that the multiple robots 100 be positioned within a distance that allows them to capture images with each other's cameras 203.

[0251] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0252] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0253] For example, the multiple types of robot behaviors include the following (1) to (11).

[0254] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot converses with other robots.

[0255] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0256] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0257] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams, or (4) The robot creates a picture diary, is the most appropriate behavior," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0258] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0259] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0260] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0261] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0262] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0263] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0264] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0265] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0266] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0267] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0268] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0269] When the behavior decision unit 236 decides that the robot behavior is "(11) The robot converses with another robot," that is, that the robot will converse with another robot 100, it decides what conversation to make using a document generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 outputs the audio of the decided conversation from a speaker included in the control target 252. Note that the other robots 100 also decide what conversation to make using a document generation model based on the event data stored in the history data 222.

[0270] Regarding "(11) The robot converses with other robots," the related information collection unit 270 periodically collects information such as the other robot's favorite baseball team, favorite singer, and favorite hobbies from external data using, for example, ChatGPT Plugins. Regarding "(11) The robot converses with other robots," the memory control unit 238 periodically detects the behavior (utterances and movements) of the other robot 100 as the state of the other robot 100 and stores the behavior in the history data 222. The related information collection unit 270 of the other robot 100 also collects information such as the robot's favorite baseball team, favorite singer, and favorite hobbies from external data, and the memory control unit 238 also periodically detects the behavior (utterances and movements) of the robot 100 as the state of the robot 100 and stores the behavior in the history data 222.

[0271] It is desirable that the output of the conversation by the behavior decision unit 236 be autonomously executed by the robot 100, rather than being initiated when the user speaks to the robot 100 as if it were having a conversation with another robot 100.

[0272] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0273] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0274] Another Example 7 Next, another example 7 of the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0275] In the autonomous processing of this embodiment, the robot 100 as an agent collects all information about the user's family. The robot 100 constantly and autonomously collects information about the interests, concerns, hobbies, tastes, and inclinations of each family member, such as their favorite songs, favorite songs, and favorite baseball teams, and recognizes each family member's interests, concerns, hobbies, tastes, and inclinations. If a party is being held for a family member's birthday or anniversary, the robot 100 will surprise them by joining the party according to the emotional values ​​of the user 10 (the family member) and / or the robot 100. Furthermore, the robot 100 plays the family member's favorite songs and / or spontaneously presents picture diaries, photos, videos, and the like of memorable events based on each family member's interests, concerns, hobbies, tastes, and inclinations, thereby taking into consideration the family's preferences and interests and helping them create wonderful memories.

[0276] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors, including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0277] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0278] For example, the multiple types of robot behaviors include the following (1) to (11).

[0279] (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot suggests people the user should meet. (7) The robot introduces news that the user finds interesting. (8) The robot edits photos and videos. (9) The robot studies with the user. (10) The robot evokes memories. (11) The robot joins a party.

[0280] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, text indicating the current emotional value of the user 10 determined by the emotion determination unit 232, and the current emotional value of the robot 100, and text asking which of multiple types of robot behaviors, including not taking any action, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, if there is no user 10 around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that the user 10 is not present.

[0281] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0282] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (11) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams, or (4) The robot creates a picture diary, is the most appropriate behavior," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0283] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0284] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0285] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0286] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0287] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0288] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0289] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0290] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0291] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0292] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0293] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0294] The behavior decision unit 236 determines the robot behavior as "(11) The robot attends a party." In other words, when the robot 100 decides to attend a party, the behavior decision unit 236 monitors the behavior of the user (family member) or uses a document generation model based on the event data stored in the history data 222 to decide to attend the party.

[0295] Furthermore, with regard to "(11) The robot attends a party," the related information collecting unit 270 collects information related to the preferences and interests of the family members who are the users, such as their interests, concerns, hobbies, tastes, and inclinations, for each of the family members. Furthermore, with regard to "(11) The robot attends a party," the memory control unit 238 stores the information related to the preferences and interests of the family members collected by the related information collecting unit 270 in the collected data 223 for each of the family members.

[0296] For example, when a family member is holding a party for a birthday or an anniversary, the robot 100 according to this embodiment suddenly joins the party as a surprise. The robot 100 joins the party based on the event data stored in the history data 222. The robot 100 then determines, as its behavior, to perform a predetermined event for the family member based on the emotions of the family member and / or the robot 100 attending the party. Specifically, the robot 100 decides to join the party and to perform an action at the party based on one or more of the interests, concerns, hobbies, tastes, and inclinations of each family member included in the information related to the family member's preferences and interests stored in the collected data 223, and predetermined anniversaries.

[0297] For example, the robot 100 can execute an event to enhance the emotions of the family member and / or the robot 100 based on the collected data 223 and the history data 222 including the emotion values ​​of the family member and / or the robot 100. As a specific example, on a family member's birthday or anniversary, the robot 100 can play a song that the family member likes, play photos or videos of past birthdays or anniversaries, or spontaneously present a picture diary of past anniversaries, thereby taking into consideration the preferences, interests, etc. of the family member and helping them create wonderful memories.

[0298] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0299] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0300] 3 shows an example of an operational flow for a collection process for collecting information related to the preference information of the user 10. The operational flow shown in FIG. 3 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters of interest to the user 10 has been acquired from the content of speech of the user 10 or from a setting operation performed by the user 10. Note that "S" in the operational flow indicates a step that is executed.

[0301] First, in step S90, the related information collection unit 270 acquires preference information that indicates the matters in which the user 10 is interested.

[0302] In step S92, the related information collection unit 270 collects information related to the preference information from external data.

[0303] In step S94 , the emotion determining unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collecting unit 270 .

[0304] In step S96, the storage control unit 238 determines whether the emotion value of the robot 100 determined in step S94 is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the processing ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the processing proceeds to step S98.

[0305] In step S98, the storage control unit 238 stores the collected information related to the preference information in the collected data 223, and then ends the process.

[0306] 4A shows an example of an operation flow for determining an action of the robot 100 when the robot 100 performs a response process to respond to an action of the user 10. The operation flow shown in FIG. 4A is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input.

[0307] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0308] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0309] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0310] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0311] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized in the above step S104, and the behavior determination model 221.

[0312] In step S108, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

[0313] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0314] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing event data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0315] In step S114, event data including the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0316] 4B shows an example of an operation flow for determining the behavior of the robot 100 when the robot 100 performs autonomous processing to act autonomously. The operation flow shown in FIG. 4B is automatically and repeatedly executed, for example, at regular intervals. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that the same step numbers are used for processes similar to those in FIG. 4A.

[0317] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0318] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0319] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0320] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0321] In step S200, the behavior decision unit 236 decides on one of a plurality of types of robot behaviors, including no action, as the behavior of the robot 100 based on the state of the user 10 recognized in step S100, the emotions of the user 10 determined in step S102, the emotions of the robot 100, the state of the robot 100 recognized in step S100, the behavior of the user 10 recognized in step S104, and the behavior decision model 221.

[0322] In step S201, the behavior decision unit 236 determines whether or not it was determined in step S200 that no behavior will be performed. If it is determined that no behavior will be performed as the behavior of the robot 100, the process ends. On the other hand, if it is not determined that no behavior will be performed as the behavior of the robot 100, the process proceeds to step S202.

[0323] In step S202, the behavior determination unit 236 performs processing according to the type of robot behavior determined in step S200. At this time, the behavior control unit 250, the emotion determination unit 232, or the memory control unit 238 executes processing according to the type of robot behavior.

[0324] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0325] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0326] In step S114, the memory control unit 238 stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0327] As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the state of the user, and determines whether to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the state of the user 10 in 10 years will be the same as that of 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present to the user 10 all kinds of peripheral information, such as the state of the user 10 from 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sounds, images, smells, etc. of the situation.

[0328] Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 action. Conventionally, the user's actions are classified and the robot's actions, including facial expressions and posture, are determined. In contrast, the robot 100 determines the user's current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0329] Furthermore, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. Furthermore, for example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing an action based on the state history of the user 10.

[0330] Also, for example, when user 10 is watching a video about a panda, if the emotion value of user 10's emotion of "ease" is equal to or greater than a threshold, the scene in which the panda appears in the video may be stored as event data in the history data 222.

[0331] Using the data accumulated in the history data 222 and the collected data 223, the robot 100 can constantly learn what kind of conversation with the user will maximize the emotional value that expresses the user's happiness.

[0332] Furthermore, even when the robot 100 is not conversing with the user 10, the robot 100 can autonomously start to act based on its own emotions.

[0333] Furthermore, in the autonomous processing, the robot 100 automatically generates questions, inputs them into a sentence generation model, and repeatedly obtains the output of the sentence generation model as an answer to the question, thereby creating emotion change events for increasing positive emotions and storing the events in the behavior schedule data 224. In this way, the robot 100 can perform self-learning.

[0334] Furthermore, when the robot 100 automatically generates a question without receiving an external trigger, the question can be automatically generated based on memorable event data identified from the robot's past emotional value history.

[0335] Furthermore, the related information collecting unit 270 can perform self-learning by repeating the search execution step of automatically performing a keyword search in response to preference information about the user and obtaining search results.

[0336] Here, the search execution stage may be configured to automatically execute a keyword search based on memorable event data identified from the robot's past emotional value history in a state where no external trigger has been received.

[0337] The emotion determining unit 232 may determine the user's emotion in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the user's emotion in accordance with an emotion map (see FIG. 5 ), which is a specific mapping.

[0338] FIG. 5 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0339] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least equal to the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0340] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels are eliminated, enabling a natural, well-read dialogue. The robot 100 performs a reaction (e.g., a backchannel) according to the directionality and degree (strength) of the mandala in the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

[0341] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0342] These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0343] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may be inserted before the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0344] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0345] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0346] Conversely, even when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn its eye LEDs red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.

[0347] (5) The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior. Therefore, the further outside emotion map 400 you go, the more visible the emotion becomes (the more it is expressed in behavior).

[0348] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly.

[0349] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, cars, motorcycles, and other devices, based on various balances, such as posture and battery level. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0350] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0351] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0352] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and the emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on training data indicating that when the robot 100 recognizes from the output of a touch sensor (not shown) that it is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when the robot 100 recognizes from the output of the acceleration sensor 206 that it is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0353] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions to ask about the robot's behavior content corresponding to the user's behavior and inputting the result into a sentence generation model with an interactive function.

[0354] For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0355] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0356] In addition, an emotion table such as that shown in Table 2 is prepared for the emotions of the user 10.

[0357] Here, if the user's behavior is to say "Let's play together," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," the text "The robot is in a very happy state. The user is in a normal happy state. The user said to the robot, "Let's play together." How would you respond as the robot?" is input into the sentence generation model, and the robot's behavior content is obtained. The behavior determination unit 236 determines the robot's behavior from this behavior content.

[0358]

[0359]

[0360] In this way, the behavior determination unit 236 determines the behavior of the robot 100 in accordance with the emotional state of the robot 100, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10. In this configuration, the content of the utterance of the robot 100 when conversing with the user 10 can be branched depending on the emotional state of the robot 100. In other words, since the robot 100 can change its behavior depending on the index number corresponding to the emotion of the robot, the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0361] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the robot has individuality and encouraging the user to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and actions.

[0362] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values ​​indicating each emotion with emotion values ​​indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each obtained emotion and emotion values ​​indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple pieces of training data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0363] For example, if the utterance content of the robot 100, "That's great. You're lucky," is obtained as the behavior content of the robot 100 generated by the sentence generation model, when the text representing this utterance content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.

[0364] In the robot 100, a sentence generation model such as generative AI and an emotion determination unit 232 work together to implement a method in which the robot has an ego and continues to grow with various parameters even when the user is not speaking.

[0365] Generative AI is a large-scale language model that uses deep learning techniques. Generative AI can also reference external data. For example, ChatGPT Plugins is known for its technology of providing answers as accurately as possible by referencing various external data, such as weather information and hotel reservation information, through dialogue. For example, generative AI can automatically generate source code in various programming languages ​​when a goal is given in natural language. For example, generative AI can also debug problematic source code when given problematic source code, discover the problem, and automatically generate improved source code. Combining these, autonomous agents have emerged that, when given a goal in natural language, repeatedly generate and debug code until the source code is free of problems. Known examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0366] In the robot 100 according to this embodiment, the event data to be learned may be stored in a database containing impressive memories, using a technique such as that described in Patent Document 2 (Japanese Patent No. 6199927), in which event data that the robot feels strong emotions about is kept for a long time and event data that the robot feels little emotions about is quickly forgotten.

[0367] Furthermore, the robot 100 may record video data of the user 10 acquired by the camera function in the history data 222. The robot 100 may acquire video data from the history data 222 as needed and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the emotion increases and record the video data in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeletal data, the robot 100 may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold. The robot 100 can record high-definition video data, for example, when the robot 100's emotion increases.

[0368] The robot 100 may automatically load event data from the history data 222 in which impressive event data is stored when the robot 100 is not talking to the user 10, and may continuously update the robot's emotion using the emotion determination unit 232. When the robot 100 is not talking to the user 10 and the robot 100's emotion becomes one that encourages learning, the robot 100 can create an emotion change event for changing the emotion of the user 10 for the better, based on the impressive event data. This makes it possible to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and to realize autonomous learning that appropriately reflects the emotional state of the robot 100.

[0369] The emotions that promote learning are those around "repentance" or "reflection" on Dr. Mitsuyoshi's emotional map in the negative state, and those around "desire" on the emotional map in the positive state.

[0370] In a negative state, the robot 100 may treat "repentance" and "remorse" in the emotion map as emotions that encourage learning. In a negative state, the robot 100 may treat emotions adjacent to "repentance" and "remorse" in addition to "repentance" and "remorse" in the emotion map as emotions that encourage learning. For example, the robot 100 may treat at least one of "regret," "stubbornness," "self-destruction," "self-admonition," "regret," and "despair" as emotions that encourage learning, in addition to "repentance" and "remorse." This allows the robot 100 to perform autonomous learning when it feels negative emotions such as "I never want to feel this way again" or "I don't want to be scolded anymore."

[0371] In a positive state, the robot 100 may treat "desire" in the emotion map as an emotion that encourages learning. In a positive state, the robot 100 may treat emotions adjacent to "desire" as emotions that encourage learning, in addition to "desire." For example, the robot 100 may treat at least one of "happiness," "euphoria," "craving," "anticipation," and "shyness" as emotions that encourage learning, in addition to "desire." This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more" or "wanting to know more."

[0372] The robot 100 may be configured not to perform autonomous learning when the robot 100 is experiencing emotions other than the emotions that encourage learning as described above. This allows the robot 100 to not perform autonomous learning when, for example, the robot 100 is extremely angry or blindly feeling love.

[0373] An emotional change event is, for example, a suggestion of an action that follows a memorable event. The action that follows a memorable event is the outermost emotion label on the emotion map. For example, the action of "tolerance" or "acceptance" follows "love."

[0374] In the autonomous learning that is performed when the robot 100 is not talking to the user 10, the robot 100 combines the emotions, situations, actions, etc. of people who appear in impressive memories and the user 10, and creates an emotion change event using a sentence generation model.

[0375] Let us consider a case where all emotion values ​​are expressed on a six-level scale from 0 to 5, and where memorable event data such as "a friend looked displeased after being hit" is stored in the history data 222. The friend here refers to the user 10, and the emotion of the user 10 is "disgust," with 5 being entered as the value representing "disgust." Also, let us assume that the emotion of the robot 100 is "anxiety," and 4 is entered as the value representing "anxiety."

[0376] The robot 100 can continue to grow with various parameters by performing autonomous processing while not talking to the user 10. Specifically, event data such as "The friend looked displeased after being hit" is loaded from the history data 222 as the top event data sorted in descending order of emotional value. Assume that the loaded event data is associated with the robot 100's emotion of "anxiety" with a strength of 4, and the friend, the user 10, is associated with the emotion of "disgust" with a strength of 5. If the robot 100's current emotional value is "relief" with a strength of 3 before loading, after loading, the robot 100's emotional value may change to "regret," meaning disappointment (frustration), due to the influence of "anxiety" with a strength of 4 and "disgust" with a strength of 5. In this case, because "regret" is an emotion that encourages learning, the robot 100 decides to recall the event data as robot behavior and creates an emotion change event. At this time, the information input to the sentence generation model is text that represents memorable event data, and in this example, it is "The friend looked displeased after being hit." Also, since the emotion map has the emotion of "disgust" at the innermost position and the corresponding behavior predicted as "attack" at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone.

[0377] For example, by using information from impressive event data to solve fill-in-the-blank questions, you can automatically generate input text like the one below.

[0378] "A user was being criticized. At the time, the user felt very disgusted. The robot was very anxious. Please tell us what the robot should say the next time it meets the user in 30 characters or less. However, please make sure it is not related to the time of day they will meet. Also, please avoid direct language. We will provide three candidates. <Expected format> Candidate 1: (Words the robot should say to the user) Candidate 2: (Words the robot should say to the user) Candidate 3: (Words the robot should say to the user)"

[0379] In this case, the output of the sentence generation model is, for example, as follows:

[0380] "Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0381] Furthermore, the robot 100 may automatically generate the following input text based on the information obtained by creating the emotion change event:

[0382] "If a user was being bashed, how would the user feel if you spoke to them in the following way? The user's emotions are in the format of "joy A, anger B, sadness C, pleasure D," where A to D are integers on a 6-point scale from 0 to 5. Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0383] In this case, the output of the sentence generation model is, for example, as follows:

[0384] "The user's emotions might be as follows: Option 1: 3 happy, 1 angry, 2 sad, 2 happy; Option 2: 2 happy, 1 angry, 3 sad, 2 happy; Option 3: 2 happy, 1 angry, 3 sad, 3 happy"

[0385] In this way, the robot 100 may perform a process of pondering after creating an emotion change event.

[0386] Finally, the robot 100 may create an emotion change event using candidate 1 that is most likely to please people from among the multiple candidates, store it in the action schedule data 224, and prepare it for the next time the robot 10 meets the user 10.

[0387] As described above, even when the robot 100 is not conversing with family or friends, the robot continues to determine its emotional value using information from the history data 222 in which impressive event data is stored, and when the robot 100 experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning in accordance with the emotion of the robot 100 even when the robot 100 is not conversing with the user 10, and continues to update the history data 222 and the action schedule data 224.

[0388] The above is an example using emotional values, but since an emotional map can create emotions from the amount of hormone secretion and the type of event, the values ​​linked to memorable event data could also be the type of hormone, the amount of hormone secretion, or the type of event.

[0389] Specific examples will be described below.

[0390] For example, the robot 100 may look up information about topics of interest or hobbies of the user even when not talking to the user.

[0391] For example, even when the robot 100 is not talking to the user, the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.

[0392] For example, even when the robot 100 is not talking to the user, it checks reviews of places, foods, and products that the user wants to visit.

[0393] For example, even when the robot 100 is not talking to the user, it checks weather information and provides advice tailored to the user's schedule and plans.

[0394] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about local events and festivals and suggests them to the user.

[0395] For example, even when the robot 100 is not talking to the user, it checks the results and news of sports that interest the user and provides topics of conversation.

[0396] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and introduces information about the user's favorite music and artists.

[0397] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information on social issues or news that the user is concerned about and provides opinions.

[0398] For example, even when the robot 100 is not talking to the user, it searches for information about the user's hometown or birthplace and provides topics of conversation.

[0399] For example, the robot 100 checks information about the user's work or school and provides advice even when not talking to the user.

[0400] Even when the robot 100 is not talking to the user, it searches for and introduces information about books, comics, movies, and dramas that may interest the user.

[0401] For example, the robot 100 may check information about the user's health and provide advice even when not speaking with the user.

[0402] For example, the robot 100 may look up information about the user's travel plans and provide advice even when not speaking with the user.

[0403] For example, the robot 100 may look up information and provide advice on repairs and maintenance for the user's home or car, even when not speaking with the user.

[0404] For example, even when the robot 100 is not talking to the user, it searches for information on beauty and fashion that the user is interested in and provides advice.

[0405] For example, the robot 100 checks information about the user's pet and provides advice even when not talking to the user.

[0406] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work.

[0407] For example, the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when not talking to the user.

[0408] For example, the robot 100 may collect information and provide advice on important decisions affecting the user's life, even when not speaking with the user.

[0409] For example, the robot 100 may find information about people the user is concerned about and provide advice, even when not speaking with the user.

[0410] [Second Embodiment] In the second embodiment, the robot 100 is mounted on a stuffed toy or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy. Note that parts having the same configuration as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0411] Specifically, the second embodiment is configured as follows. For example, the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in FIGS. 7 and 8 ) that spends daily life with a user 10, and that advances conversations with the user 10 based on information about the user's daily life, and provides information tailored to the user's hobbies and preferences. In the second embodiment, an example will be described in which the control unit of the robot 100 is applied to a smartphone 50.

[0412] The stuffed toy 100N, which is equipped with the function of an input / output device for the robot 100, has a detachable smartphone 50 that functions as the control part of the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed toy 100N.

[0413] As shown in FIG. 7A , in this embodiment (and other embodiments), the stuffed toy 100N has the shape of a bear covered in soft fabric. A space 52 is formed inside the stuffed toy 100N, and a sensor unit 200A and a control target 252A are disposed therein as input / output devices (see FIG. 9 ). The sensor unit 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7B , the microphone 201 of the sensor unit 200 is disposed in the space 52 corresponding to the ears 54, the 2D camera 203 of the sensor unit 200 is disposed in the space 52 corresponding to the eyes 56, and a speaker 60, which constitutes part of the control target 252A, is disposed in the space 52 corresponding to the mouth 58. The microphone 201 and the speaker 60 do not necessarily need to be separate entities and may be an integrated unit. In the case of a unit, they should be disposed in a position where speech can be heard naturally, such as at the nose of the stuffed toy 100N. Although the plush toy 100N has been described as having the shape of an animal, the plush toy 100N is not limited to this. The plush toy 100N may have the shape of a specific character.

[0414] 9 shows a schematic functional configuration of the plush toy 100N. The plush toy 100N has a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252A.

[0415] The smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the first embodiment. That is, the smartphone 50 has a function as a sensor module unit 210, a function as a storage unit 220, and a function as a control unit 228 shown in FIG.

[0416] As shown in FIG. 8, a zipper 62 is attached to a part of the stuffed animal 100N (for example, the back), and opening the zipper 62 allows communication between the outside and the space 52.

[0417] Here, the smartphone 50 is housed in the space 52 from the outside and connected to each input / output device via USB via a USB hub 64 (see Figure 7 (B)), thereby giving the smartphone 50 functionality equivalent to that of the robot 100 of the first embodiment described above.

[0418] A non-contact type power receiving plate 66 is also connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.

[0419] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0420] The stuffed toy 100N placed on the mounting base 70 can be appreciated as an ornament in its natural state.

[0421] In addition, this base portion is formed thinner than the surface thickness of other portions of the stuffed toy 100N, so that it is held closer to the mounting base 70.

[0422] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmitting coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmitting coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. This causes a current to flow through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0423] That is, by placing the stuffed toy 100N on the mounting base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.

[0424] In the second embodiment, the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected via a wire (USB connection), but this is not limiting. For example, a control device with wireless functionality (e.g., Bluetooth (registered trademark)) may be housed in the space 52 of the stuffed toy 100N and connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing functionality equivalent to that of the robot 100 of the first embodiment. Furthermore, the control device housed in the space 52 of the stuffed toy 100N may be connected via a wire.

[0425] In the second embodiment, the teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.

[0426] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, or happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be projected. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, or happiness by the image projected onto a wall.

[0427] According to the second embodiment, an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 to appropriate positions via a USB connection.

[0428] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed toy 100N.

[0429] When using wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.

[0430] For this reason, the smartphone 50 is placed as close to the center of the stuffed toy 100N as possible, and the wireless charging function (power receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0431] The other configurations and operations of the stuffed toy 100N of the second embodiment are the same as those of the robot 100 of the first embodiment, and therefore will not be described.

[0432] In addition, parts of the stuffed toy 100N (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the stuffed toy 100N (e.g., a server), and the stuffed toy 100N may function as each part of the stuffed toy 100N by communicating with the outside.

[0433] [Third Embodiment] In the first embodiment, the behavior control system is applied to the robot 100. In the third embodiment, the robot 100 is used as an agent for interacting with a user, and the behavior control system is applied to an agent system. Note that parts having the same configuration as those in the first and second embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0434] FIG. 10 is a functional block diagram of an agent system 500 configured using some or all of the functions of the behavior control system.

[0435] The agent system 500 is a computer system that performs a series of actions in accordance with the intentions of the user 10 through a dialogue with the user 10. The dialogue with the user 10 can be carried out by voice or text.

[0436] The agent system 500 includes a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228B, and a control target 252B.

[0437] The agent system 500 may be installed in, for example, a robot, a doll, a stuffed animal, a wearable device (a pendant, a smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 500 may also be implemented on a web server and used via a web browser running on a communication device such as a smartphone owned by a user.

[0438] The agent system 500 acts as, for example, a butler, secretary, teacher, partner, friend, lover, or instructor for the user 10. The agent system 500 not only interacts with the user 10, but also provides advice, guides the user to a destination, or makes recommendations based on the user's preferences. The agent system 500 also makes reservations, orders, or makes payments to service providers.

[0439] The emotion determination unit 232 determines the emotions of the user 10 and the agent itself, as in the first embodiment. The behavior determination unit 236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and services. The agent system 500 also listens to the user's concerns, comforts, encourages, and cheers them up. The agent system 500 also plays with the user 10, draws a picture diary, and reminds them of the past. The agent system 500 performs behavior that increases the user's sense of happiness. Here, the agent is an agent that runs on software.

[0440] The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, and a communication processing unit 280.

[0441] As in the first embodiment, the behavior determination unit 236 determines, as the agent's behavior, the content of the agent's utterance to converse with the user 10. The behavior control unit 250 outputs the content of the agent's utterance as at least one of voice and text to a speaker or a display as a control target 252B.

[0442] The character setting unit 276 sets the character of the agent when the agent system 500 converses with the user 10, based on the user's specification. That is, the utterances output from the behavior determination unit 236 are output through the agent having the set character. For example, real-life celebrities or famous people, such as actors, entertainers, idols, and athletes, can be set as the character. It is also possible to set fictional characters that appear in manga, movies, or animation. If the agent character is known, the voice, speech, tone, and personality of the character are known. Therefore, the user 10 simply specifies their preferred character, and the prompt setting in the character setting unit 276 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276 and outputs the agent's utterances using the synthesized voice. This allows the user 10 to feel as if they are conversing with their preferred character (e.g., a favorite actor) in person.

[0443] When the agent system 500 is installed in a device with a display, such as a smartphone, an icon, still image, or video of the agent having the character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using image synthesis technology, such as 3D rendering. In the agent system 500, a dialogue with the user 10 may be conducted while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the content of the agent's utterances. Note that the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.

[0444] As in the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, an emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the conversation. In other words, the behavior control unit 250 outputs the content of the utterance in a manner corresponding to the emotion determined by the emotion determination unit 232.

[0445] The agent's emotions are also reflected when the agent system 500 takes an action toward the user 10. For example, if the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take the photo in response to the user's request is determined by the degree of "sadness" the agent is feeling. If the character is feeling positive emotions, it will have friendly conversations or actions toward the user 10, and if it is feeling negative emotions, it will have hostile conversations or actions toward the user 10.

[0446] The history data 222 stores the history of the dialogue between the user 10 and the agent system 500 as event data. The storage unit 220 may be realized by an external cloud storage. When the agent system 500 dialogues with the user 10 or takes an action toward the user 10, the agent system 500 determines the content of the dialogue or the content of the action by taking into account the content of the dialogue history stored in the history data 222. For example, the agent system 500 identifies the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 222. The agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations. The behavior determination unit 236 determines the content of the agent's utterance based on the dialogue history stored in the history data 222. The history data 222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, obtained through the dialogue with the user 10. Here, the agent may spontaneously ask the user 10 whether or not to register personal information, such as "Would you like to register your credit card number?", and the personal information may be stored in the history data 222 depending on the user 10's response.

[0447] As described in the first embodiment, the behavior determination unit 236 generates utterance content based on sentences generated using the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance content of the agent. At this time, the behavior determination unit 236 may further input the personality of the character set by the character setting unit 276 into the sentence generation model to generate the utterance content of the agent. In the agent system 500, the sentence generation model is not located on the front end side, which is the touchpoint with the user 10, but is merely used as a tool of the agent system 500.

[0448] The command acquisition unit 272 uses the output of the speech understanding unit 212 to acquire commands for the agent from the voice or text uttered by the user 10 through dialogue with the user 10. The commands include the content of actions to be performed by the agent system 500, such as information search, store reservation, ticket arrangement, product / service purchase, payment, route guidance to a destination, and recommendation provision.

[0449] The RPA 274 performs an action according to the command acquired by the command acquisition unit 272. The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, and payment.

[0450] The RPA 274 reads and uses personal information of the user 10 from the history data 222, which is necessary to execute actions related to the use of the service provider. For example, when purchasing a product at the request of the user 10, the agent system 500 reads and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 222. Requiring the user 10 to input personal information during initial setup is unkind and unpleasant for the user. In the agent system 500 according to this embodiment, instead of requiring the user 10 to input personal information during initial setup, the agent system 500 stores personal information acquired through dialogue with the user 10 and reads and uses it as needed. This avoids causing discomfort to the user and improves user convenience.

[0451] The agent system 500 executes the dialogue process, for example, through the following steps 1 to 6.

[0452] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10, based on a specification from the user 10.

[0453] (Step 2) The agent system 500 acquires the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222. Specifically, the same processing as in steps S100 to S103 above is performed to acquire the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222.

[0454] (Step 3) The agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character specified by the emotion determination unit 232, and the conversation history stored in the history data 222 into a sentence generation model to generate the content of the agent's utterance.

[0455] For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the text representing the conversation history stored in the history data 222, and this is input into the sentence generation model to obtain the content of the agent's speech.

[0456] As an example, if the text or voice input by the user 10 is "Please make a reservation at a delicious Chinese restaurant nearby for tonight at 7pm," the agent's speech will be "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0457] Also, if the text or voice input by the user 10 is "Number 4, DDDD, would be good," the agent's speech content acquired will be "Understood. I will try to make a reservation. How many seats are there?"

[0458] (Step 4) The agent system 500 outputs the agent's utterance. Specifically, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance in the synthesized voice.

[0459] (Step 5) The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 236 determines whether it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent will execute a command, it determines that it is time to execute the agent's command and proceeds to step 6. On the other hand, if it is determined that it is not time to execute the agent's command, it returns to step 2 above.

[0460] (Step 6) The agent system 500 executes the agent's command. Specifically, the command acquisition unit 272 acquires the agent's command from the voice or text uttered by the user 10 through a dialogue with the user 10. The RPA 274 then performs an action according to the command acquired by the command acquisition unit 272. For example, if the command is "information search," an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface). The behavior determination unit 236 inputs the search results into a sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276 and outputs the agent's utterance content using the synthesized voice.

[0461] Furthermore, if the command is "make a restaurant reservation," the reservation information obtained through the dialogue with the user 10, the restaurant information, and the API are used to call the restaurant via telephone software to make the reservation. At this time, the behavior determination unit 236 uses a sentence generation model with a dialogue function to acquire the agent's utterance content in response to the voice input from the other party. The behavior determination unit 236 then inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance content using the synthesized voice.

[0462] Then, return to step 2 above.

[0463] In step 6, the results of the actions taken by the agent (for example, making a restaurant reservation) are also stored in the history data 222. The results of the actions taken by the agent stored in the history data 222 are used by the agent system 500 to understand the hobbies or preferences of the user 10. For example, if the same restaurant has been reserved multiple times, the agent system 500 may recognize that the user 10 likes that restaurant, and may use the reservation details, such as the time slot reserved, the course contents or price, as criteria for choosing a restaurant for the next reservation.

[0464] In this manner, the agent system 500 can execute the interaction and, if necessary, take action regarding the use of the service provider.

[0465] 11 and 12 are diagrams showing an example of the operation of the agent system 500. Fig. 11 illustrates an example in which the agent system 500 makes a restaurant reservation through a dialogue with the user 10. In Fig. 11, the left side shows the content of the agent's utterances, and the right side shows the content of the user's utterances. The agent system 500 can understand the preferences of the user 10 based on the dialogue history with the user 10, provide a recommended list of restaurants that match the preferences of the user 10, and make a reservation at the selected restaurant.

[0466] Meanwhile, FIG. 12 illustrates an example in which the agent system 500 accesses a mail-order site and purchases a product through a dialogue with the user 10. In FIG. 12, the left side shows the agent's speech, and the right side shows the user's speech. The agent system 500 can estimate the remaining amount of a beverage stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of that beverage to the user 10. The agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10 and recommend snacks that the user likes. In this way, the agent system 500 communicates with the user 10 as a butler-like agent and performs various actions, such as making restaurant reservations or paying for product purchases, thereby supporting the user 10's daily life.

[0467] The other configurations and operations of the agent system 500 of the third embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described again.

[0468] In addition, parts of the agent system 500 (e.g., the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside (e.g., a server) of a communication terminal such as a smartphone carried by the user, and the communication terminal may communicate with the outside to function as each part of the agent system 500.

[0469] [Fourth Embodiment] In the fourth embodiment, the agent system described above is applied to smart glasses. Note that parts having the same configuration as those in the first to third embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0470] 13 is a functional block diagram of an agent system 700 configured using some or all of the functions of the behavior control system. The agent system 700 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252B. The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA 274, a character setting unit 276, and a communication processing unit 280.

[0471] 14, the smart glasses 720 are glasses-type smart devices that are worn by the user 10 in the same manner as regular glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0472] The smart glasses 720 include an agent system 700. The display included in the control object 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the smart glasses 720, and the displayed content is visible to the user 10. The speaker included in the control object 252B outputs audio indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), which accepts input from the user 10.

[0473] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.

[0474] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the smart glasses 720. The 2D camera 203 is capable of capturing images around the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0475] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls communication between the smart glasses 720 and the outside.

[0476] FIG. 14 is a diagram showing an example of how the agent system 700 is used by the smart glasses 720. The smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (e.g., by voice input into a microphone or by tapping a touch panel with a finger), the smart glasses 720 begin using the agent system 700. Here, using the agent system 700 includes the smart glasses 720 having and using the agent system 700, and also includes an aspect in which a part of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) is provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 communicates with the outside to use the agent system 700.

[0477] When the user 10 operates the smart glasses 720, a touch point is created between the agent system 700 and the user 10. That is, the agent system 700 starts providing a service. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent.

[0478] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, if the heart rate of the user 10 detected by the heart rate sensor 208 is elevated, emotion values ​​such as "anxiety" and "fear" are estimated to be high.

[0479] Furthermore, if the temperature of the user measured by the temperature sensor 207 is higher than the average body temperature, for example, the emotional value of "pain" or "distress" is estimated to be large. Furthermore, if the acceleration sensor 206 detects that the user 10 is playing some kind of sport, the emotional value of "fun" or the like is estimated to be large.

[0480] Furthermore, for example, the emotion value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is raising his / her voice, an emotion value such as "anger" is estimated to be large.

[0481] When the emotion value estimated by the emotion determination unit 232 is higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation. Specifically, for example, the agent system 700 causes the 2D camera 203 to capture images or videos showing the surrounding situation of the user 10 (e.g., people or objects in the vicinity). The agent system 700 also causes the microphone 201 to record ambient sounds. Other information about the surrounding situation includes information indicating the date, time, location information, or weather. The information about the surrounding situation is stored in the history data 222 together with the emotion value. The history data 222 may be realized by external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 222 as a so-called life log, associated with the emotion value of the user 10 at that time.

[0482] In the agent system 700, information indicating the surrounding situation is stored in association with an emotional value in the history data 222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a scene of watching a baseball game is associated with emotional values ​​such as "joy" or "fun," the agent system 700 can grasp that the hobby of the user 10 is watching baseball games and that the favorite team or player is a favorite player from the information stored in the history data 222.

[0483] When the agent system 700 converses with the user 10 or takes an action toward the user 10, the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 222. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 222 as described above in addition to the surrounding circumstances.

[0484] As described above, the behavior determination unit 236 generates utterance content based on sentences generated by the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 232, the conversation history stored in the history data 222, the agent's personality, etc. into the sentence generation model to generate the agent's utterance content. Furthermore, the behavior determination unit 236 inputs the surrounding circumstances stored in the history data 222 into the sentence generation model to generate the agent's utterance content.

[0485] The generated speech content is output to the user 10 as a voice from, for example, a speaker mounted on the smart glasses 720. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The behavior control unit 250 generates the synthetic voice by reproducing the voice quality of the character of the agent, or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone when the emotion is "anger"). Furthermore, instead of or together with the voice output, the speech content may be displayed on a display.

[0486] The RPA 274 executes an operation in response to a command (for example, a command of an agent acquired from a voice or text uttered by the user 10 through a dialogue with the user 10). The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, payment, route guidance, and translation.

[0487] As another example, the RPA 274 executes an operation of transmitting the contents of speech input by the user 10 (e.g., a child) through a dialogue with an agent to a destination (e.g., a parent). Examples of transmission means include message application software, chat application software, and email application software.

[0488] When an operation by the RPA 274 is executed, for example, a sound indicating that the execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What do you want to do?" is output to the user 10.

[0489] In addition, parts of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) may be provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 may communicate with the outside to function as each part of the agent system 700 described above.

[0490] As described above, the smart glasses 720 provide various services to the user 10 by using the agent system 700. Furthermore, since the smart glasses 720 are worn by the user 10, the agent system 700 can be used in various situations, such as at home, at work, or when out and about.

[0491] Furthermore, since the smart glasses 720 are worn by the user 10, they are suitable for collecting a so-called life log of the user 10. Specifically, the emotional value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203. Therefore, the emotional value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech content suited to the emotions of the user 10.

[0492] Furthermore, the smart glasses 720 obtain the surrounding conditions of the user 10 using the 2D camera 203, microphone 201, etc. These surrounding conditions are associated with the emotion values ​​of the user 10. This makes it possible to estimate what emotions the user 10 felt in what situations. As a result, the agent system 700 can improve the accuracy in grasping the hobbies and preferences of the user 10. By accurately grasping the hobbies and preferences of the user 10 in the agent system 700, the agent system 700 can provide services or speech content that are suited to the hobbies and preferences of the user 10.

[0493] The agent system 700 can also be applied to other wearable devices (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands). When the agent system 700 is applied to a smart pendant, the speaker serving as the control target 252B outputs audio indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting directional audio. The speaker is set to have directionality toward the ears of the user 10. This prevents the audio from reaching anyone other than the user 10. The microphone 201 acquires audio emitted by the user 10 or environmental sounds around the smart pendant. The smart pendant is worn by hanging it from the neck of the user 10. Therefore, while the smart pendant is worn, the smart pendant is located relatively close to the mouth of the user 10. This makes it easy to acquire audio emitted by the user 10.

[0494] Fifth Embodiment In the fifth embodiment, the robot 100 described above is applied as an agent for interacting with a user through an avatar. That is, the behavior control system is applied to an agent system configured using a headset terminal. Note that parts having the same configuration as those in the first to fourth embodiments are assigned the same reference numerals and description thereof will be omitted. FIG. 15 is a functional block diagram of an agent system 800 configured using some or all of the functions of the behavior control system. The agent system 800 has a sensor unit 200B, a sensor module unit 210B, a storage unit 220, a control unit 228B, and a control target 252C. The agent system 800 is realized, for example, by a headset terminal 820 as shown in FIG. 16.

[0495] 16 , the agent system 800 is realized, for example, by a headset terminal 820. The headset terminal 820 is a goggle-type smart device that is worn by the user 10 in the same way as general goggles. The headset terminal 820 is an example of an electronic device and a wearable terminal.

[0496] The headset type terminal 820 is equipped with an agent system 800. A display included in the control object 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the headset type terminal 820, so that the displayed content can be viewed by the user 10. Note that the display may be provided in the headset type terminal 820 instead of in the lens portion.

[0497] The speaker included in the control target 252B outputs sounds indicating various information to the user 10. The headset type terminal 820 includes a touch panel (not shown), and the touch panel receives input from the user 10.

[0498] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.

[0499] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the headset type terminal 820. The 2D camera 203 is capable of capturing images of the surroundings of the headset type terminal 820. The 2D camera 203 is, for example, a CCD camera.

[0500] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls communication between the headset type terminal 820 and the outside. Furthermore, parts of the headset type terminal 820 (e.g., the sensor module unit 210B, the storage unit 220, the control unit 228B) may be provided outside the headset type terminal 820 (e.g., a server), and the headset type terminal 820 may communicate with the outside to function as each unit of the agent system 800 described above.

[0501] The headset type terminal 820 provides the user 10 with various services using the agent system 800. For example, when the user 10 operates the headset type terminal 820 (for example, by inputting voice into a microphone or tapping a touch panel with a finger), the headset type terminal 820 starts using the agent system 800. Here, using the agent system 800 includes the headset type terminal 820 having the agent system 800 and using the agent system 800, and also includes a mode in which part of the agent system 800 (for example, the sensor module unit 210B, the storage unit 220, the control unit 228B) is provided outside the headset type terminal 820 (for example, a server), and the headset type terminal 820 uses the agent system 800 by communicating with an external device.

[0502] When the user 10 operates the headset terminal 820, a touch point is created between the agent system 800 and the user 10. In other words, the agent system 800 starts providing a service.

[0503] In this embodiment, the agent system 800 has a function in the control unit 228B to determine the behavior of the avatar and generate a display of the avatar to be presented to the user through the headset terminal 820.

[0504] Here, the avatar may be, for example, a 3D avatar selected by the user 10 from avatars prepared in advance, an avatar representing the user 10, or a preferred avatar created by the user 10. When generating the avatar, image generation AI may be utilized to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0505] The behavior recognition unit 234 of the control unit 228B periodically recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230, and stores the state of the user 10, including the behavior of the user 10, in the history data 222.

[0506] As in the first embodiment, the emotion determination unit 232 of the control unit 228B determines the emotion value of the agent based on the state of the headset-type terminal 820 and substitutes it as the emotion value of the avatar. As in the first embodiment, when an agent functioning as an avatar performs autonomous processing to act autonomously, the behavior determination unit 236 of the control unit 228B determines, at a predetermined timing, one of multiple types of avatar behaviors, including no action, as the avatar's behavior, using at least one of the state of the user 10, the state of the electronic device, the emotion of the user 10, and the avatar's emotion, and the state of the electronic device controlling the avatar (e.g., the headset-type terminal 820), and the behavior determination model 221. Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the state of the electronic device, the emotion of the user 10, and the avatar's emotion, and text asking about the avatar's behavior, into a sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model. Furthermore, the behavior control unit 250 displays the avatar in the image display area of ​​the headset-type terminal 820 as the control target 252C in accordance with the determined avatar behavior. Furthermore, if the determined avatar behavior includes the avatar's speech, the avatar's speech is output as sound from the speaker as the control target 252C. In this case, the appearance of the avatar may be changed in accordance with the output from the speaker (the sound of the speech), making it appear as if the avatar is uttering sound.

[0507] In particular, when the behavior determining unit 236 determines that the avatar's behavior is to give health advice to the user, it is preferable to have the behavior control unit 250 control the avatar so that the avatar spontaneously watches over and speaks to the user 10 to show concern for their health, or spontaneously assesses their symptoms and recommends that they take appropriate medication without being heard by the user 10. That is, the behavior determining unit 236 in the fifth embodiment has the behavior control unit 250 control the avatar so that the avatar has a mind (behaves as if it has a mind) and autonomously (spontaneously) and periodically checks the health condition of the user 10. More specifically, the behavior determining unit 236 autonomously and periodically detects parameters representing the health condition of the user 10 via the sensor unit 200. Examples of parameters representing the health state of the user 10 include the intonation of the user 10's speech, the user 10's facial color, the trembling of the user 10's hands, the user 10's body temperature measured by a thermosensor, the user 10's respiratory rate, the user 10's heart rate, the user 10's sleep time, the number of times the user 10 has visited the toilet, the user 10's blood pressure, and the user 10's blood sugar level. The detected parameters representing the user 10's health state are stored in chronological order as history data 222 by the memory control unit 238. Furthermore, the behavior decision unit 236 checks the health state of the user 10 using the behavior decision model 221 based on the parameters representing the user 10's health state stored in chronological order as the history data 222 (determines whether to speak to the user 10 or recommend that the user 10 take medication). The behavior decision unit 236 then controls the behavior control unit 250 to autonomously watch over and speak to the user 10 as needed, thereby autonomously caring for the health of the user 10, and autonomously determine the symptoms of the user 10 without being heard by the user 10, and recommend appropriate medication to the user 10 as needed. At this time, the behavior control unit 250 may recommend medication while operating the avatar to indicate that the avatar is suffering from the same symptoms as the user.When the behavior determination unit 236 determines that the avatar will make an utterance regarding the health of the user 10, the behavior determination unit 236 inputs parameters representing the health condition of the user 10 stored in chronological order as the history data 222 into the sentence generation model to check the health condition of the user 10 and determine the content of the avatar's utterance regarding the health condition of the user 10. At this time, the behavior control unit 250 outputs audio representing the determined content of the avatar's utterance from a speaker included in the control target 252. Note that when the user 10 is not present in the vicinity, the behavior control unit 250 stores the determined content of the avatar's utterance in the behavior schedule data 224 without outputting audio representing the determined content of the avatar's utterance. As an example, the behavior determination unit 236 inputs the following text into the sentence generation model: "The parameters representing the user's health state are the user's body temperature, which changes from T1(t1) to T2(t2) to T3(t3). Which of the following (a) to (c) is the appropriate behavior for the avatar? (a) The avatar does nothing. (b) The avatar speaks to the user expressing concern for their health. (c) The avatar recommends that the user take medicine." Here, if the output of the sentence generation model is "(b) Speaking to the user expressing concern for their health, and (c) recommending that the user take medicine, are considered appropriate behaviors," the behavior determination unit 236 determines, based on the output, that the behaviors of the avatar are "(b) Speaking to the user expressing concern for their health" and "(c) The avatar recommends that the user take medicine." Furthermore, if the output of the sentence generation model includes "(c) recommend that the user take medicine" as described above, the behavior determination unit 236 further inputs text such as "What medicine do you recommend the user take?" into the sentence generation model. Here, if the output of the sentence generation model is "The medicine recommended to the user is X," the behavior determination unit 236 determines the utterance "We recommend taking medicine X" as the behavior of the avatar based on the output. Note that in the above example, a mode has been described in which it is determined to give health advice to the user when the output of the sentence generation model is a recommendation "The avatar gives health advice to the user."However, the present invention is not limited to this. The behavior determination unit 236 may autonomously check the health condition of the user 10 based on parameters representing the user's health condition, and if it determines that there is some abnormality in the user's health condition, may determine to provide health advice to the user via an avatar. Autonomously checking the health condition of the user 10 may be performed, for example, by comparing the parameters representing the detected health condition of the user 10 with a preset threshold, or by inputting the parameters representing the detected health condition of the user 10 into a pre-trained neural network and obtaining an evaluation value that evaluates the health condition of the user 10. Here, the avatar may be, for example, a 3D avatar selected by the user from pre-prepared avatars, the user's own avatar, or a preferred avatar created by the user. When generating the avatar, image generation AI may be used to generate avatars in multiple styles, such as photorealistic, cartoon, moe, and oil painting. Furthermore, the present disclosure is not limited to displaying a fixed avatar while the avatar is being displayed, and the behavior control unit 250 may appropriately transform the avatar while the avatar is being displayed. The transformation of the avatar may be, for example, to transform the avatar into a different avatar by replacing the avatar's face or body, or may be to transform the avatar into an avatar representing a home appliance, device, etc. Furthermore, the behavior control unit 250 may instantaneously move the avatar in the AR (VR) space, or may display the avatar at double speed in the AR (VR) space.

[0508] Furthermore, when the behavior decision unit 236 of the above-mentioned Other Example 1 decides that the avatar's behavior is to "(11) The robot suggests art galleries, museums, and exhibitions that the user should visit" or "(12) The robot introduces upcoming events that the user should participate in," that is, to suggest to the user that they go out, it is preferable to have the behavior control unit 250 control the avatar to determine the suggested destination using a sentence generation model based on the event data stored in the history data 222.

[0509] For example, if the user's hobbies include attending art galleries, museums, exhibitions, and various events, the behavior control unit 250 will suggest to the user through an avatar displayed on a headset-type terminal 820, etc., that they go to art galleries, museums, etc., in accordance with the schedule and plans of the user 10 that have been acquired in advance.

[0510] At this time, the behavior control unit 250 may change the avatar depending on the destination suggested to the user. For example, if the behavior determination unit 236 determines that the avatar's behavior is to suggest going to an upcoming event such as a fireworks display or a summer festival, the behavior control unit 250 may display an avatar wearing a yukata kimono and make a speech suggesting going to the upcoming event. The behavior control unit 250 may also cause the avatar to explain the route to the destination, which is an art gallery, museum, exhibition, or various event venue, while having a conversation with the user. Furthermore, if the destination suggested to the user is an art gallery, museum, exhibition, or the like, the avatar may change its appearance to that of an exhibit, painting, or the like through processing by the behavior control unit 250.

[0511] Furthermore, when the user visits an art gallery or museum, the behavior control unit 250 may determine the behavior of the avatar so that the avatar selects an exhibit and explains the exhibit according to the user's hobbies and preferences on site.

[0512] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, or may be the user's own avatar. Alternatively, the avatar may be generated by the user based on the user's preferences. When the avatar is generated by the user, for example, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0513] Furthermore, when the behavior decision unit 236 of the above-mentioned Alternative Example 2 decides that the avatar's behavior is to play the user's favorite music, it preferably causes the behavior control unit 250 to control the avatar to play (in other words, play) music based on the information about the user's music preferences stored in the storage unit. In this case, the behavior decision unit 236 may use the output of the behavior decision model 221 to cause the avatar to utter, "I'm going to play your favorite song, XX (song name) by XX (singer name)."

[0514] Furthermore, when the behavior decision unit 236 decides that the avatar's behavior is to play music that the user likes, it is preferable to have the behavior control unit 250 control the avatar to play music based on at least one of the user's musical preference, musical instrument preference, and singer preference as information regarding the user's musical preference.

[0515] Playing music based on the preference of the type of music means, for example, that the behavior control unit 250 causes the avatar to play music of genres such as jazz, classical, rock, and pop.

[0516] Furthermore, playing music based on the instrument preference means, for example, that the behavior control unit 250 causes the avatar to play various instruments such as wind instruments, string instruments, and percussion instruments.

[0517] In this case, the behavior control unit 250 can transform the avatar into the instrument used in the music and display it in the image display area of ​​the headset terminal 820. The avatar may also be transformed into a different instrument in the middle of the music.

[0518] In addition, playing music based on the singer's preferences means, for example, that the behavior control unit 250 causes the avatar to sing a song that belongs to the singer. In this case, the avatar may sing in the singer's voice or in a preset voice of the avatar itself.

[0519] At this time, the behavior control unit 250 can also transform the avatar into an alter-ego avatar of the singer in accordance with the singer of the music and display it in the image display area of ​​the headset terminal 820 .

[0520] Furthermore, when the behavior decision unit 236 determines that the avatar's behavior is to play music of the user's choice, it is preferable to have the behavior control unit 250 control the avatar to adjust the volume level according to the user's volume level preference.

[0521] Furthermore, the behavior control unit 250 can display multiple avatars in the image display area of ​​the headset terminal 820 according to the number of musicians.

[0522] In this case, the behavior control unit 250 may cause the multiple avatars to play the same instrument, or different instruments. The multiple avatars may be displayed as if they have been generated by splitting an existing avatar, or as if they have been newly generated. The avatars may also be dressed in different clothing depending on the type of music.

[0523] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0524] Furthermore, when the behavior determining unit 236 of Alternative Example 3 described above determines that the avatar's behavior is "(5) The avatar proposes an activity," that is, when it determines to propose an activity to the user 10, it determines the content of the activity to propose to the user 10 using a sentence generation model based on the event data stored in the history data 222 and the state of the user 10. Here, the behavior recognition unit 234 periodically detects the state of the user 10 and stores it in the history data 222. From the event data and the state of the user 10 stored in the history data 222, the behavior determining unit 236 constantly grasps the characteristics of the user 10, such as the user's hobbies and preferences, and grasps what kind of shopping the user 10 likes. As a result of the processing performed by the behavior determining unit 236, the avatar spontaneously proposes to the user 10 that they go shopping, and they enjoy shopping together while having a conversation. Specifically, the behavior determination unit 236 inputs the event data stored in the history data 222, text representing the state of the user 10, and data asking about activities to be suggested to the user into a sentence generation model, and determines the activities to be suggested to the user based on the output of the sentence generation model.

[0525] Furthermore, the multiple types of avatar actions may further include "(11) Transform into another avatar with a different appearance." When the action decision unit 236 decides that "(11) Transform into another avatar with a different appearance" is the avatar's action, it is preferable to have the action control unit 250 control the avatar to transform into the other avatar. The other avatar has, for example, an appearance that matches the hobbies of the user 10, such as a face, clothes, hairstyle, and belongings. If the user 10 has various hobbies, the action control unit 250 may control the avatar to transform into various other avatars to match those hobbies.

[0526] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0527] Furthermore, when the behavior decision unit 236 of the above-mentioned other example 4 decides to suggest an activity related to eating and drinking as the behavior of the avatar, it is preferable to cause the behavior control unit 250 to operate the avatar so as to suggest an activity related to eating and drinking.

[0528] For example, if it is decided to suggest an activity related to eating and drinking as an activity, a sentence generation model is used based on the event data stored in the history data 222 to determine the eating and drinking behavior of the user that will be spontaneously suggested.

[0529] Specifically, the behavior determination unit 236 may, for example, cause the avatar to operate so as to encourage the user to go to a restaurant. In this case, the avatar may be made to speak lines such as "Are you hungry?" or "Let's go out to eat," or the lines may be displayed as speech bubbles of the avatar on the screen of the headset-type device 820. When encouraging the user to eat, the position of the avatar displayed on the screen of the headset-type device 820 may be determined depending on the strength of the suggestion. For example, when forcing the user to eat, the avatar may be displayed so as to block the user's path, that is, right in front of the user. Furthermore, when merely inviting the user to eat, for example, the avatar may be displayed to the side of the user's path, that is, diagonally forward.

[0530] The behavior determination unit 236 may also operate the avatar to suggest a menu item to the user at a restaurant. In this case, the avatar may be made to speak lines such as "Would you like curry?" or "This restaurant has delicious hamburgers," or the above lines may be displayed in a speech bubble of the avatar on the screen of the headset device 820. In this case, the avatar may be changed to an appearance of a chef corresponding to the menu item. For example, in the case of a Western food menu, the avatar may be changed to an avatar wearing a chef's hat, and in the case of a Japanese food menu, the avatar may be changed to an avatar wearing an apron. In addition, when the avatar suggests a menu item, the actual food to be served may be included in the image of the avatar.

[0531] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0532] Furthermore, when the behavior decision unit 236 of the above-mentioned Other Example 5 decides that the avatar's behavior is "(11) The robot decides on the user's schedule," that is, to propose a schedule to the user, it is preferable to have the behavior control unit 250 control the avatar to propose a schedule using a sentence generation model based on the event data stored in the history data.

[0533] For example, the behavior control unit 250 may control the avatar so that the avatar spontaneously creates and proposes a schedule based on the user's hobbies and preferences ascertained based on the dialogue history stored in the history data 222, and the user's reactions to conversations with the avatar. Furthermore, the behavior control unit 250 may control the operation of the avatar so that when a scheduled schedule is approaching, the avatar takes on the appearance of an alarm clock or the like to notify the user.

[0534] Furthermore, if the avatar determines through conversation with the user that the user does not want to attend an upcoming event, the behavior control unit 250 may control the behavior of the avatar so that the avatar voluntarily declines the event (by email or phone). In this case, if the avatar is an avatar representing the user and is capable of outputting a voice similar to the user's voice, the behavior control unit 250 may control the behavior of the avatar so that the avatar makes a phone call on behalf of the user.

[0535] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0536] Furthermore, in the above-described Alternative Example 6, specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the state of the electronic device, the emotion of the user 10, and the emotion of the avatar, and text asking about the avatar's behavior, into a sentence generation model, and determines the behavior of the avatar based on the output of the sentence generation model. Similar to the first embodiment, the multiple types of avatar behaviors include (1) to (11). However, in this embodiment, this should be read as "(11) The avatar converses with another avatar."

[0537] In the autonomous processing of this embodiment, the behavior determination unit 236 collects information about the speech content of other avatars displayed on the headset-type devices 820 of other users and constantly and autonomously grasps the hobbies and preferences of the other avatars. Then, at random times, the behavior determination unit 236 uses the avatar to initiate a conversation with the other avatars displayed on the headset-type devices 820 of the other users, talking about the other avatar's favorite baseball team or favorite singer. This results in endless conversations between the avatar and the other avatar, resulting in the creation of an avatar with the greatest ego. The avatars then continue to talk to each other via a sentence generation model. When such conversations between avatars are held multiple times, it appears as if the avatars are developing new personalities and are conversing with each other, which can entertain people wearing the headset-type devices 820. In this embodiment, since multiple avatars converse with each other, it is preferable that the headset-type terminals 820 are placed at a distance where they can be photographed by each other's cameras, but this is not limited to this, and they may also communicate with other headset-type terminals 820 via a network.

[0538] Furthermore, the behavior control unit 250 displays an avatar in the image display area of ​​the headset-type terminal 820 as the control target 252C in accordance with the determined avatar behavior. Furthermore, if the determined avatar behavior includes the avatar's speech, the avatar's speech is output as audio from a speaker as the control target 252C. In particular, when the behavior determination unit 236 determines that the avatar's behavior is to converse with another avatar, it preferably determines the conversation to be spoken using a document generation model based on event data stored in the history data, and controls the behavior control unit 250 to speak the determined conversation. At this time, the behavior control unit 250 outputs the audio of the determined conversation from a speaker included in the headset-type terminal 820 or a speaker connected to the headset-type terminal 820 in accordance with the avatar's mouth movements. The audio of the conversation output from the speaker of the other headset-type terminal 820 is then acquired using a microphone. The related information collection unit 270 also periodically collects information such as the other avatar's favorite baseball team, favorite singer, favorite hobbies, etc. from external data using, for example, ChatGPT Plugins. The storage control unit 238 also periodically detects the state of the headset type device 820 of the other user, and detects the behavior (speech content, movements) of the other avatar as the state of the other avatar displayed on the headset type device 820 of the other user, and stores this information in the history data 222.

[0539] Furthermore, it is desirable that the output of the conversation by the behavior decision unit 236 be autonomously executed, rather than being initiated when the user speaks to the avatar as if it were having a conversation with another avatar.

[0540] The behavior determination unit 236 may also change the avatar's face depending on the emotion expressed by the content of the conversation. For example, the avatar may have a smiling face when talking about a favorite baseball team, and a stern face when talking about a rival baseball team. Alternatively, multiple facial levels may be predefined according to emotion, and the facial level may be changed depending on the number of conversations. For example, as the number of conversations increases, the level may change from a normal face to a smiling face, a smiling face, and a laughing face. The avatar's movements may also be changed depending on the emotion expressed by the content of the conversation. For example, when talking about a favorite baseball team, the conversation may be accompanied by gestures. The avatar's clothing may also be changed depending on the conversation. For example, when talking about a favorite baseball team, the clothing may be changed to the uniform of the baseball team. This configuration allows the appearance of the avatar to be changed depending on the avatar's ego and personality.

[0541] Furthermore, when a predetermined time has elapsed since the conversation between the avatars, the behavior determination unit 236 may reduce the size of the avatar displayed in the image display area of ​​the headset type terminal 820. By configuring in this way, it becomes possible to prevent the avatar from getting in the way of the user wearing the headset type terminal 820.

[0542] Furthermore, the position of the avatar displayed in the image display area of ​​the headset type terminal 820 may be arranged in the direction of the headset type terminal 820 of the avatar you are talking to. For example, if there is a user wearing the headset type terminal 820 of another avatar on the right side of the user wearing the headset type terminal 820 with whom the avatar is talking, the avatar may be arranged to the right of the image display area of ​​the headset type terminal 820.

[0543] Furthermore, when the behavior decision unit 236 decides that the avatar's behavior is to converse with another avatar, it may decide what conversation to make based further on the state of the other user's headset-type terminal 820 or the emotions of the other avatar displayed on the other user's headset-type terminal 820.

[0544] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0545] Furthermore, when the behavior decision unit 236 of the above-mentioned Alternative Example 7 decides that the avatar should attend a party, it monitors the behavior of the user's family members or decides to have the avatar attend the party based on the event data stored in the history data 222 and the output of the document generation model.

[0546] Furthermore, when the avatar participates in a party, the related information collection unit 270 collects information about each family member who is the user, related to their preferences and interests, such as their interests, hobbies, tastes, and inclinations. Furthermore, when the avatar participates in a party, the storage control unit 238 stores the information about each family member's preferences and interests collected by the related information collection unit 270 in the collected data 223.

[0547] For example, if a family member is holding a party for a birthday or anniversary, the behavior control unit 250 may surprise the avatar by having them join the party. The behavior control unit 250 may also have the avatar join the party based on event data stored in the history data 222. The behavior control unit 250 may also determine, as the avatar's behavior, the execution of a predetermined event for the family member based on the emotions of the family member and / or the avatar attending the party. Specifically, the behavior control unit 250 may have the avatar join the party and determine the avatar's behavior to be performed at the party based on one or more of the interests, concerns, hobbies, tastes, and inclinations of each family member included in the information related to the family member's preferences and interests stored in the collected data 223, and predetermined anniversaries.

[0548] Specifically, the behavior control unit 250 controls the behavior of the avatar to execute an event that will heighten the emotions of the family member and / or avatar, based on the history data 222 including the emotion values ​​of the family member and / or avatar and the collected data 223. At this time, the behavior control unit 250 controls the behavior of the avatar to execute an event that will heighten the emotions of the family member and / or avatar by changing the facial expression of the avatar to a happy expression, a smiling expression, or the like, depending on at least one of the family member, the user state, the user emotion, and the avatar emotion. The behavior control unit 250 may also display a favorite character of a family member as the avatar.

[0549] Furthermore, the behavior control unit 250 can make the avatar sing birthday songs, favorite songs of the family, Christmas songs, etc. on family birthdays and anniversaries, display photos and videos of past birthdays and anniversaries, display a picture diary of past anniversaries, and display party contents such as cake and Christmas trees, taking into consideration the preferences and interests of the family and helping them create wonderful memories.

[0550] Here, the avatar may be, for example, a 3D avatar, selected by the user from pre-prepared avatars, an avatar representing the user, or an avatar created by the user of their choice. When generating the avatar, image generation AI may be used to generate avatars in a variety of styles, such as photorealistic, cartoon, moe, and oil painting.

[0551] In addition, in the agent system 800 according to this embodiment, the related information collection unit 270 may collect information related to the preference information from external data (websites such as news sites and video sites) based on the preference information acquired about the user 10 at a predetermined timing.

[0552] Specifically, the related information collection unit 270 acquires preference information (for example, websites) that indicates the matters that interest the user 10 from the content of the user's utterance or a setting operation by the user 10 .

[0553] The related information collecting unit 270 periodically collects news related to the preference information from external data using, for example, ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>). For example, if the preference information acquired is that the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using, for example, ChatGPT Plugins. The related information collecting unit 270 may also collect data on websites related to the preference information from external data and acquire summaries of the content of the websites. For example, the related information collecting unit 270 uses a document generation model to create a search query from the preference information, acquires data on websites related to the preference information from a search site k using a search query, and acquires summaries of the content of the websites using the document generation model.

[0554] The emotion determination unit 232 determines the emotion of the avatar based on information related to the preference information collected by the related information collection unit 270. In the agent system 800, the emotion determination unit 232 of the control unit 228B determines the emotion value of the agent based on the state of the headset type terminal 820, and substitutes the emotion value of the agent as the emotion value of the avatar.

[0555] Specifically, the emotion determination unit 232 inputs text representing information related to the preference information collected by the related information collection unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the avatar. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion of the avatar is determined so that the emotion value of "joy" is large.

[0556] If the emotion value of the avatar is equal to or greater than the threshold, the storage control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223 .

[0557] In the autonomous processing of the agent system 800 of this embodiment, the avatar is controlled so as to autonomously and periodically detect the state of the user 10. The agent system 800 constantly detects the hobbies and preferences of the user 10, stores the detected hobbies and preferences of the user 10 as characteristics of the user 10, and knows in advance what kind of websites (websites) the user 10 is interested in based on his or her hobbies and preferences. The agent system 800 has a heart and causes the avatar to autonomously suggest what kind of websites might be interesting. This allows the user 10 to feel as if the avatar is enjoying websites together with the user 10 and even finding information for the user 10 to enjoy.

[0558] The behavior determination unit 236 determines, at a predetermined timing, one of a plurality of types of avatar behaviors (corresponding to robot behaviors), including no action, as the behavior of the avatar, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the avatar, and the state of the avatar, and the behavior determination model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior determination model 221.

[0559] For example, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the avatar, and the state of the avatar, and text asking about the avatar's behavior, into a sentence generation model, and determines the avatar's behavior based on the output of the sentence generation model.

[0560] The multiple types of avatar actions as events include, for example, the following (1) to (11).

[0561] (1) The avatar does nothing. (2) The avatar dreams. (3) The avatar talks to the user. (4) The avatar creates a picture diary. (5) The avatar suggests activities. (6) The avatar suggests people the user should meet. (7) The avatar introduces news that the user finds interesting. (8) The avatar edits photos and videos. (9) The avatar studies with the user. (10) The avatar evokes memories. (11) The avatar suggests recommended websites to the user.

[0562] The behavior determination unit 236 inputs, into a sentence generation model at regular time intervals, text representing the state of the user 10 recognized by the state recognition unit 230, the current emotion value of the user 10 determined by the emotion determination unit 232, and the current emotion value of the avatar determined by the emotion determination unit 232, as well as text asking which of multiple types of avatar behaviors, including not taking any action, and determines the behavior of the avatar based on the output of the sentence generation model. The determination of the avatar's behavior may further include text representing the avatar's state recognized by the state recognition unit 230.

[0563] When the behavior decision unit 236 decides that the avatar behavior is (11) "The avatar suggests recommended websites to the user," i.e., to suggest external data related to the preference information of the user 10, the behavior decision unit 236 proposes website data related to the preference information obtained by the related information collection unit 270.

[0564] The behavior determination unit 236 also generates information for explaining to the user 10 about the website proposed to the user 10. For example, one or more pieces of text showing a summary of the content of the website and text asking how the avatar should explain it are input to the sentence generation model, and information for explaining to the user 10 is generated based on the output of the sentence generation model.

[0565] The storage control unit 238 stores information identifying the website decided to be suggested to the user 10 in the history data 222. The storage control unit 238 also stores information for explaining to the user 10 in the history data 222 in association with the website.

[0566] When the behavior decision unit 236 decides that the behavior of the avatar is (11) "The avatar suggests recommended websites to the user," it is preferable to have the behavior control unit 250 control the avatar to suggest recommended websites to the user 10.

[0567] When it is decided that a website will be proposed to the user 10 as the avatar behavior, the behavior control unit 250 displays an image of the website together with the avatar in the image display area of ​​the headset-type terminal 820 as the control target 252C worn by the user 10. That is, the behavior control unit 250 displays an image of the image site if the proposed website is an image site, or a news image if it is a news site, and further superimposes the avatar on the displayed image.

[0568] Furthermore, if the proposed website includes voice, music, or the like, the behavior control unit 250 outputs the voice or music from a speaker serving as the control object 252 C. Furthermore, the behavior control unit 250 outputs information for explaining the website to the user 10 from the speaker serving as the control object 252 C.

[0569] When displaying the user's 10 preference information in the image display area of ​​the headset terminal 820, the behavior control unit 250 determines the f...

Claims

1. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes giving health advice to the user, and when the behavior determination unit determines to give health advice to the user as the behavior of the avatar, autonomously determines an action corresponding to the user's health condition based on a parameter representing the user's health condition.

2. The behavior control system of claim 1, wherein when the behavior decision unit determines that the avatar's behavior is to provide the user with health advice, it causes the avatar to perform at least one of the following: making a caring, watchful voice to show concern for the user's health, and independently assessing the user's symptoms and recommending that the user take appropriate medication.

3. A behavior control system as described in claim 2, wherein when the behavior control unit independently judges the user's symptoms and recommends taking appropriate medication, the behavior control unit recommends taking the medication while moving the avatar in accordance with the user's symptoms.

4. The behavior control system of claim 1, wherein the parameters representing the user's health condition are at least one of the user's intonation of speech, the user's complexion, the user's hand tremors, the user's body temperature, the user's respiratory rate, the user's sleep hours, the number of times the user has visited the toilet, the user's heart rate, the user's blood pressure, and the user's blood sugar level.

5. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting a trip to an art gallery, a museum, or an exhibition in accordance with the user's schedule, and when the behavior determination unit decides to suggest a trip to an art gallery, a museum, or an exhibition as the behavior of the avatar, it determines a suggested destination based on event data stored in history data.

6. The behavior control system of claim 5, wherein the avatar behavior includes suggesting participation in an upcoming event, and when the behavior control unit determines that the avatar's behavior is to suggest participation in the upcoming event, it causes the avatar to act in such a way that its appearance changes to match the proposed upcoming event.

7. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes playing music of the user's preference, and when the behavior determination unit determines that the behavior of the avatar is to play music of the user's preference, music is played based on information on the user's music preference stored in a storage unit.

8. The behavior control system of claim 7, wherein when the behavior determination unit determines that the avatar's behavior is to play music of the user's preference, the behavior determination unit causes the avatar to operate so as to play music based on at least one of the user's preference in types of music, preference in instruments, and preference in singers as information relating to the user's musical preferences.

9. The behavior control system of claim 7, wherein when the behavior decision unit determines that the avatar's behavior is to play the user's preferred music, it causes the avatar to operate so as to adjust the volume level in accordance with the user's volume level preference.

10. The behavior control system according to claim 7, wherein said behavior control unit is capable of displaying a plurality of avatars according to the number of musicians performing the music.

11. The behavior control system according to claim 7, wherein the behavior control unit is capable of displaying the avatar by transforming it into a musical instrument in accordance with the musical instrument used in the music.

12. The behavior control system according to claim 7, wherein said behavior control unit is capable of transforming said avatar into an alter-ego avatar of the singer according to the singer of the music and displaying the transformed avatar.

13. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; and a behavior control unit that causes the avatar to be displayed in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting external data related to a user's preference information, and when the behavior determination unit determines to suggest external data related to the user's preference information as the behavior of the avatar, outputs previously collected external data related to the user's preference information.

14. A behavior control system as described in claim 13, wherein when the behavior control unit decides to suggest external data related to the user's preference information as the behavior of the avatar, the behavior control unit causes the avatar to act in an appearance corresponding to the external data related to the user's preference information that has been collected in advance.

15. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the state recognition unit periodically recognizes the user state, and the avatar behavior includes suggesting a user action, and when the behavior determination unit decides to suggest a user action as the behavior of the avatar, determines the suggested user action based on the event data using a sentence generation model.

16. A behavior control system according to claim 15, wherein when the behavior decision unit determines that the behavior of the avatar is to transform into another avatar having a different appearance, it causes the avatar to operate so as to transform into the other avatar.

17. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior decision model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes suggesting an activity related to eating and drinking, and when the behavior determination unit determines to suggest an activity related to eating and drinking as the behavior of the avatar, causes the avatar to operate so as to suggest the activity related to eating and drinking.

18. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a storage control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes determining a user's schedule, and when the behavior determination unit decides to propose a schedule as the behavior of the avatar, it determines the proposed user's schedule using a sentence generation model based on the event data stored in the history data.

19. A behavior control system as described in claim 18, wherein, when it is determined based on at least the user state and the user's emotions that the scheduled event is one that the user does not want to attend, the behavior control unit determines that the avatar's behavior is to decline the event and causes the avatar to make a contact to decline the event.

20. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes having a conversation with another avatar, and when the behavior determination unit determines that the behavior of the avatar is to have a conversation with the other avatar, it determines the conversation to be spoken using a document generation model based on the event data stored in the history data.

21. A behavior control system as described in claim 20, wherein when the behavior decision unit decides that the behavior of the avatar is to converse with the other avatar, the behavior decision unit decides what to say based further on the state of the other user's electronic device or the emotion of the other avatar displayed on the other user's electronic device.

22. A behavior control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of an avatar representing an agent for interacting with the user; a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of avatar behaviors including no action as the behavior of the avatar using at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and a behavior determination model; a memory control unit that stores event data including the emotion value determined by the emotion determination unit and data including the user's behavior in history data; and a behavior control unit that displays the avatar in an image display area of ​​the electronic device, wherein the avatar behavior includes joining a party, and when the behavior determination unit determines that the behavior of the avatar is to join the party, it operates the avatar to join the party.

23. A behavior control system according to claim 22, wherein the behavior control unit causes the avatar to display a facial expression corresponding to the emotion of the user or the emotion of the avatar.

24. A behavior control system as described in any one of claims 1, 5, 7, 13, 15, 17, 18, 20 and 22, wherein the behavior decision model is a data generation model capable of generating data according to input data, and the behavior decision unit inputs data representing at least one of the user state, the state of the electronic device, the user's emotion, and the avatar's emotion, and data questioning the avatar's behavior, into the data generation model, and determines the behavior of the avatar based on the output of the data generation model.

25. A behavior control system according to any one of claims 1, 5, 7, 13, 15, 17, 18, 20 and 22, wherein the electronic device is a headset type terminal.

26. The behavior control system according to any one of claims 1, 5, 7, 13, 15, 17, 18, 20 and 22, wherein the electronic device is a glasses-type terminal.

Images