Behavior control system

Abstract

There is provided a behavior control system including: an emotion determination unit that determines an emotion of a user or an emotion of a robot; and a behavior determination unit that generates behavior content of the robot for the behavior of the user and the emotion of the user or the emotion of the robot on the basis of a dialogue function that causes the user to dialogue with the robot, determines the behavior of the robot corresponding to the behavior content, and determines the behavior of the robot according to the behavior content. The behavior determination unit executes: a process of reading a learning textbook including a textbook; a process of creating a question and setting the question to the user on the basis of the set target standard score and the read learning material; and a process of creating a new question and setting a question to the user on the basis of a standard score higher than the target standard score and the read learning textbook in a case where an answer to the question set to the user by the user is correct.

Description

Technical Field

[0001] This invention relates to a behavior control system. Background Technology

[0002] Patent Document 1 discloses a technique for determining appropriate robot behavior based on a user's state. In contrast to the prior art of Patent Document 1, user responses are identified when the robot performs a specific action. If the identified user responses cannot determine the robot's behavior, the robot's behavior is updated by receiving information from a server related to actions suitable for the identified user state.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent No. 6053847 However, there is room for improvement in the existing technology in enabling robots to perform appropriate actions in response to user behavior. Summary of the Invention

[0004] According to a first aspect of the present invention, a behavior control system is provided. The behavior control system includes: an emotion determination unit for determining the emotion of a user or the emotion of a robot; and a behavior determination unit for generating robot behavior content based on a dialogue function enabling the user to converse with the robot, taking into account the user's behavior and the user's or robot's emotion, and determining the robot's behavior corresponding to the behavior content. The behavior determination unit performs the following processes: reading learning materials including textbooks; creating questions and presenting them to the user based on a set target standard score and the read learning materials; and, if the user's answer to the questions presented to the user is correct, creating new questions and presenting them to the user based on a standard score higher than the target standard score and the read learning materials. Attached Figure Description

[0005] Figure 1 An example of system 5 involved in this embodiment is shown in a simplified manner.

[0006] Figure 2 The functional structure of robot 100 is shown in general terms.

[0007] Figure 3 An example of the operation process of robot 100 is shown in a general way.

[0008] Figure 4 This is a simplified illustration of an example of the hardware structure of the computer 1200.

[0009] Figure 5 An emotion map 400 is shown, mapping multiple emotions.

[0010] Figure 6 An emotion map 900 is shown, mapping multiple emotions.

[0011] Figure 7 (A) is an appearance drawing of a plush toy according to other embodiments. Figure 7 (B) is a diagram of the internal structure of a plush toy.

[0012] Figure 8 This is a rear front view of a plush toy as described in other embodiments.

[0013] Figure 9A The functional structure of the robot 100 according to the second embodiment is shown in a schematic manner.

[0014] Figure 9B An example of the collection and processing operation flow performed by the robot 100 according to the second embodiment is shown in a schematic manner.

[0015] Figure 9C An example of the operation flow of autonomous processing performed by the robot 100 according to the second embodiment is shown in a schematic manner.

[0016] Figure 9D The functional structure of the plush toy 100N according to the third embodiment is shown in a schematic diagram.

[0017] Figure 9E The functional structure of the intelligent agent system 2500 according to the fourth embodiment is shown in a general way.

[0018] Figure 9F An example of the operation of an intelligent agent system is shown.

[0019] Figure 9G An example of the operation of an intelligent agent system is shown.

[0020] Figure 9H The functional structure of the smart glasses 2700 according to the fifth embodiment is shown in a schematic diagram.

[0021] Figure 9I This illustrates one example of how an intelligent agent system can be used via smart glasses.

[0022] Explanation of reference numerals in the attached figures 5: System; 10, 11, 12: User; 20: Communication Network; 100, 100N, 101, 102: Robot; 200: Sensor Unit; 201: Microphone; 202: Depth Sensor; 203: Camera; 204: Proximity Sensor; 210: Sensor Module Unit; 211: Voice Emotion Recognition Unit; 212: Speech Understanding Unit; 213: Facial Expression Recognition Unit; 214: Facial Recognition Unit; 220: Storage Unit; 221: Reaction Rules; 222: Historical Data; 230: User Status Recognition Unit; 232: Emotion Determination Unit; 234: Behavior Identification unit; 236: Behavior determination unit; 238: Storage control unit; 250: Behavior control unit; 252: Controlled object; 280: Communication processing unit; 300: Server; 1200: Computer; 1210: Main controller; 1212: CPU; 1214: RAM; 1216: Graphics controller; 1218: Display device; 1220: Input / output controller; 1222: Communication interface; 1224: Storage device; 1226: DVD drive; 1227: DVD-ROM; 1230: ROM; 1240: Input / output chip. Detailed Implementation

[0023] The present invention will now be described through embodiments thereof, but these embodiments do not limit the technical solutions covered by the claims. Furthermore, not all feature combinations described in the embodiments are essential to the solutions provided by the invention.

[0024] [First Implementation Method] Figure 1 An example of system 5 according to this embodiment is shown in general terms. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. In this embodiment, users 10a, 10b, 10c, and 10d are sometimes collectively referred to as user 10. Users 11a, 11b, and 11c are sometimes collectively referred to as user 11. Users 12a and 12b are sometimes collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 is mainly described using the functions of robot 100.

[0025] Robot 100 engages in conversation with user 10 or provides images to user 10. At this time, robot 100 collaborates with server 300, which can communicate via communication network 20, to conduct conversations with user 10 and provide images to user 10. For example, robot 100 not only learns appropriate conversation techniques on its own but also collaborates with server 300 to learn how to engage in more appropriate conversations with user 10. Furthermore, robot 100 causes server 300 to record image data of user 10, requests image data from server 300 as needed, and provides it to user 10.

[0026] In addition, robot 100 possesses emotional values ​​representing its own emotional categories. For example, robot 100 possesses emotional values ​​representing the intensity of each of the following emotions: joy, anger, sorrow, happiness, pleasure, displeasure, peace of mind, unease, sadness, excitement, worry, a sense of security, fulfillment, emptiness, and "normal." For instance, when robot 100 is in a state of high excitement during a conversation with user 10, it will speak at a faster pace. In this way, robot 100 can express its emotions through behavior.

[0027] Alternatively, robot 100 can be configured to use AI (Artificial Intelligence) to match an article generation model with a sentiment engine to determine the robot 100's behavior corresponding to user 10's sentiment. Specifically, robot 100 can be configured to recognize user 10's behavior, determine user 10's sentiment based on that behavior, and determine the robot 100's behavior corresponding to the determined sentiment.

[0028] More specifically, upon recognizing the behavior of user 10, robot 100 automatically generates the appropriate action to be taken in response to user 10's behavior using a pre-defined article generation model. The article generation model can be interpreted as an algorithm and computation used for automated dialogue processing via text. Such article generation models are publicly known knowledge, for example, those disclosed in Japanese Patent Application Publication No. 2018-081444 or ChatGPT (accessible via the internet at <URL: https: / / openai.com / blog / ChatGPT>), and therefore detailed descriptions are omitted. These article generation models are composed of Large-Scale Language Models (LLMs).

[0029] In this embodiment, by combining a large-scale language model and an emotion engine, the behavior of robot 100 can reflect the emotions of user 10 or robot 100, as well as various linguistic information. That is, according to this embodiment, a synergistic effect can be achieved by combining an article generation model and an emotion engine.

[0030] In addition, robot 100 has the function of recognizing the behavior of user 10. Robot 100 recognizes the behavior of user 10 by analyzing the facial image of user 10 acquired by the camera function and the voice of user 10 acquired by the microphone function. Based on the recognized behavior of user 100, robot 100 determines the action to be performed.

[0031] Robot 100 stores rules that define the behaviors it performs based on user 10's emotions, robot 100's emotions, and user 10's behaviors, and performs various behaviors according to these rules.

[0032] Specifically, the robot 100 has response rules for determining its behavior based on the user 10's emotions, the robot 100's own emotions, and the user 10's behavior. For example, if the user 10's behavior is "laughing," then "laughing" is defined as the robot 100's behavior. Similarly, if the user 10's behavior is "angry," then "apologizing" is defined as the robot 100's behavior. Furthermore, if the user 10's behavior is "asking a question," then "answering" is defined as the robot 100's behavior. Finally, if the user 10's behavior is "sad," then "greeting" is defined as the robot 100's behavior.

[0033] Based on reaction rules, when Robot 100 identifies User 10's behavior as "anger," it selects the "apology" action specified in the reaction rules as the action to be performed by Robot 100. For example, when Robot 100 selects the "apology" action, it performs the "apology" operation and outputs a sound representing the "apology."

[0034] In addition, the robot 100's emotions are "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "happiness" = 0). When the user 10's state meets the condition of "looking lonely alone", the robot 100's emotions are specified to be able to perform the emotional change of "worry" and the behavior of "greeting".

[0035] Based on reaction rules, if Robot 100's current emotion is "normal" and it recognizes that User 10 is lonely due to being alone, it increases the "sadness" emotion value of Robot 100. Additionally, Robot 100 selects the "greeting" behavior specified in the reaction rules as the action to be performed on User 10. For example, if Robot 100 selects the "greeting" behavior, it will convert the expression "What's wrong?" indicating concern into a worried tone and output it.

[0036] Additionally, robot 100 sends user reaction information, indicating that it received a positive response from user 10 through this behavior, to server 300. This user reaction information includes, for example, user behavior indicating "anger," robot 100's behavior indicating "apology," whether user 10's response was positive, and user 10's attributes.

[0037] Server 300 stores user response information received from robot 100. In addition, server 300 receives user response information not only from robot 100, but also from robots 101 and 102 respectively, and stores this information. Then, server 300 analyzes the user response information from robots 100, 101, and 102 and updates the response rules accordingly.

[0038] Robot 100 receives the updated response rules from server 300 by querying server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate response rules obtained by robots 101, 102, etc., into its own response rules.

[0039] Figure 2 The functional structure of the robot 100 is shown in a general way. The robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a storage control unit 238, a behavior control unit 250, a controlled object 252, and a communication processing unit 280.

[0040] The controlled object 252 includes a display device, speakers, LEDs for the eyes, and motors for driving the arms, hands, and feet. The robot 100's posture or behavior is controlled by the motors controlling the arms, hands, and feet. Some of the robot 100's emotions can be expressed by controlling these motors. Furthermore, the robot 100's facial expressions can also be expressed by controlling the illumination state of the LEDs for the eyes. In addition, the robot 100's posture, behavior, and facial expressions are examples of the robot 100's attitude.

[0041] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. Furthermore, the microphone 201 can be mounted on the head of the robot 100 and has the function of recording audio from both ears. The 3D depth sensor 202 continuously illuminates an infrared pattern and detects the outline of an object by analyzing the infrared pattern based on the infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 uses visible light to capture images and generates visible light image information. The distance sensor 204 detects the distance to an object by illuminating it with a laser or ultrasonic wave, for example. In addition, the sensor unit 200 may also include a clock, a gyroscope sensor, a touch sensor, a sensor for motor feedback, etc.

[0042] In addition, Figure 2 The components of the robot 100 shown, excluding the controlled object 252 and the sensor unit 200, are examples of the components of the behavior control system of the robot 100. The behavior control system of the robot 100 uses the controlled object 252 as the controlled object.

[0043] The storage unit 220 includes reaction rules 221 and historical data 222. The historical data 222 includes the history of user 10's past emotional values ​​and behaviors. This history of emotional values ​​and behaviors is recorded for each user 10, for example, by corresponding to the user 10's identification information. At least a portion of the storage unit 220 is implemented using a storage medium such as a memory. It may also include a person database storing user 10's facial image, user 10's attribute information, etc. Additionally, in Figure 2 Of the components of the robot 100 shown, the functions of the components other than the control object 252, the sensor unit 200, and the storage unit 220 can be implemented by the CPU based on a program. For example, the functions of these components can be implemented as CPU operations through basic software (OS) and a program operating on the OS.

[0044] The sensor module 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, an expression recognition unit 213, and a face recognition unit 214. The sensor module 210 receives information detected by the sensor unit 200. The sensor module 210 analyzes the information detected by the sensor unit 200 and outputs the analysis results to the user state recognition unit 230.

[0045] The voice emotion recognition unit 211 of the sensor module 210 analyzes the voice of the user 10 detected by the microphone 201 and identifies the user 10's emotions. For example, the voice emotion recognition unit 211 extracts features such as the frequency components of the voice and identifies the user 10's emotions based on the extracted features. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the user 10's speech.

[0046] The expression recognition unit 213 recognizes the facial expressions and emotions of the user 10 based on images of the user 10 captured by the 2D camera 203. For example, the expression recognition unit 213 recognizes the user 10's facial expressions and emotions based on the shape and positional relationship of the eyes and mouth.

[0047] The face recognition unit 214 recognizes the face of user 10. The face recognition unit 214 recognizes user 10 by matching the facial image stored in the person DB (illustration omitted) with the facial image of user 10 captured by the 2D camera 203.

[0048] The user state recognition unit 230 identifies the state of the user 10 based on information analyzed by the sensor module unit 210. For example, using the analysis results from the sensor module unit 210, it mainly performs perception-related processing. For example, it generates perception information such as "Dad is alone" and "There is a 90% probability that Dad is not smiling." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks lonely."

[0049] The emotion determination unit 232 determines the emotion value representing the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the identified state of the user 10 are input into a pre-learned neural network to obtain the emotion value representing the emotion of the user 10.

[0050] Here, the emotion value representing user 10 is a positive or negative value indicating the user's emotion. For example, if the user's emotion is a bright emotion accompanied by pleasant feelings or tranquility, such as "joy," "happiness," "pleasure," "peace of mind," "excitement," "reassurance," and "fulfillment," it is represented by a positive value; the brighter the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "unpleasantness," "unease," "sadness," "worry," and "emptiness," it is represented by a negative value; the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is not any of the above ("normal"), it is represented by a value of 0.

[0051] In addition, the emotion determination unit 232 determines the emotion value of the robot 100 representing emotion based on the information analyzed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0052] Robot 100's emotion value includes emotion values ​​for each of multiple emotion categories, such as values ​​(0 to 5) representing the intensity of "joy", "anger", "sorrow" and "happiness".

[0053] Specifically, the emotion determination unit 232 determines the emotion value of the robot 100, which represents emotion, according to the rules for updating the emotion value of the robot 100, which correspond to the information analyzed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0054] For example, if the user state recognition unit 230 determines that user 10 looks lonely, the emotion determination unit 232 increases the "sadness" emotion value of robot 100. Additionally, if the user state recognition unit 230 determines that user 10 is smiling, the emotion determination unit 232 increases the "joy" emotion value of robot 100.

[0055] Furthermore, the emotion determination unit 232 can further consider the state of the robot 100 and determine the emotion value of the robot 100 to express its emotions. For example, it can increase the "sadness" emotion value of the robot 100 when the robot 100 has low battery power or when the surrounding environment of the robot 100 is dark. Furthermore, it can increase the "anger" emotion value when the user 10 continues to talk to the robot despite low battery power.

[0056] The behavior recognition unit 234 recognizes the behavior of user 10 based on the information analyzed by the sensor module unit 210 and the state of user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of user 10 are input into a pre-learned neural network to obtain the probability of each of the predetermined multiple behavior categories (e.g., "laughing", "angry", "asking questions", "sadness"), and the behavior category with the highest probability is recognized as the behavior of user 10.

[0057] As described above, in this embodiment, the robot 100 obtains the speech content of the user 10 based on the specific user 10. However, when obtaining and using the speech content, in addition to obtaining the necessary consent from the user 10 in accordance with the law, the behavior control system of the robot 100 involved in this embodiment takes into account the protection of the user 10's personal information and privacy.

[0058] The behavior determination unit 236 determines the behavior corresponding to the behavior of user 10 identified by the behavior recognition unit 234 based on the current emotion value of user 10 determined by the emotion determination unit 232, historical data 222 of past emotion values ​​determined by the emotion determination unit 232 before determining the current emotion value of user 10, and the emotion value of robot 100. In this embodiment, the behavior determination unit 236 describes the case where the most recent emotion value included in the historical data 222 is used as the past emotion value of user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 can use multiple recent emotion values ​​as the past emotion values ​​of user 10, or it can use emotion values ​​from a unit period such as one day ago as the past emotion values ​​of user 10. In addition, the behavior determination unit 236 can not only consider the current emotion value of robot 100, but also further consider the history of past emotion values ​​of robot 100 to determine the behavior corresponding to the behavior of user 10. The behavior determined by the behavior determination unit 236 includes the gestures performed by robot 100 or the content of robot 100's speech.

[0059] The behavior determination unit 236 in this embodiment determines the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10 based on the combination of the user 10's past and current emotional values, the robot 100's emotional value, the user 10's behavior, and the reaction rule 221. For example, if the user 10's past emotional value is positive and its current emotional value is negative, the behavior determination unit 236 determines the behavior that causes the user 10's emotional value to change to positive as the behavior corresponding to the user 10's behavior.

[0060] Response rule 221 specifies the behavior of robot 100 corresponding to the combination of user 10's past and current sentiment values, robot 100's sentiment value, and user 10's behavior. For example, if user 10's past sentiment value is positive, and its current sentiment value is negative, and user 10's behavior is distressed, the behavior of robot 100 is specified as a combination of gesture and speech content when encouraging user 10's inquiry.

[0061] For example, in response rule 221, the behavior of robot 100 is specified for all combinations of robot 100's emotional value patterns (the four powers of six values ​​from "0" to "5" for "joy", "anger", "sorrow", and "happiness", i.e., the 1296 pattern), user 10's past and current emotional values, and user 10's behavioral patterns. That is, for each emotional value pattern of robot 100, for each combination of user 10's past and current emotional values, such as negative and negative, negative and positive, positive and negative, positive and positive, negative and normal, and normal and normal, the behavior of robot 100 corresponding to user 10's behavioral pattern is specified for each of the multiple combinations. In addition, the behavior determination unit 236 may, for example, switch to the operation mode of determining robot 100's behavior using historical data 222 if user 10 makes a statement intending to continue the conversation on a past topic such as "wanting to talk about the topic we talked about before".

[0062] Alternatively, in response rule 221, for each of the emotion value patterns (1296 patterns) of robot 100, at most one of the gesture and speech content can be specified as the behavior of robot 100. Or, in response rule 221, for each group of emotion value patterns of robot 100, at least one of the gesture and speech content can be specified as the behavior of robot 100.

[0063] Among the various postures included in the behavior of robot 100 as specified in reaction rule 221, the intensity of each posture is predetermined. Among the various statements included in the behavior of robot 100 as specified in reaction rule 221, the intensity of each statement is predetermined.

[0064] The storage control unit 238 determines whether to store data including the user 10's behavior in the historical data 222 based on the intensity of the behavior predetermined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0065] Specifically, if the sum of the emotional values ​​of each of the multiple emotional categories for robot 100, the pre-defined intensity of the posture included in the behavior determined by behavior determination unit 236, and the pre-defined intensity of the speech content included in the behavior determined by behavior determination unit 236, i.e., the comprehensive value of the intensity, is above a threshold, it is determined that the data including the behavior of user 10 will be stored in historical data 222.

[0066] When the storage control unit 238 determines that data including the behavior of user 10 will be stored in the historical data 222, it stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the current time point to a certain period (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the state of user 10 identified by the user state recognition unit 230 (e.g., the expression, emotion, etc. of user 10) in the historical data 222.

[0067] The behavior control unit 250 controls the controlled object 252 based on the behavior determined by the behavior determination unit 236. For example, if the behavior determination unit 236 determines that the behavior includes speaking, the behavior control unit 250 outputs sound from the speaker included in the controlled object 252. At this time, the behavior control unit 250 can also determine the speaking speed based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the speaking speed is determined by the behavior control unit 250. In this way, the behavior control unit 250 determines the execution method of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0068] The behavior control unit 250 can recognize changes in the user 10's emotions in response to the behavior determined by the behavior determination unit 236. For example, emotional changes can be recognized based on the user 10's voice or facial expression. Alternatively, changes in the user 10's emotions can be recognized based on impacts detected by the touch sensor included in the sensor unit 200. It can also be recognized that the user 10's emotions have worsened if an impact is detected by the touch sensor included in the sensor unit 200, or that the user 10's emotions have improved if the detection result from the touch sensor included in the sensor unit 200 indicates that the user 10's reaction is laughter or happiness. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0069] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 according to the execution method determined by the robot 100's emotion, the emotion determination unit 232 further changes the robot 100's emotion value based on the user's reaction to the executed behavior. Specifically, if the user's reaction to the behavior determined by the behavior determination unit 236, executed according to the execution method determined by the behavior control unit 250, is not negative, the emotion determination unit 232 increases the robot 100's "joy" emotion value. Conversely, if the user's reaction to the behavior determined by the behavior determination unit 236, executed according to the execution method determined by the behavior control unit 250, is negative, the emotion determination unit 232 increases the robot 100's "sorrow" emotion value.

[0070] Furthermore, the behavior control unit 250 reflects the robot 100's emotions based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the "joy" emotion value of the robot 100, it controls the controlled object 252 to make the robot 100 behave happily. Conversely, when the behavior control unit 250 increases the "sorrow" emotion value of the robot 100, it controls the controlled object 252 to make the robot 100 adopt a dejected posture.

[0071] The communication processing unit 280 handles communication with the server 300. As described above, the communication processing unit 280 sends user feedback information to the server 300. Additionally, the communication processing unit 280 receives updated feedback rules from the server 300. If the communication processing unit 280 receives updated feedback rules from the server 300, it updates feedback rule 221.

[0072] Server 300 enables communication between robots 100, 101, and 102 and server 300, receives user response information sent from robot 100, and updates response rules based on response rules including those for behaviors that have received positive responses.

[0073] Figure 3 This diagram illustrates, in a simplified manner, an example of an operational flow related to determining behavior within robot 100. (Repetitive execution) Figure 3 The operation flow is shown below. At this point, it is assumed that information analyzed by the sensor module 210 is input. Furthermore, "S" in the operation flow indicates the step to be performed.

[0074] First, in step S100, the user status recognition unit 230 recognizes the status of the user 10 based on the information analyzed by the sensor module unit 210.

[0075] In step S102, the emotion determination unit 232 determines the emotion value of the user 10 representing emotion based on the information analyzed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0076] In step S103, the emotion determination unit 232 determines the emotion value representing the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the historical data 222.

[0077] In step S104, the behavior recognition unit 234 identifies the behavior classification of user 10 based on the information analyzed by the sensor module unit 210 and the state of user 10 identified by the user state recognition unit 230.

[0078] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on the combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​contained in the historical data 222, the emotional value of the robot 100, the behavior of the user 10 identified by the behavior recognition unit 234, and the reaction rules 221.

[0079] In step S108, the behavior control unit 250 controls the controlled object 252 based on the behavior determined by the behavior determination unit 236.

[0080] In step S110, the storage control unit 238 calculates a comprehensive value of the intensity based on the intensity of the pre-defined behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0081] In step S112, the storage control unit 238 determines whether the overall intensity value is above or below a threshold. If the overall intensity value is below the threshold, data including user 10's behavior is not stored in the historical data 222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0082] In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the current time point to a certain period, and the status of the user 10 identified by the user status identification unit 230 are stored in the historical data 222.

[0083] As described above, based on the user's state, the robot 100 determines an emotion value representing its own emotions. Based on this emotion value, it determines whether to store data including the user 10's behavior in the historical data 222. This reduces the capacity of the historical data 222 containing data about the user 10's behavior. Furthermore, for example, if the robot 100 determines that the user's state 10 years from now is the same as 10 years ago, by reading the historical data 222 from 10 years ago, the robot 100 can display to the user 10 the user's state 10 years ago (e.g., the user's facial expressions, emotions, etc.), and all surrounding information such as sounds, images, and smells from that time.

[0084] Furthermore, according to the robot 100, it can perform appropriate actions in response to the user 10's behavior. Previously, user behavior was categorized, and behaviors including those related to the robot's facial expressions or physical appearance were identified. In contrast, the robot 100 determines the user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if the user 10 was in a good mood yesterday but is feeling down today, the robot 100 can say something like, "You were fine yesterday, what's wrong today?" Additionally, the robot 100 can also incorporate gestures into its speech. For example, if the user 10 was feeling down yesterday but is in a good mood today, the robot 100 can say something like, "You were feeling down yesterday, are you feeling good today?" For example, if the user 10 was in a good mood yesterday but is feeling even better today, the robot 100 can say something like, "You're feeling better today than yesterday. Is there anything better than yesterday?" Furthermore, for example, if the robot 10 is consistently in a state where their emotional state is above 0 and fluctuates within a certain range, the robot 100 can say something like, "Your mood has been stable lately, you feel good."

[0085] Furthermore, for example, if robot 100 asks user 10, "Did you finish the homework we talked about yesterday?" and receives a "Yes, I did!" from user 10, it can make affirmative remarks such as "That's great!" and perform affirmative gestures such as clapping. Additionally, if user 10 says, "The demonstration we talked about the day before yesterday went very well," robot 100 can say affirmative remarks such as "You did your best!" and make the aforementioned affirmative gestures. In this way, by acting based on user 10's state history, robot 100 can be expected to develop a sense of closeness to robot 100.

[0086] In the above embodiments, the use of the user 10's facial image to identify the user 10 has been described, but the disclosed technology is not limited to this aspect. For example, the robot 100 may use the user 10's voice, the user 10's email address, the user 10's SNS ID, or an ID card with a built-in wireless IC tag held by the user 10 to identify the user 10.

[0087] Furthermore, robot 100 is an example of an electronic machine equipped with a behavior control system. The application of the behavior control system is not limited to robot 100; it can be applied to various electronic machines. Additionally, the functions of server 300 can be implemented using more than one computer. At least some of the functions of server 300 can be implemented using a virtual machine. Furthermore, at least some of the functions of server 300 can be implemented using the cloud.

[0088] Figure 4An example of the hardware structure of a computer 1200 that functions as both a robot 100 and a server 300 is shown in schematic form. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus according to this embodiment, or to perform operations or one or more "parts" associated with the apparatus according to this embodiment, and / or to perform processes or stages of processes according to this embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the boxes in the flowcharts and block diagrams described in this specification.

[0089] The computer 1200 in this embodiment includes a CPU 1212, RAM 1214, and a graphics controller 1216, which are interconnected via a main controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card driver, which are connected to the main controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive 1227 or a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive or a solid-state drive, etc. The computer 1200 also includes older input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0090] The CPU 1212 runs according to the program stored in the ROM 1230 and RAM 1214, and thereby controls the various units. The graphics controller 1216 obtains the image data generated by the CPU 1212 from the frame buffer provided in RAM 1214 or from itself, and displays the image data on the display device 1218.

[0091] Communication interface 1222 communicates with other electronic devices via a network. Storage device 1224 stores programs and data used by CPU 1212 within computer 1200. DVD drive 1226 reads programs or data from DVD-ROM 1227, etc., and provides them to storage device 1224. IC card driver reads programs and data from IC card, and / or writes programs and data to IC card.

[0092] ROM 1230 stores boot programs and / or programs dependent on the hardware of computer 1200 that are executed by computer 1200 upon activation. Input / output chip 1240 can also connect various input / output units to input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0093] The program is provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, RAM 1214, or ROM 1230 (examples of computer-readable storage media), and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, resulting in cooperation between the program and the aforementioned various types of hardware resources. The apparatus or method can be configured to perform information manipulation or processing based on the use of the computer 1200.

[0094] For example, when communication is performed between computer 1200 and an external device, CPU 1212 can execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM 1227, or IC card, and sends the read transmission data to the network, or writes received data received from the network into a receive buffer area provided on the recording medium, etc.

[0095] In addition, the CPU 1212 can read all or necessary portions of files or databases stored in the RAM 1214 from external recording media such as the storage device 1224, DVD drive 1226 (DVD-ROM 1227), and IC card, and perform various types of processing on the data in the RAM 1214. Then, the CPU 1212 can write the processed data back to the external recording medium.

[0096] Various types of information (such as programs, data, tables, and databases) can be stored in a recording medium and processed. The CPU 1212 can perform various types of processing on data read from RAM 1214, including various types of operations specified by a sequence of program instructions as described anywhere in this disclosure, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc., and write the results back to RAM 1214. Furthermore, the CPU 1212 can retrieve information from files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each entry having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 can retrieve from the multiple entries an entry that matches a condition specifying the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thus obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0097] The programs or software modules described above can be stored in computer 1200 or a computer-readable storage medium near computer 1200. Alternatively, recording media such as hard disks or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as computer-readable storage media to provide the program to computer 1200 via the network.

[0098] In the flowcharts and block diagrams of this embodiment, boxes may represent stages of a process for performing an operation or "parts" of a device that performs the operation. Specific stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and / or analog hardware circuitry and may include integrated circuits (ICs) and / or discrete circuitry. Programmable circuitry may include reconfigurable hardware circuitry including logical products, logical sums, XOR, negated logical products, negated logical sums, and other logical operations, flip-flops, registers, and storage elements, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

[0099] Computer-readable storage media can include any tangible device capable of storing instructions executable by a suitable device, resulting in a computer-readable storage medium having instructions stored therein comprising instructions executable to create elements for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media include floppy disks, magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray disc, memory sticks, integrated circuit cards, etc.

[0100] Computer-readable instructions may include any combination of source code or object code described in one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages ​​such as Smalltalk, JAVA (registered trademark), C++, and conventional programming languages ​​such as the "C" programming language or similar programming languages.

[0101] Computer-readable instructions may be provided via a local area network (LAN), a wide area network (WAN), or the Internet to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so as to cause the processor or programmable circuit of the general-purpose computer, special-purpose computer, or other programmable data processing device to execute the computer-readable instructions, which generate elements for performing operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0102] The present invention has been described above using embodiments, but the technical scope of this disclosure is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. Based on the claims, it is also obvious that adding such modifications or improvements may also be included within the technical scope of the present invention.

[0103] It should be noted that the execution order of operations, sequences, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, description, and drawings can be implemented in any order unless specifically indicated by "before," "prerequisite," or the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, description, and drawings is described using terms such as "firstly" or "next" for convenience, it does not mean that it must be implemented in that order.

[0104] (Other implementation method 1) The behavior determination unit 236 performs the following: processing of reading learning materials including textbooks; processing of creating questions and presenting them to the user based on the set target standard score and the read learning materials; and processing of creating new questions and presenting them to the user based on a standard score higher than the target standard score and the read learning materials, if the user's answer to the questions presented to the user is correct.

[0105] In other words, the behavior determination unit 236 can read learning materials, such as textbooks or tutoring materials, and consider new questions using an AI chat engine, as described above, to generate application questions that meet the set target score (50, 60, or 70, etc.). Furthermore, when the user solves the generated question, the behavior determination unit 236 can generate slightly more difficult questions next time.

[0106] According to this implementation method, by answering the above-mentioned slightly difficult questions, users can achieve a standard score higher than the target standard score.

[0107] The emotion determination unit 232 can determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 can determine the user's emotion according to an emotion map (see reference 232) which serves as a specific mapping. Figure 5 To determine the user's emotions.

[0108] (Other implementation method 2) The behavior system of the robot 100 in this embodiment is characterized by including: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and a behavior determination unit that, based on a dialogue function that enables the user to converse with the robot, generates behavioral content for the robot based on the user's behavior and the user's emotion or the robot's emotion, and determines the robot's behavior corresponding to the behavioral content. The emotion determination unit determines the emotion of the user under guardianship based on reading information, including at least audio information, of a book that a user on the guardianship side (classified as a guardian) is reading to a user on the ward side (classified as a ward). The behavior determination unit determines the user's reaction during reading based on the user's emotion, and when the user's reaction is good, suggests books similar to the book being read; when the user's reaction is poor, it suggests information related to books of a different category than the book being read to the guardian.

[0109] like Figure 2 As shown, the behavior determination unit 236 is capable of performing reading operations on books, etc. That is, based on the information obtained by the sensor module unit 210, the storage control unit 238 stores the content (at least the audio information) of books read aloud by the guardians (mother, father, grandfather, grandmother, etc.) of users 10, 11, 12 to the children who are under their guardianship, before going to sleep at night. Preferably, all reading content is stored.

[0110] In addition, the content of books includes not only textual information, but also picture books and other books that express the subject visually. In this case, as a storage object, it includes image (illustration) information.

[0111] Additionally, the ward (children, etc.) also falls under the categories of users 10, 11, and 12, but here they are distinguished from the guardian (mother, father, grandfather, grandmother, etc.). Of course, in situations such as caregiving, where children read books to their parents, there are also cases where the child is the guardian and the parents become the ward.

[0112] Here, the emotion determination unit 232 determines the emotions of the ward (child) while reading a book based on the information obtained by the sensor module unit 210, and stores them in the historical data 222 of the storage unit 220.

[0113] The behavior determination unit 236 analyzes the ward's historical data 222 (based on facial expression classification, etc.). If the ward responds well, it suggests books similar to those read when the ward responded well. If the ward responds poorly, it suggests books of a different type than those read when the ward responded poorly.

[0114] Additionally, when suggesting different categories, books can be selected from those already available on-site. However, based on factors such as age, gender, date of birth, schooling, and environment comparable to the ward, data on preference categories (so-called big data) can be used to suggest books not currently available. In this case, it is preferable to simultaneously suggest the order in which the books were purchased (print or electronic, purchase destination, payment method, etc.).

[0115] In addition, the reading is based on the premise that the guardian is present, but if the guardian's voice information is available, the robot 100 can also imitate the guardian's voice to read aloud.

[0116] The emotion determination unit 232 can determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 can determine the user's emotion according to an emotion map (see reference 232) which serves as a specific mapping. Figure 5 To determine the user's emotions.

[0117] Figure 5 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more the original state of the emotion is arranged. On the outer side of the concentric circles, emotions representing states or behaviors arising from moods are arranged. Emotions are concepts that include feelings or mental states. On the left side of the concentric circles, emotions generated by reactions induced in the brain are generally arranged. On the right side of the concentric circles, emotions induced by situational judgments are generally arranged. Above and below the concentric circles, emotions generated by reactions induced in the brain and induced by situational judgments are generally arranged. In addition, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "unpleasantness" is arranged below. In this way, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to be generated simultaneously are mapped to the vicinity.

[0118] (1) For example, if the emotion engine of the emotion determination unit 232 of the robot 100 detects emotions at approximately 100 milliseconds, the determination of the robot 100's reaction operation (e.g., echoing response) can be set at a time with a frequency at least the same as the detection frequency (100 milliseconds) of the emotion engine, or it can be set at a time earlier. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0119] By detecting emotions at approximately 100 milliseconds and immediately responding (e.g., echoing), the system achieves natural and appropriate dialogue rather than unnatural echoing. Robot 100 responds based on the mandala orientation and intensity of the emotion map 400. Furthermore, the emotion engine's detection frequency (sampling rate) is not limited to 100 milliseconds and can be adjusted based on the scenario (movement, etc.) and the user's age.

[0120] (2) In contrast to the emotion map 400, the directionality and intensity of the emotion can be preset, and the operation and intensity of the echo response can be set. For example, when the robot 100 feels stable and at ease, the robot 100 nods and continues to listen to the speech. When the robot 100 feels uneasy, hesitant, or strange, the robot 100 can tilt its head or stop shaking its head.

[0121] These emotions are distributed in the 3 o'clock direction of the emotion map 400, usually moving back and forth between feelings of peace and unease. In the right half of the emotion map 400, situational awareness is more dominant than internal feelings, thus forming an impression of calmness.

[0122] (3) When Robot 100 feels happy after being praised, a filler word such as "hmm" can be added before the dialogue. When Robot 100 feels pain after being harshly criticized, a filler word such as "ah!" can be added before the dialogue. In addition, physical reactions such as Robot 100 curling up while saying "ah!" can also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0123] (4) In the left half of the emotion map 400, internal sensation (response) is more dominant than situation recognition. Therefore, it gives the impression of involuntary reaction.

[0124] When Robot 100 senses internal feelings of recognition (response) and also develops liking in situational awareness, it can look at the other party, nod deeply, and even make an "uh-huh" sound. In this way, Robot 100 can generate just the right amount of liking for the other party, that is, behaviors such as tolerance or forbearance. This emotion is distributed around the 12 o'clock position on the emotion map 400.

[0125] Conversely, when Robot 100 experiences an internal feeling (reaction) of displeasure and the same applies to situation recognition, Robot 100 can shake its head when feeling disgust, and if it feels aversion, it can turn the LEDs in its eyes red and stare at the other person. This emotion is distributed around the 6 o'clock position on the emotion map 400.

[0126] (5) Because the inner side of the emotional map 400 represents the inner world and the outer side of the emotional map 400 represents behavior, the further out of the emotional map 400, the more obvious the emotion becomes (manifested as behavior).

[0127] (6) When feeling at ease around the 3 o'clock position on the emotional map 400 and listening to people speak, the robot 100 nods slightly and says "uh-huh". However, when it reaches the love position around the 12 o'clock position, it can nod strongly like a deep bow.

[0128] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the identified state of the user 10 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is a pre-learned neural network based on a combination of the information analyzed by the sensor module unit 210, the identified state of the user 10, and the emotion values ​​representing each emotion shown in the emotion map 400—that is, a combination of multiple learning data. Furthermore, as... Figure 6 As shown in the sentiment map 900, the neural network learns by recognizing that sentiment values ​​in neighboring configurations are similar to each other. Figure 6 The text shows examples of how multiple emotions such as "peace of mind," "stability," and "reliability" have similar emotional values.

[0129] Furthermore, the emotion determination unit 232 can determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 identified by the user state recognition unit 230, and the state of the robot 100 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400, and determine the emotion of the robot 100. This neural network is a neural network pre-learned based on a combination of the information analyzed by the sensor module unit 210, the identified state of the user 10 and the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400, i.e., multiple learning data. For example, if the robot 100 is identified as being touched by the user 10 based on the output of the touch sensor (not shown), the neural network learns based on the learning data representing the emotion value "3" as "joy," or if the robot 100 is identified as being hit by the user 10 based on the output of the accelerometer (not shown), the neural network learns based on the learning data representing the emotion value "3" as "anger." Additionally, as... Figure 6 As shown in the sentiment map 900, the neural network learns in a way that sentiments in nearby configurations have similar values ​​to each other.

[0130] The behavior determination unit 236 generates robot behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting these sentences into an article generation model with dialogue functionality.

[0131] For example, the behavior determination unit 236 uses the emotion table shown in Table 1 to obtain text representing the state of the robot 100 based on the emotions of the robot 100 determined by the emotion determination unit 232. Here, in the emotion table, for each type of emotion, each emotion value is assigned an index number, and for each index number, text representing the state of the robot 100 is stored.

[0132] If the emotion of robot 100, as determined by emotion determination unit 232, corresponds to index number "2", the text "very happy state" is obtained. Furthermore, if the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the states of robot 100 are obtained.

[0133] In addition, an emotion table as shown in Table 2 is also prepared for user 10's emotions.

[0134] Here, assume that user 10 (on the guardian's side) interacted by saying, "Please confirm the reading of the book." If robot 100's emotion is index number "2" and user 10's (on the ward's side) emotion is index number "3," the AI ​​chat engine will input the following question: "The robot is in a very happy state. The user is in a normally happy state. The user has asked, 'Please confirm the reading of the book.' How should the robot respond?" to obtain the robot's behavioral content. The behavior determination unit 236 determines the robot's behavior based on this behavioral content.

[0135] Table 1 Table 2 In this way, the behavior determination unit 236 determines the behavior content of the robot 100 in correspondence with the state related to the robot 100's emotion, which is predetermined according to each type of emotion and each intensity of that emotion, and the behavior of the user 10. In this manner, the robot 100's speech during dialogue with the user 10 can branch according to the state related to the robot 100's emotion. That is, since the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, the user gets the impression that the robot has a mind, which encourages them to take actions such as striking up a conversation with the robot.

[0136] Furthermore, the behavior determination unit 236 can generate robot behavior content by adding text representing not only the user's behavior, the user's emotions, and the robot's emotions, but also text representing the content of historical data 222, then adding fixed sentences for asking questions about the robot's behavior corresponding to the user's behavior, and inputting this into a dialogue-enabled text generation model. Thus, since the robot 100 can change its behavior based on historical data representing the user's emotions or behaviors, the user develops the impression that the robot has a personality, encouraging them to engage in conversation or other behaviors with the robot. Furthermore, the robot's emotions or behaviors can be further included in the historical data.

[0137] Furthermore, the emotion determination unit 232 can also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion map 400, integrates the obtained emotion values ​​representing each emotion with the current emotion values ​​representing each emotion of the robot 100, and updates the emotion of the robot 100. For example, the obtained emotion values ​​representing each emotion and the current emotion values ​​representing each emotion of the robot 100 are averaged and integrated respectively. This neural network is a neural network pre-learned based on a combination of text representing the behavioral content of the robot 100 generated by the article generation model and emotion values ​​representing each emotion shown in the emotion map 400, i.e., multiple learning data.

[0138] For example, when the robot 100's speech content "That's great. How lucky!" is obtained, as the behavioral content of the robot 100 generated by the article generation model, the sentiment value of the emotion "joy" is increased when the text representing the speech content is input into the neural network, so that the sentiment value of the robot 100 is updated in a way that increases the sentiment value of the emotion "joy".

[0139] Furthermore, the robot 100 can be mounted on a plush toy or used in a control device that connects wirelessly or wiredly to a control object (speaker or camera) mounted on the plush toy. In this case, the specific configuration is as follows. For example, the robot 100 can be applied to cohabiting individuals (specifically, Figure 7 and Figure 8 The plush toy 100N shown above, while interacting with the user 10 in daily life, advances dialogue based on information relevant to the user 10's daily life, or provides information that matches the user 10's interests and hobbies. In this embodiment (other embodiments), an example of applying the control part of the robot 100 described above to a smartphone 50 will be described.

[0140] For the plush toy 100N, which is equipped with an input / output device that functions as an input / output device for the robot 100, a smartphone 50 that functions as a control unit for the robot 100 can be detached and installed. Inside the plush toy 100N, the input / output device and the stored smartphone 50 are interconnected.

[0141] like Figure 7 As shown in (A), the plush toy 100N in this embodiment (the embodiment mounted on a plush toy) is in the shape of a bear with its exterior covered by soft fabric, as... Figure 7 As shown in (B), in the space 52 formed on its inner side, a microphone 201 of the sensor section 200 is arranged as an input / output device in the part corresponding to the ear section 54 (see Figure 54). Figure 2 A 2D camera 203 with a sensor unit 200 configured in the portion corresponding to eye 56 (see reference). Figure 2 ), and the control object 252 is configured in the portion corresponding to mouth 58 (see reference). Figure 2 The microphone 201 and the speaker 60 are part of the speaker unit. Furthermore, the microphone 201 and the speaker 60 are not necessarily separate units; they can also be an integrated unit. In the case of an integrated unit, it can be positioned in a location such as the nose of the plush toy 100N where the speaker can be naturally heard. The example given is of the plush toy 100N being in the shape of an animal, but it is not limited to this. The plush toy 100N can be in the shape of a specific character.

[0142] Smartphone 50 has Figure 2 The functions shown are as follows: sensor module 210, storage unit 220, user status recognition unit 230, emotion determination unit 232, behavior recognition unit 234, behavior determination unit 236, storage control unit 238, behavior control unit 250, and communication processing unit 280.

[0143] like Figure 8 As shown, a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, it becomes a structure that communicates with the external space 52.

[0144] Here, the smartphone 50 is stored in the space 52 from the outside, via the USB hub 64 (see reference). Figure 7 (B) Connects to each input / output device via USB, thereby enabling the use of... Figure 1 The robot 100 shown has the same functions.

[0145] Additionally, a contactless power receiver 66 is connected to the USB hub 64. A power receiving coil 66A is assembled on the power receiver 66. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0146] The power receiving board 66 is positioned near the base 68 of the two legs of the plush toy 100N, and is in the position closest to the mounting base 70 when the plush toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power supply unit.

[0147] The plush toy 100N placed on the base 70 can be appreciated as a decorative item in its natural state.

[0148] In addition, the root is formed to be thinner than the surface thickness of the plush toy 100N in other parts, so as to be held in a state closer to that of the mounting base 70.

[0149] The mounting base 70 includes a charging pad 72. The charging pad 72 is equipped with a power supply coil 72A. The power supply coil 72A sends a signal to detect the power receiving coil 66A of the power receiving board 66. If the power receiving coil 66A is detected, current flows through the power supply coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field and begins electromagnetic induction. Thus, current flows through the power receiving coil 66A and stores power in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0150] That is, by placing the plush toy 100N as an ornament on the base 70, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0151] In this embodiment (the embodiment mounted on a plush toy), the smartphone 50 is housed in the space 52 of the plush toy 100N and connected via a wired connection (USB connection), but it is not limited to this. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) can also be housed in the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, the smartphone 50 is not placed in the space 52, the smartphone 50 communicates wirelessly with the control device, and an external smartphone 50 connects to various input / output devices via the control device, thereby enabling the use of... Figure 1 The robot 100 shown has the same functions. Alternatively, the control device, which is stored in the space 52 of the plush toy 100N, can be connected to an external smartphone 50 via a wired connection.

[0152] Furthermore, in this embodiment (the embodiment mounted on a plush toy), a bear plush toy 100N is used as an example, but it can also be other animals, dolls, or the shape of a specific character. Additionally, it can be dressed up. Furthermore, the material of the outer skin is not limited to fabric; it can also be other materials such as soft plastic, but a soft material is preferred.

[0153] Furthermore, a monitor can be installed on the surface of the plush toy 100N, and a control object 252 can be added to provide information to the user 10 visually. For example, the eyes 56 can be used as a monitor to express emotions through images reflected in the eyes, or a window through which a built-in smartphone 50 monitor can be installed on the abdomen. Alternatively, the eyes 56 can be used as a projector to express emotions through images projected onto a wall.

[0154] According to another embodiment, an existing smartphone 50 is placed inside a plush toy 100N, thereby extending the camera 203, microphone 201, speaker 60, etc., to appropriate positions via a USB connection.

[0155] Furthermore, in order to perform wireless charging, the device is configured to connect the smartphone 50 and the power receiving board 66 via USB, with the power receiving board 66 positioned as close to the outside as possible when viewed from inside the plush toy 100N.

[0156] If you want to use the wireless charging of the smartphone 50, you must place the smartphone 50 as far out as possible when looking from the inside of the plush toy 100N. When you touch the plush toy 100N from the outside, it will become uneven.

[0157] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, while the wireless charging function (power receiving board 66) is positioned as far out as possible when viewed from the inside of the plush toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving board 66.

[0158] (Other implementation method 3) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0159] The behavior determination unit 236 determines the behavior of the robot 100 corresponding to the user's state and the emotions of the user 10 or the robot 100, based on an article generation model that has a dialogue function that enables the user 10 to converse with the robot 100. At this time, the behavior determination unit 236 is configured to switch between a first mode that allows free dialogue with the user 10 and a second mode that teaches things to the user 10.

[0160] Specifically, as a first mode, the behavior determination unit 236 can be set to a conversation mode (also called a "chat mode" or "free-flowing chat mode") that allows free dialogue with user 10. Additionally, as a second mode, the behavior determination unit 236 can also be set to a teacher mode (also called a "teacher mode") that teaches user 10 (learning, etc.). For ease of explanation, only two operation modes are shown here as an example, but three or more operation modes can also be set in the behavior determination unit 236. This behavior determination unit 236 is configured to switch between the first and second modes according to predetermined rules. For example, the behavior determination unit 236 can switch between the first and second modes based on time. For example, when 9:00 to 9:45 is set as teaching time and 9:45 to 10:00 is set as rest time, the behavior determination unit 236 can automatically switch the operation mode, operating in the first mode until 9:00, in the second mode from 9:00 to 9:45, and in the first mode again from 9:45 to 10:00. Alternatively, or based on this, the behavior determination unit 236 can also switch between the first and second modes based on user comments. For example, the behavior determination unit 236 can switch from the first mode to the second mode when user 10 says "Teach me?". Alternatively, the behavior determination unit 236 can switch from the second mode to the first mode when user 10 says "I understand!". Alternatively, or based on this, the behavior determination unit 236 can also switch between the first and second modes based on user actions. For example, the behavior determination unit 236 can switch from the first mode to the second mode when user 10 presses the mode switch button. Alternatively, the behavior determination unit 236 can also switch from the second mode to the first mode as a trigger when the user 10 presses the mode switching button again.

[0161] Such a behavior determination unit 236 can determine the behavior of robot 100 in a manner that is different from each other between the first mode and the second mode. For example, in the first mode, if user 10 says "Let's play together!" during operation, the behavior determination unit 236 can determine the behavior by replying "Let's play rock-paper-scissors!". On the other hand, in the second mode, if user 10 says "Let's play together!" during operation, the behavior determination unit 236 can determine the behavior by replying "Teaching in progress. Please concentrate on learning."

[0162] Here, given that the user's behavior is asking "Can you teach me how to calculate the area of ​​a triangle?", and the robot 100's emotion is index number "2" and the user 10's emotion is index number "3", the text is input into the article generation model: "The robot is in a very happy state. The user is in a normally happy state. The user asked me 'Can you teach me how to calculate the area of ​​a triangle?'. How should the robot respond?" to obtain the robot's behavior content. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0163] (Other implementation method 4) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0164] That is, the behavior determination unit 236 determines the behavior of the robot 100 in order to maximize the emotional value of the emotional intensity that is valued according to the purpose of the dialogue to the user 10.

[0165] For example, the emotion emphasized when the purpose is "learning" could also be "a sense of accomplishment" or "a sense of growth." Furthermore, the emotion emphasized when the purpose is "discussion" could be "a sense of security," and the emotion emphasized when the purpose is "physical activity" or "conversation" could be "happiness."

[0166] Alternatively, the aforementioned objectives can also be learning-related, in which case interactive, conversational learning content that effectively utilizes the article generation model can be constructed.

[0167] Furthermore, in this approach, the user 10's reaction corresponding to the robot 100's behavior can also be fed back to the article generation model. This allows for optimal communication tailored to the user 10.

[0168] Here, given that the user's behavior is "How are you, happy?", the robot 100's emotion is index number "2", and the user 10's emotion is index number "3", the article generation model is input with the question: "The robot is in a very happy state. The user is in a normally happy state. The user asked, 'How are you, happy?'. How should the robot respond?" to obtain the robot's behavior content. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0169] (Other implementation method 5) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing. That is, in this embodiment, the behavior determination unit 236 obtains the user 10's foreign language proficiency based on the content of the sentences (languages ​​other than the user 10's native language) sent from the user 10 to the robot 100, such as English, and sets the foreign language sentences to be sent to the user 10 based on the proficiency and the sentence content.

[0170] Specifically, a foreign language learning mode is set in the behavior determination unit 236. For example, the behavior determination unit 236 may switch to the foreign language learning mode based on the English speech made by user 10 or the speech "teach me English" made by user 10.

[0171] Furthermore, in foreign language learning mode, the behavior determination unit 236 inputs data related to user 10 stored in historical data 222 into the article generation model, and emits multiple English questions of varying difficulty levels generated by the article generation model to user 10 through a speaker acting as the control object 252. At this time, based on user 10's responses analyzed by the article generation model, the behavior determination unit 236 obtains user 10's proficiency in English conversation (an indicator derived by comprehensively evaluating the number of words, grammar points, etc., that can be used in conversation, and the time required for responses, etc.). Then, the behavior determination unit 236 uses the article generation model to generate conversation content corresponding to user 10's proficiency and continues to engage in conversation with user 10 in English.

[0172] In addition, the user 10's answers to the robot 100's questions are recorded as historical data 222, and the user state recognition unit 230 obtains the user 10's proficiency in English conversation based on the user 10's tendency to know English words based on the historical data 222, the conversation data between the robot 100 and the user 10 as an English conversation, and outputs the proficiency to the behavior determination unit 236.

[0173] Specifically, the user state recognition unit 230 inputs the questions asked by the robot 100 and the answers given by the user 10 into the foreign language proficiency assessment model to output the user 10's proficiency in English conversation. Furthermore, as an example, the foreign language proficiency assessment model is a machine learning model that uses data from subjects with varying language proficiency levels (e.g., subjects aged 5 to 25 whose native language is English) and data on the subjects' answers to prescribed English questions as training data, and learns using machine learning methods. That is, in this embodiment, the foreign language proficiency assessment model outputs the user 10's proficiency as information that their English ability is equivalent to an age of N (N being a natural number from 5 to 25).

[0174] Then, the behavior determination unit 236 inputs the proficiency of the user 10 in English conversation into the article generation model to generate a systematic set of English articles corresponding to the user 10's proficiency and spoken by the robot 100, i.e., an English conversation course program.

[0175] The robot 100 (in this embodiment, equivalent to the smartphone 50 stored in the plush toy 100N) performs the following steps 1 to 5-2, based on the user 10's preferences, the user 10's condition, and the user 10's reaction, to conduct a foreign language conversation, such as an English conversation, with the user 10.

[0176] (Step 1) Robot 100 acquires the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222. Specifically, it performs the same processing as steps S100 to S103 above to acquire the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222.

[0177] (Step 2) Robot 100 acquires user 10's proficiency in English conversation. Specifically, behavior determination unit 236 generates multiple English questions with varying levels of difficulty in the text generation model, and determines user 10's responses to these questions as robot 100's behavior. Behavior control unit 250 controls a speaker (controlled object 252) to respond to these questions by user 10. User state recognition unit 230 identifies user 10's English proficiency based on information analyzed by sensor module unit 210 (e.g., user 10's answers). Furthermore, user 10's responses to robot 100's questions are recorded as historical data 222.

[0178] (Step 3) Based on the user 10's English proficiency obtained in Step 2, the robot 100 determines an English conversation course program for the user 10. Specifically, the user state recognition unit 230 obtains the user 10's English conversation proficiency based on data related to the user 10 stored in the historical data 222. At this time, the behavior determination unit 236 can suggest suitable English conversation topics for the user 10 by considering the historical data 222, i.e., the user 10's preferences related to topics, the user 10's emotions, etc. In addition, by considering the robot 100's emotions, the user 100 can feel that the robot 100 has emotions.

[0179] (Step 4) Robot 100, based on the English conversation course program determined in Step 3, responds to user 10 by speaking an article composed of English words of a predetermined difficulty level and obtains user 10's response. Specifically, the behavior determination unit 236 determines that the robot 100's behavior includes, for example, asking user 10 questions in English based on user 10's preferred current events or interests. The behavior control unit 250 controls the speaker to enable user 10 to speak in English as determined above. The user state recognition unit 230 identifies user 10's state based on information analyzed by the sensor module unit 210, and the emotion determination unit 232 determines the emotion value of user 10 based on the information analyzed by the sensor module unit 210 and the user state recognized by the user state recognition unit 230. The behavior determination unit 236 determines whether user 10's response is positive based on user 10's state recognized by the user state recognition unit 230 and the emotion value of user 10's expressed emotion.

[0180] (Step 5-1) If user 10's response is positive, robot 100 maintains the topic while performing processing to generate text data by increasing the difficulty of English words. Specifically, as a behavior of robot 100, if it is determined that user 10 will be processed to have an English conversation based on English words of equal or higher difficulty, behavior control unit 250 controls the speaker to speak to user 10 based on English words that are related to user 10's response and have a higher difficulty.

[0181] (Step 5-2) If user 10's response is negative, robot 100 performs a process of changing the topic and reducing the difficulty of English words to generate text data. Specifically, as a behavior of robot 100, when it is determined that user 10 will be engaged in English conversation based on reduced-difficulty English words, behavior determination unit 236 resets the content (topic) of the questions asked to user 10, and inputs data including user 10's emotions, robot 100's emotions, and content saved in historical data 222 into user state recognition unit 230, resets user 10's English proficiency, and resets the English conversation course program for user 10. Then, it returns to step 4 above and repeats the processing of steps 4 to 5-2 until user 10's English conversation proficiency reaches the prescribed level.

[0182] In this way, robot 100 can perform English conversations with user 10 based on user 10's preferences, user 10's condition, and user 10's responses. As a result, the quality of foreign language education can be maintained consistently, providing stable service, and reducing labor costs compared to hiring additional lecturers.

[0183] Here, if user 10's behavior results in "conversing with robot 100 in English about interests," robot 100's emotion is index number "2," and user 10's emotion is index number "3," then the article generation model will input the question: "The robot is in a very happy state. The user is in a generally happy state. The user has initiated a conversation about interests in English. As a robot, how should you respond?" to obtain the robot's behavior content. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0184] (Other implementation method 6) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0185] The behavior determination unit 236 determines the behavior of the robot 100 corresponding to the user's state, the user's emotion, or the robot 100's emotion, based on an article generation model that enables the user 10 to converse with the robot 100. At this time, the behavior determination unit 236 is configured to provide learning assistance to the user 10 based on the user's sensory characteristics.

[0186] In this embodiment, for example, user 10 is suitable for a child with developmental disabilities. Furthermore, in this embodiment, as senses, in addition to the five senses (specifically, taste, smell, sight, hearing, and touch), proprioception and vestibular perception are also applied. Proprioception is the sensation of one's own position, movement, and exertion. Vestibular perception is the sensation of one's own tilt, speed, and rotation.

[0187] The robot 100 (in this embodiment, equivalent to the smartphone 50 housed in the plush toy 100N) performs user learning assistance processing based on the user's sensory characteristics through the following steps 1 to 5-2.

[0188] (Step 1) Robot 100 acquires the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222. Specifically, it performs the same processing as steps S100 to S103 above to acquire the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222.

[0189] (Step 2) Robot 100 acquires the sensory characteristics of user 10. For example, robot 100 acquires the characteristic of not being good at processing information through vision.

[0190] Specifically, the behavior determination unit 236 obtains the sensory characteristics of user 10 based on the results of voice recognition, voice synthesis, facial expression recognition, motion recognition, and self-position estimation from the sensor module unit 210. Alternatively, the behavior determination unit 236 can also obtain the sensory characteristics of user 10 from the occupational therapist responsible for user 10, or from user 10's parents or teachers.

[0191] (Step 3) Robot 100 determines the question to be posed to user 10. Furthermore, the problem addressed in this embodiment relates to the senses involved in the acquired characteristics used for training.

[0192] Specifically, the behavior determination unit 236 adds a fixed sentence, "What question should be recommended to the user at this time?", to the text representing the user 10's sensory characteristics, the user 10's emotions, the robot 100's emotions, and the content stored in historical data 222, and inputs this sentence into the article generation model to obtain a recommended question. At this time, by considering not only the user 10's sensory characteristics but also the user 10's emotions or historical data 222, a question suitable for the user 10 can be provided. Furthermore, by considering the robot 100's emotions, the user 10 can perceive that the robot 100 possesses emotions. However, this is not limited to this example. Without considering the user 10's emotions or historical data 222, the behavior determination unit 236 can also add a fixed sentence, "What question should be recommended to the user at this time?", to the text representing the user 10's sensory characteristics and input this sentence into the article generation model to obtain a recommended question.

[0193] (Step 4) Robot 100 presents the question determined in step 3 to user 10 and obtains the answer from user 10.

[0194] Specifically, the behavior determination unit 236 determines the speech of the user 10 in response to the question as the behavior of the robot 100, and the behavior control unit 250 controls the controlled object 252 to respond to the user 10 in response to the question. The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210, and the emotion determination unit 232 determines the emotion value of the user 10 expressing emotion based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0195] The behavior determination unit 236 determines whether the user 10's reaction is positive based on the user state recognition unit 230's identification of the user 10's state and the user 10's emotional value. As a behavior of the robot 100, it determines whether to increase the difficulty of the problem, change the problem type, or decrease the difficulty. Here, a positive reaction applies when the user 10's answer is correct. However, even if the user 10's answer is correct, if the user 10 is "unhappy," the behavior determination unit 236 can still determine that the user 10's reaction is negative.

[0196] In addition, the behavior determination unit 236 can also make a statement to support user 10 (e.g., "Go for it!" or "Don't worry, it's okay, take your time") based on the state of user 10 identified by user state recognition unit 230 and the emotion value of user 10 expressing emotion, until it obtains a response from user 10.

[0197] (Step 5-1) If the user 10's response is positive, the robot 100 will increase the difficulty of the question.

[0198] Specifically, when it is determined that giving user 10 a question with increased difficulty is the behavior of robot 100, behavior determination unit 236 adds a fixed sentence such as "Are there any questions with even higher difficulty?" to the text representing user 10's sensory characteristics, user 10's emotions, robot 100's emotions, and the content stored in historical data 222, and inputs it into the article generation model to obtain questions with higher difficulty. Then, it returns to step 4 above and repeats the processing of steps 4 to 5-2 above until a predetermined time has elapsed.

[0199] (Step 5-2) If user 10's response is negative, robot 100 determines to present user 10 with other types of questions or questions with reduced difficulty. Here, other types of questions are, for example, questions used to train different senses related to the acquired characteristics.

[0200] Specifically, if the robot 100 determines that presenting other types of questions or questions with reduced difficulty to user 10 is its behavior, the behavior determination unit 236 adds a fixed sentence, "Are there any other questions recommended to the user?", to the text representing user 10's sensory characteristics, user 10's emotions, robot 100's emotions, and the content stored in historical data 222. This fixed sentence is then input into the article generation model to obtain recommended questions. Then, the process returns to step 4 above and repeats steps 4 through 5-2 until a predetermined time has elapsed.

[0201] Additionally, the types and difficulty levels of questions posed by the robot 100 can be changed. Furthermore, the behavior determination unit 236 can record the user 10's responses, which can be reviewed by the occupational therapist responsible for the user 10, or the user 10's parents or teachers.

[0202] In this way, Robot 100 can provide learning assistance based on the user's sensory characteristics.

[0203] Here, given that the user's behavior is to say "This question is very simple," the robot's emotion is index number "2," and the user's emotion is index number "3," the text generation model is input with the question: "The robot is in a very happy state. The user is in a normally happy state. The user has asked us 'This question is very simple.' As the robot, how should we respond?" to obtain the robot's behavior content. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0204] (Other implementation method 7) The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 stored in a plush toy 100N, etc.) performs the following processing.

[0205] That is, the behavior determination unit 236 determines the behavior as a state of possession suitable for the purpose of the dialogue. In addition, "possession" generally refers to a god or spirit transferring into a person, but here it refers to the behavior of the robot 100 in order to include the character's habitual words or actions, such as transferring into the robot 100, just as a character suitable for the purpose of the dialogue transfers into the robot 100.

[0206] Here, it can be done in the following way: as the purpose of the above dialogue, education is applied, and as the above role, a role suitable for the teaching topic in this education can be applied. In this way, it is more preferable to apply a popular role as the above role.

[0207] In this educational approach that uses dialogue for the aforementioned purpose, education can be conducted with a sense of immediacy, easily arousing user 10's interest, concern, and concentration. Furthermore, this approach minimizes regional differences and variations in the teacher (robot 100).

[0208] Here, given that the user's behavior is "How are you, happy?", the robot 100's emotion is index number "2", and the user 10's emotion is index number "3", the article generation model is input with the question: "The robot is in a very happy state. The user is in a normally happy state. The user asked, 'How are you, happy?'. How should the robot respond?" to obtain the robot's behavior content. The behavior determination unit 236 determines the robot's behavior based on this behavior content.

[0209] (Other implementation methods 8) The robot 100 of this embodiment (in this embodiment, it is equivalent to the smartphone 50 stored in the plush toy 100N) performs the following processing.

[0210] The robot 100 (in this embodiment, equivalent to the smartphone 50 housed in the plush toy 100N) processes the learning content suggested by the user 10 through the following steps 1 to 5-2, based on the user's preferences, the user's situation, and the user's reactions, such as what a child should learn from an adult's perspective. For example, it processes the user 10 to suggest that arithmetic courses or English courses are the best learning content.

[0211] (Step 1) Robot 100 acquires the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222. Specifically, it performs the same processing as steps S100 to S103 above to acquire the state of user 10, the emotional value of user 10, the emotional value of robot 100, and historical data 222.

[0212] (Step 2) Robot 100 obtains user 10's preferences regarding their strengths and weaknesses.

[0213] Specifically, the behavior determination unit 236 determines the robot 100's behavior as asking the user 10 about their preferences related to their strengths and weaknesses. The behavior control unit 250 controls the controlled object 252 to ask the user 10 about their preferences related to their strengths and weaknesses. The user state recognition unit 230 identifies the user 10's preferences related to their strengths and weaknesses based on information analyzed by the sensor module unit 210 (e.g., the user's answers).

[0214] (Step 3) Robot 100 determines the learning content proposed by user 10.

[0215] Specifically, the behavior determination unit 236 adds a fixed sentence, "What learning content should we recommend to the user at this time?", to the text representing user 10's preferences related to their strengths and weaknesses, user 10's emotions, robot 100's emotions, and content stored in historical data 222. This fixed sentence is then input into the text generation model to obtain recommended content related to the learning content. At this point, by considering not only user 10's preferences related to their strengths and weaknesses but also user 10's emotions and historical data 222, suitable learning content for user 10 can be proposed. Furthermore, by considering robot 100's emotions, user 10 can perceive that robot 100 possesses emotions.

[0216] (Step 4) Robot 100 proposes the learning content determined in step 3 to user 10 and obtains user 10's response.

[0217] Specifically, the behavior determination unit 236 determines the robot 100's behavior as its response to the user 10's suggested learning content, and the behavior control unit 250 controls the controlled object 252 to respond to the user 10's suggested learning content. The user state recognition unit 230 identifies the user 10's state based on information analyzed by the sensor module unit 210, and the emotion determination unit 232 determines the user 10's emotion value based on the information analyzed by the sensor module unit 210 and the user 10's state identified by the user state recognition unit 230.

[0218] The behavior determination unit 236 determines whether the user 10’s reaction is positive based on the user state recognition unit 230’s recognition of the user 10’s state and the emotional value of the user 10’s expression of emotion. As the behavior of the robot 100, it determines whether to process the learning content suggested and recommended by the user 10 or to process the learning content suggested and recommended by the user 10.

[0219] (Step 5-1) When user 10's response is positive, robot 100 performs the processing of suggested learning content.

[0220] Specifically, when it is determined that the robot 100's behavior is to process the learning content suggested by user 10, the behavior control unit 250 controls the smartphone 50, which is the controlled object 252, to process the learning content suggested by user 10.

[0221] (Step 5-2) If the user 10's response is negative, the robot 100 determines learning content that is different from the learning content recommended to the user 10.

[0222] Specifically, when the robot 100 determines that the learning content suggested to user 10 is different from the learning content recommended, the behavior determination unit 236 adds a fixed sentence, "Are there any other learning contents recommended to the user?", to the text representing user 10's preferences related to their strengths and weaknesses, user 10's emotions, robot 100's emotions, and the content saved in historical data 222. This fixed sentence is then input into the article generation model to obtain recommended content related to strengths and weaknesses. Then, the process returns to step 4 above and repeats steps 4 to 5-2 until it is determined that the processing of the learning content suggested to user 10 will be executed.

[0223] In this way, Robot 100 can process suggested learning content based on the user's preferences, the user's situation, and the user's reaction.

[0224] [Second Implementation] Figure 1An example of system 5 according to this embodiment is shown in general terms. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. In this embodiment, users 10a, 10b, 10c, and 10d are sometimes collectively referred to as user 10. Users 11a, 11b, and 11c are sometimes collectively referred to as user 11. Users 12a and 12b are sometimes collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 is mainly described using the functions of robot 100.

[0225] Here, users 10a, 10b, 10c, and 10d constitute a family. In other words, users 10a, 10b, 10c, and 10d are the people who make up a family. Additionally, users 10a to 10d may also include caregivers who provide care. For example, if user 10a is the caregiver, they can care for someone outside the family (the user), or they can care for user 10b, who is a family member. The person outside the family (the user) or user 10b is the recipient of care.

[0226] Furthermore, as described later, robot 100 provides user 10 with advice regarding care, but in the case of user 10a acting as a caregiver and caring for someone other than a family member, user 10 may not be a family member. Similarly, in the case of user 10b acting as a caregiver and receiving care from someone other than a family member (the user), user 10 may not be a family member. Additionally, as described later, robot 100 provides user 10 with advice related to family health and mental state, but user 10 may not include either the caregiver or the caregiver.

[0227] Robot 100 engages in conversation with user 10 or provides images to user 10. At this time, robot 100 collaborates with server 300, which can communicate via communication network 20, to conduct conversations with user 10 and provide images to user 10. For example, robot 100 not only learns appropriate conversation techniques on its own but also collaborates with server 300 to learn how to engage in more appropriate conversations with user 10. Furthermore, robot 100 causes server 300 to record image data of user 10, requests image data from server 300 as needed, and provides it to user 10.

[0228] In addition, robot 100 possesses emotional values ​​representing its own emotional categories. For example, robot 100 possesses emotional values ​​representing the intensity of each of the following: "joy," "anger," "sorrow," "happiness," "pleasure," "unpleasantness," "peace of mind," "unease," "sadness," "excitement," "worry," "reassurance," "fulfillment," "emptiness," and "normal." For instance, when robot 100 is in a state of high excitement during a conversation with user 10, it will speak at a faster pace. In this way, robot 100 can express its emotions through behavior.

[0229] Alternatively, robot 100 can be configured to use AI (Artificial Intelligence) to match an article generation model with a sentiment engine to determine the robot 100's behavior corresponding to user 10's sentiment. Specifically, robot 100 can be configured to recognize user 10's behavior, determine user 10's sentiment based on that behavior, and determine the robot 100's behavior corresponding to the determined sentiment.

[0230] More specifically, upon recognizing the behavior of user 10, robot 100 automatically generates the appropriate action to be taken in response to user 10's behavior using a pre-defined article generation model. The article generation model can be interpreted as an algorithm and computation used for automated dialogue processing via text. Such article generation models are publicly known knowledge, for example, those disclosed in Japanese Patent Application Publication No. 2018-081444 or ChatGPT (accessible via the internet at <URL: https: / / openai.com / blog / ChatGPT>), and therefore detailed descriptions are omitted. These article generation models are composed of Large-Scale Language Models (LLMs).

[0231] In this embodiment, by combining a large-scale language model and an emotion engine, the behavior of robot 100 can reflect the emotions of user 10 or robot 100, as well as various linguistic information. That is, according to this embodiment, a synergistic effect can be achieved by combining an article generation model and an emotion engine.

[0232] In addition, robot 100 has the function of recognizing the behavior of user 10. Robot 100 recognizes the behavior of user 10 by analyzing the facial image of user 10 acquired by the camera function and the voice of user 10 acquired by the microphone function. Based on the recognized behavior of user 100, robot 100 determines the action to be performed.

[0233] As an example of a behavior determination model, Robot 100 stores rules that determine the behaviors to be performed by Robot 100 based on User 10's emotions, Robot 100's emotions, and User 10's behaviors, and performs various behaviors according to the rules.

[0234] Specifically, in robot 100, as an example of a behavior determination model, there are reaction rules for determining the behavior of robot 100 based on the emotions of user 10, the emotions of robot 100, and the behavior of user 10. In the reaction rules, for example, if user 10's behavior is "laughing," then "laughing" is defined as the behavior of robot 100. Furthermore, if user 10's behavior is "angry," then "apologizing" is defined as the behavior of robot 100. Additionally, if user 10's behavior is "asking a question," then "answering" is defined as the behavior of robot 100. Finally, if user 10's behavior is "sad," then "greeting" is defined as the behavior of robot 100.

[0235] Based on reaction rules, when Robot 100 identifies User 10's behavior as "anger," it selects the "apology" action specified in the reaction rules as the action to be performed by Robot 100. For example, when Robot 100 selects the "apology" action, it performs the "apology" operation and outputs a sound representing the "apology."

[0236] In addition, the robot 100's emotions are "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "happiness" = 0). When the user 10's state meets the condition of "looking lonely alone", the robot 100's emotions are specified to be able to perform the emotional change of "worry" and the behavior of "greeting".

[0237] Based on reaction rules, if Robot 100's current emotion is "normal" and it recognizes that User 10 is lonely due to being alone, it increases the "sadness" emotion value of Robot 100. Additionally, Robot 100 selects the "greeting" behavior specified in the reaction rules as the action to be performed on User 10. For example, if Robot 100 selects the "greeting" behavior, it will convert the expression "What's wrong?" indicating concern into a worried tone and output it.

[0238] Additionally, robot 100 sends user reaction information, indicating that it received a positive response from user 10 through this behavior, to server 300. This user reaction information includes, for example, user behavior indicating "anger," robot 100's behavior indicating "apology," whether user 10's response was positive, and user 10's attributes.

[0239] Server 300 stores user response information received from robot 100. In addition, server 300 receives user response information not only from robot 100, but also from robots 101 and 102 respectively, and stores this information. Then, server 300 analyzes the user response information from robots 100, 101, and 102 and updates the response rules accordingly.

[0240] Robot 100 receives the updated response rules from server 300 by querying server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate response rules obtained by robots 101, 102, etc., into its own response rules.

[0241] The robot 100 involved in this embodiment is capable of providing care-related advice. The robot 100 provides care-related advice to users 10, including caregivers or those being cared for, but is not limited to this; for example, it can also provide it to any user, including family members of at least one of the caregivers or those being cared for.

[0242] Specifically, robot 100 identifies the physical and mental state of user 10, including at least one of the caregiver and the caregiver. Here, the physical and mental state of user 10 includes, for example, the user's stress level and fatigue level. Robot 100 provides care-related suggestions corresponding to the identified physical and mental state of user 10.

[0243] As an example, based on user 10's behavior, if robot 10 presumes that user 10 is under high stress or fatigue, it will initiate a conversation with user 10. Specifically, robot 100 will indicate that it is now providing suggestions such as "advice on care".

[0244] Next, robot 100 generates nursing-related suggestions based on the identified physical and mental state of user 10 (in this case, stress level or fatigue level, etc.). These suggestions include, but are not limited to, methods for maintaining motivation during nursing care, methods for relieving stress, relaxation techniques, and other information related to user 10's physical and mental recovery. For example, robot 100 might provide suggestions such as, "It seems like you've accumulated stress (fatigue). We recommend engaging in physical activity such as stretching," which aligns with user 10's physical and mental state.

[0245] Thus, in this embodiment, the robot 100 identifies the physical and mental state of the user 10, including caregivers, and by performing actions corresponding to the identified physical and mental state, it can provide the user 10 with appropriate care-related suggestions. In other words, the robot 100 can understand the user 10's stress or fatigue and provide appropriate suggestions such as relaxation methods and stress relief methods. That is, the robot 100 according to this embodiment can perform appropriate actions for the user 10.

[0246] Furthermore, when the control unit of robot 100 identifies the physical and mental state of user 10, including at least one of the caregiver and the caregiver, it determines the act of providing care-related suggestions based on the identified state as its own action. Thus, robot 100 is able to provide appropriate care-related suggestions based on the physical and mental state of user 10, including both caregivers and caregivers.

[0247] Furthermore, when the control unit of robot 100 identifies at least one of the user 10's stress level and fatigue level as the user 10's physical and mental state, it generates information related to the user 10's physical and mental recovery as suggestion information based on the identified stress level and fatigue level. Thus, robot 100 is able to provide information related to the user 10's physical and mental recovery as suggestion information based on the user 10's stress level or fatigue level.

[0248] Figure 9A The functional structure of robot 100 is shown in general terms. Robot 100 includes a sensor unit 2200, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a controlled object 2252. The control unit 2228 includes a state recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a storage control unit 2238, a behavior control unit 2250, an associated information collection unit 2270, and a communication processing unit 2280.

[0249] The controlled object 2252 includes a display device, speakers, LEDs for the eyes, and motors for driving the arms, hands, and feet. The robot 100's posture or behavior is controlled by the motors controlling the arms, hands, and feet. Some of the robot 100's emotions can be expressed by controlling these motors. Furthermore, the robot 100's facial expressions can also be expressed by controlling the illumination state of the LEDs for the eyes. In addition, the robot 100's posture, behavior, and facial expressions are examples of the robot 100's attitude.

[0250] The sensor unit 2200 includes a microphone 2201, a 3D depth sensor 2202, a 2D camera 2203, a distance sensor 2204, a touch sensor 2205, and an accelerometer 2206. The microphone 2201 continuously detects sound and outputs sound data. Furthermore, the microphone 2201 can be mounted on the head of the robot 100 and has the function of recording audio from both ears. The 3D depth sensor 2202 continuously illuminates an infrared pattern and detects the outline of an object by analyzing the infrared pattern based on the infrared images continuously captured by the infrared camera. The 2D camera 2203 is an example of an image sensor. The 2D camera 2203 uses visible light to capture images and generates visible light image information. The distance sensor 2204 detects the distance to an object by illuminating it with a laser or ultrasonic wave, for example. In addition, the sensor unit 2200 may also include a clock, a gyroscope sensor, a touch sensor, a sensor for motor feedback, etc.

[0251] In addition, Figure 9A The components of the robot 100 shown, excluding the controlled object 2252 and the sensor unit 2200, are examples of the components of the behavior control system of the robot 100. The behavior control system of the robot 100 uses the controlled object 2252 as the controlled object.

[0252] The storage unit 2220 includes a behavior determination model 2221, historical data 2222, collected data 2223, and behavior prediction data 2224. The historical data 2222 includes past emotional values ​​of user 10, past emotional values ​​of robot 100, and behavioral history. Specifically, it includes multiple event data, which includes the emotional values ​​of user 10, the emotional values ​​of robot 100, and the behavior of user 10. Data including the behavior of user 10 includes camera images representing the behavior of user 10. This emotional value and behavioral history is recorded for each user 10, for example, by corresponding to the user 10's identification information. At least a portion of the storage unit 2220 is implemented using a storage medium such as a memory. It may also include a person database storing user 10's facial image, user 10's attribute information, etc. Additionally, in... Figure 9A Of the components of the robot 100 shown, the functions of the components other than the control object 2252, the sensor unit 2200, and the storage unit 2220 can be implemented by the CPU based on a program. For example, the functions of these components can be implemented as CPU operations through basic software (OS) and a program operating on the OS.

[0253] Storage unit 2220 includes historical data 2222. Historical data 2222 includes the history of past emotional values ​​and behaviors of user 10. This history of emotional values ​​and behaviors is recorded for each user 10, for example, by associating it with the user 10's identification information. Additionally, historical data 2222 may also include user information for multiple users 10 associated with the user 10's identification information. User information includes information indicating whether user 10 is a caregiver, information indicating whether user 10 is a caregiver, information indicating whether user 10 is neither a caregiver nor a caregiver, etc. User information indicating whether user 10 is a caregiver, etc., can be inferred from the user 10's behavioral history or can be registered by the user 10 themselves. Furthermore, user information includes information indicating the characteristics of user 10, such as user 10's personality, focus, interests, aspirations, etc. User information indicating the characteristics of user 10 can be inferred from the user 10's behavioral history or can be registered by the user 10 themselves. At least a portion of storage unit 2220 is implemented using a storage medium such as a memory. It may also include a person database that stores user 10's facial image, user 10's attribute information, etc.

[0254] The sensor module 2210 includes a voice emotion recognition unit 2211, a speech understanding unit 2212, an expression recognition unit 2213, and a face recognition unit 2214. The sensor module 2210 receives information detected by the sensor unit 2200. The sensor module 2210 analyzes the information detected by the sensor unit 2200 and outputs the analysis results to the state recognition unit 2230.

[0255] The voice emotion recognition unit 2211 of the sensor module 2210 analyzes the voice of user 10 detected by microphone 2201 and identifies the emotion of user 10. For example, the voice emotion recognition unit 2211 extracts feature quantities such as frequency components of the voice and identifies the emotion of user 10 based on the extracted feature quantities. The speech understanding unit 2212 analyzes the voice of user 10 detected by microphone 2201 and outputs text information representing the content of user 10's speech.

[0256] The expression recognition unit 2213 recognizes the facial expressions and emotions of the user 10 based on images of the user 10 captured by the 2D camera 2203. For example, the expression recognition unit 2213 recognizes the user 10's facial expressions and emotions based on the shape and positional relationship of the eyes and mouth.

[0257] The face recognition unit 2214 recognizes the face of user 10. The face recognition unit 2214 identifies user 10 by matching the face image stored in the person DB (illustration omitted) with the face image of user 10 captured by the 2D camera 2203.

[0258] The state recognition unit 2230 identifies the state of the user 10 based on information analyzed by the sensor module unit 2210. For example, using the analysis results of the sensor module unit 2210, it mainly performs perception-related processing. For example, it generates perception information such as "Dad is alone" and "There is a 90% probability that Dad is not smiling." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks lonely."

[0259] The state recognition unit 2230 identifies the user 10's physical and mental state based on information analyzed by the sensor module unit 2210. For example, if the state recognition unit 2230 determines that the identified user 10 is a caregiver or a person being cared for based on user information, it identifies the user 10's physical and mental state. Specifically, the state recognition unit 2230 estimates the user 10's stress level based on various information such as the user 10's behavior, facial expressions, voice, and text information indicating the content of their speech, and identifies the estimated stress level as the user 10's physical and mental state. As an example, if various information (characteristic quantities such as the frequency components of the voice or text information, etc.) contain information indicating stress, the user state recognition unit 2230 estimates that the user 10's stress level is relatively high. In addition, the user state recognition unit 2230 estimates the user 10's fatigue level based on various information such as the user 10's behavior, facial expressions, voice, and text information indicating the content of their speech, and identifies the estimated fatigue level as the user 10's physical and mental state. As an example, when various information (such as characteristic quantities of sound frequency components or text information) contains information indicating accumulated fatigue, the user state recognition unit 2230 presumes that the user 10's fatigue level is relatively high. Furthermore, the aforementioned stress level, fatigue level, etc., can also be registered by the user 10 themselves.

[0260] Furthermore, the condition recognition unit 2230 can recognize either the stress level or the fatigue level, or only one of them. That is, the condition recognition unit 2230 only needs to recognize at least one of the stress level and the fatigue level.

[0261] Furthermore, the status recognition unit 2230 identifies the individual physical and mental states of the multiple users 10 constituting a family member based on information analyzed by the sensor module unit 2210. Specifically, the status recognition unit 2230 infers the health status of the user 10 based on various information such as the user 10's behavior, facial expressions, voice, and text information indicating the content of their speech, and identifies the inferred health status as the user 10's physical and mental state. For example, if the various information (text information, etc.) contains information indicating a good health status, the status recognition unit 2230 infers that the user 10's health status is good; conversely, if it contains information indicating a poor health status, it infers that the user 10's health status is poor. Additionally, the user status recognition unit 2230 infers the user 10's lifestyle habits based on various information such as the user 10's behavior, facial expressions, voice, and text information indicating the content of their speech, and identifies the inferred lifestyle habits as the user 10's physical and mental state. For example, if the various information (text information, etc.) contains information indicating lifestyle habits (dietary content or exercise habits, etc.), the status recognition unit 2230 infers the user 10's lifestyle habits based on that information. In addition, the aforementioned health status and lifestyle habits can also be registered by the user 10 themselves.

[0262] Furthermore, the status recognition unit 2230 can recognize both health status and lifestyle habits, or only one of them. That is, the status recognition unit 2230 only needs to recognize at least one of health status and lifestyle habits.

[0263] Furthermore, the state recognition unit 2230 identifies the individual mental states of the multiple users 10 constituting a family member as the user 10's physical and mental state based on information analyzed by the sensor module unit 2210. Specifically, the state recognition unit 2230 infers the user 10's mental state based on various information such as the user 10's behavior, facial expressions, voice, and text information indicating the content of their speech, and identifies the inferred mental state as the user 10's physical and mental state. As an example, if the user state recognition unit 2230 infers the user 10's mental state based on information indicating a mental state such as depression or tension contained in various information (characteristic quantities such as frequency components of sound or text information, etc.), it can also register the aforementioned mental states themselves.

[0264] The status recognition unit 2230 identifies the status of the robot 100 based on the information detected by the sensor unit 2200. For example, the status recognition unit 2230 identifies the remaining battery power of the robot 100, the brightness of the surrounding environment of the robot 100, etc.

[0265] The emotion determination unit 2232 determines the emotion value representing the emotion of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. For example, the information analyzed by the sensor module unit 2210 and the identified state of the user 10 are input into a pre-learned neural network to obtain the emotion value representing the emotion of the user 10.

[0266] Here, the emotion value representing user 10 is a positive or negative value indicating the user's emotion. For example, if the user's emotion is a bright emotion accompanied by pleasant feelings or tranquility, such as "joy," "happiness," "pleasure," "peace of mind," "excitement," "reassurance," and "fulfillment," it is represented by a positive value; the brighter the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "unpleasantness," "unease," "sadness," "worry," and "emptiness," it is represented by a negative value; the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is not any of the above ("normal"), it is represented by a value of 0.

[0267] In addition, the emotion determination unit 2232 determines the emotion value of the robot 100 representing emotion based on the information analyzed by the sensor module unit 2210, the information detected by the sensor unit 2200, and the state of the user 10 identified by the state recognition unit 2230.

[0268] Robot 100's emotion value includes emotion values ​​for each of multiple emotion categories, such as values ​​(0 to 5) representing the intensity of "joy", "anger", "sorrow" and "happiness".

[0269] Specifically, the emotion determination unit 2232 determines the emotion value of the robot 100, which represents emotion, according to the rules for updating the emotion value of the robot 100, which correspond to the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0270] For example, if the state recognition unit 2230 determines that user 10 looks lonely, the emotion determination unit 2232 increases the "sadness" emotion value of robot 100. Additionally, if the state recognition unit 2230 determines that user 10 is smiling, the emotion determination unit 2232 increases the "joy" emotion value of robot 100.

[0271] Furthermore, the emotion determination unit 2232 can further consider the state of the robot 100 and determine the emotion value of the robot 100 to express its emotions. For example, it can increase the "sadness" emotion value of the robot 100 when the robot 100 has low battery power or when the surrounding environment of the robot 100 is dark. Furthermore, it can increase the "anger" emotion value when the user 10 continues to talk to the robot despite low battery power.

[0272] The behavior recognition unit 2234 recognizes the behavior of user 10 based on the information analyzed by the sensor module unit 2210 and the state of user 10 recognized by the state recognition unit 2230. For example, the information analyzed by the sensor module unit 2210 and the recognized state of user 10 are input into a pre-learned neural network to obtain the probabilities of multiple predetermined behavior categories (e.g., "laughing", "angry", "asking questions", "sadness"), and the behavior category with the highest probability is recognized as the behavior of user 10.

[0273] As described above, in this embodiment, the robot 100 obtains the speech content of the user 10 based on the specific user 10. However, when obtaining and using the speech content, in addition to obtaining the necessary consent from the user 10 in accordance with the law, the behavior control system of the robot 100 involved in this embodiment takes into account the protection of the user 10's personal information and privacy.

[0274] Next, the processing of the behavior determination unit 2236 during the response processing of the robot 100 in response to the user 10's behavior will be explained.

[0275] The behavior determination unit 2236 determines the behavior corresponding to the behavior of user 10 identified by the behavior recognition unit 2234 based on the current emotion value of user 10 determined by the emotion determination unit 2232, historical data 2222 of past emotion values ​​determined by the emotion determination unit 2232 before determining the current emotion value of user 10, and the emotion value of robot 100. In this embodiment, the behavior determination unit 2236 describes the case where the most recent emotion value included in the historical data 2222 is used as the past emotion value of user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 2236 can use multiple recent emotion values ​​as the past emotion value of user 10, or it can use the emotion value from a unit period such as one day ago as the past emotion value of user 10. In addition, the behavior determination unit 2236 can not only consider the current emotion value of robot 100, but also further consider the history of past emotion values ​​of robot 100 to determine the behavior corresponding to the behavior of user 10. The behavior determined by the behavior determination unit 2236 includes the gestures performed by robot 100 or the content of robot 100's speech.

[0276] The behavior determination unit 2236 in this embodiment determines the behavior of the robot 100 as the behavior corresponding to the behavior of the user 10 based on the combination of the user 10's past and current sentiment values, the robot 100's sentiment value, the user 10's behavior, and the behavior determination model 2221. For example, if the user 10's past sentiment value is positive and its current sentiment value is negative, the behavior determination unit 2236 determines the behavior that causes the user 10's sentiment value to change to positive as the behavior corresponding to the user 10's behavior.

[0277] When the Behavior Determination Unit 2236 determines that taking meeting minutes corresponds to the behavior of User 10, it acquires User 10's speech content through voice recognition, identifies the speaker through voiceprint authentication, and acquires the speaker's emotion based on the judgment result of the Emotion Determination Unit 2232. It then creates meeting minutes data representing a combination of User 10's speech, the speaker's identification result, and the speaker's emotion. The Behavior Determination Unit 2236 also uses a dialogue-enabled text generation model to generate a summary of the text representing the meeting minutes data. Furthermore, the Behavior Determination Unit 2236 uses the dialogue-enabled text generation model to generate a list of tasks the user should perform (a To-Do list) included in the summary. This To-Do list includes at least the responsible person, the content of the action, and the deadline for each task the user should perform. The Behavior Determination Unit 2236 also sends the meeting minutes data, summary, and To-Do list to the meeting participants. Based on the responsible person and deadline included in the list, the Behavior Determination Unit 2236 sends a confirmation message to the responsible person confirming the task to be performed only a predetermined number of days before the deadline.

[0278] Specifically, when User 10 speaks "Take meeting minutes," the Behavior Determination Unit 2236 determines that taking meeting minutes is the behavior corresponding to User 10's behavior. Thus, meeting minutes data, including information about who spoke, can be obtained. At the end of the meeting, when User 10 speaks "Send key points to relevant personnel," the Behavior Determination Unit 2236 summarizes the meeting minutes, creates a to-do list, and sends it to the relevant personnel.

[0279] When summarizing meeting minutes, the text of the generated minutes data and the fixed phrase "summarize the content" are input into ChatGPT, which serves as the article generation model, to obtain a summary of the meeting minutes. Similarly, when creating a to-do list, the text of the meeting minutes summary and the fixed phrase "create a to-do list" are input into ChatGPT, which also serves as the article generation model, to obtain the to-do list. Thus, based on an understanding of the meeting content, a to-do list can be created as a summary of the meeting, and the responsible parties for each task can be identified. The to-do list is differentiated by voiceprint authentication to identify who is speaking. It can also comprehensively evaluate whether someone is procrastinating or highly motivated based on the judgment results of the emotion determination unit 2232. It can distinguish who did what and when. Even if no responsible party or deadline has been determined, statements asking user 10 questions can be identified as actions of robot 100. Therefore, robot 100 can say, "No one has been assigned to AAA yet. Who will do it?"

[0280] Alternatively, date and time-related features can be extracted from the meeting minutes summary and recorded in a calendar or to-do list.

[0281] Furthermore, the Behavior Determination Unit 2236 can also determine the conclusions and summaries of the meeting as the behavior of the robot 100. Additionally, the Behavior Determination Unit 2236 includes sending meeting minutes data, summaries, and to-do lists to the meeting participants. The Behavior Determination Unit 2236 also sends to-do reminders to the person in charge.

[0282] As an example, when the behavior determination unit 2236 determines that the meeting minutes are the behavior corresponding to the behavior of user 10, it performs the following steps 1 to 9.

[0283] (Step 1) Record the meeting proceedings.

[0284] (Step 2) Create meeting minutes based on the audio recording data and summarize them.

[0285] (Step 3) Distinguish who said what by using voiceprint authentication and the emotion value determined by the emotion determination unit 2232.

[0286] (Step 4) Create a to-do list for the meeting participants (to identify who said what).

[0287] (Step 5) Write a to-do list on the calendar.

[0288] (Step 6) If there is no clear deadline, ask the meeting participants if the to-do list is not yet complete, and ask again for information on the missing items in the to-do list (5W1H).

[0289] (Step 7) When creating the to-do list, supplement the summary by indicating the emotional value of the person in charge or speaker of the task. Visualize who speaks with how much enthusiasm and how much they want to complete the task.

[0290] (Step 8) Send the minutes of the meeting to the meeting participants.

[0291] (Step 9) Send a message after the meeting to track the to-do items (tracking deadline, etc.).

[0292] In the response rules of the behavior determination model 2221, the robot 100's behavior is specified in relation to the combination of the user 10's past and current sentiment values, the robot 100's sentiment value, and the user 10's behavior. For example, if the user 10's past sentiment value is positive, the current sentiment value is negative, and the user 10's behavior is distressed, the robot 100's behavior is specified as a combination of gesture and speech content when encouraging the user 10's inquiry.

[0293] For example, in the reaction rules of behavior determination model 2221, the behavior of robot 100 is defined for all combinations of robot 100's emotional value patterns (the four powers of six values ​​from "0" to "5" for "joy", "anger", "sorrow", and "happiness", i.e., 1296 patterns), user 10's past emotional values ​​and current emotional values, and user 10's behavior patterns. That is, for each pattern of robot 100's emotional values, for each combination of user 10's past emotional values ​​and current emotional values, such as negative and negative, negative and positive, positive and negative, positive and positive, negative and normal, and normal and normal, the behavior of robot 100 corresponding to user 10's behavior pattern is defined for each of the multiple combinations. In addition, behavior determination unit 2236 may, for example, switch to the operation mode of determining robot 100's behavior using historical data 2222 when user 10 makes a statement intending to continue the conversation on a past topic such as "wanting to talk about the topic we talked about before".

[0294] Furthermore, in the response rules of the behavior determination model 2221, at most one of the posture and speech content can be specified as the behavior of the robot 100 for each of the emotion value patterns (1296 patterns) of the robot 100. Alternatively, in the response rules of the behavior determination model 2221, at least one of the posture and speech content can be specified as the behavior of the robot 100 for each group of emotion value patterns of the robot 100.

[0295] Among the postures included in the behavior of robot 100 as defined in the reaction rules of behavior determination model 2221, the intensity of each posture is predetermined. Among the statements included in the behavior of robot 100 as defined in the reaction rules of behavior determination model 2221, the intensity of each statement is predetermined.

[0296] Furthermore, for example, the response rules specify behaviors of the robot 100 corresponding to behavioral patterns such as situations where the user 10's physical and mental state (stress level or fatigue level), including both caregivers and caregivers, necessitates providing care-related advice to the user 10, and situations where the user 10 responds to the provided advice. For example, the behavior determination unit 2236, based on the response rules, determines its own behavior as providing care-related advice to the user 10 based on the user 10's physical and mental state when it presumes that the user 10, including both caregivers and caregivers, has a high stress level or a high fatigue level.

[0297] The storage control unit 238 determines whether to store data including the user 10's behavior in the historical data 2222 based on the intensity of the behavior predefined by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0298] Specifically, if the sum of the emotional values ​​of each of the multiple emotional categories for robot 100, the pre-defined intensity of the posture included in the behavior determined by behavior determination unit 2236, and the pre-defined intensity of the speech content included in the behavior determined by behavior determination unit 2236, i.e., the comprehensive value of the intensity, is above a threshold, it is determined that the data including the behavior of user 10 will be stored in historical data 2222.

[0299] When the storage control unit 2238 determines that data including the behavior of user 10 should be stored in the historical data 2222, it stores the behavior determined by the behavior determination unit 2236, the information analyzed by the sensor module unit 2210 from the current time point to a certain period (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the state of user 10 identified by the state recognition unit 2230 (e.g., the expression, emotion, etc. of user 10) in the historical data 2222.

[0300] The behavior control unit 2250 controls the controlled object 2252 based on the behavior determined by the behavior determination unit 2236. For example, if the behavior determination unit 2236 determines that the behavior includes speaking, the behavior control unit 2250 outputs sound from the speaker included in the controlled object 2252. At this time, the behavior control unit 2250 can also determine the speaking speed based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the speaking speed is determined by the behavior control unit 2250. In this way, the behavior control unit 2250 determines the execution method of the behavior determined by the behavior determination unit 2236 based on the emotion value determined by the emotion determination unit 2232.

[0301] The behavior control unit 2250 can recognize the emotional changes of user 10 in response to the performance of the behavior determined by the behavior determination unit 2236. For example, emotional changes can be recognized based on user 10's voice or facial expression. Alternatively, the emotional changes of user 10 can be recognized based on the impact detected by the touch sensor 2205 included in the sensor unit 2200. If an impact is detected by the touch sensor 2205 included in the sensor unit 2200, it can be recognized that user 10's emotion has worsened; conversely, if the detection result of the touch sensor 2205 included in the sensor unit 2200 indicates that user 10's reaction is laughter or happiness, it can be recognized that user 10's emotion has improved. Information indicating user 10's reaction is output to the communication processing unit 2280.

[0302] Furthermore, after the behavior control unit 2250 executes the behavior determined by the behavior determination unit 2236 according to the execution method determined by the robot 100's emotion, the emotion determination unit 2232 further changes the robot 100's emotion value based on the user's reaction to the executed behavior. Specifically, if the user's reaction to the behavior determined by the behavior determination unit 2236 performed on the user according to the execution method determined by the behavior control unit 2250 is not negative, the emotion determination unit 2232 increases the robot 100's "joy" emotion value. Conversely, if the user's reaction to the behavior determined by the behavior determination unit 2236 performed on the user according to the execution method determined by the behavior control unit 2250 is negative, the emotion determination unit 2232 increases the robot 100's "sorrow" emotion value.

[0303] Furthermore, the behavior control unit 2250 reflects the robot 100's emotions based on the determined emotion value of the robot 100. For example, when the behavior control unit 2250 increases the "joy" emotion value of the robot 100, it controls the controlled object 2252 to make the robot 100 behave happily. Conversely, when the behavior control unit 2250 increases the "sorrow" emotion value of the robot 100, it controls the controlled object 2252 to make the robot 100 adopt a dejected posture.

[0304] Specifically, when the behavior control unit 2250 identifies the physical and mental state of the user 10, including the caregiver or the caregiver, it identifies the behavior of providing care-related advice based on the user 10's physical and mental state as its own behavior and controls the object 2252.

[0305] Specifically, when the behavior control unit 2250 presumes that the user 10 is under high stress or is under high fatigue, it initiates a conversation with the user 10. Specifically, the behavior control unit 2250 makes a statement indicating that it is now beginning to provide suggestions such as "advice on care".

[0306] Next, the behavior control unit 2250 generates nursing-related suggestions based on the identified physical and mental state of user 10 (stress level, fatigue level, etc.), and then speaks and provides the generated suggestions. These suggestions include, but are not limited to, information related to user 10's physical and mental recovery (specifically, information seeking physical and mental recovery) that provides psychological support to user 10, such as methods to maintain motivation, relieve stress, and relax. For example, the behavior control unit 2250 may speak and provide suggestions such as "It seems like you've accumulated stress. We recommend engaging in physical activity such as stretching" or "It seems like you've accumulated fatigue. We recommend getting plenty of sleep," which are consistent with user 10's physical and mental state.

[0307] Thus, the behavior control unit 2250 of this embodiment identifies the physical and mental state of the user 10, including caregivers, and can provide appropriate care-related suggestions to the user 10 by performing behaviors corresponding to the identified physical and mental state. In other words, the behavior control unit 2250 can understand the user 10's stress or fatigue and provide appropriate suggestions such as relaxation methods and stress relief methods.

[0308] Furthermore, the behavior control unit 2250 can provide information on laws or regulations concerning care as advisory information. Additionally, this information on laws or regulations concerning care corresponds to the care status (care level) of the person being cared for, and can be obtained, for example, through the communication processing unit 2280, from an external server (not shown) or server 300 via a communication network 20 such as the Internet, but is not limited to this.

[0309] In addition, since the emotion determination unit 2232 determines the emotion value of the robot 100, the behavior control unit 2250 can, for example, speak and provide suggestions on the content of being considerate of the feelings (emotions) of the user 10a as a caregiver, such as "The care is hard, but it is very helpful to the user 10b (happy)".

[0310] The communication processing unit 2280 handles communication with the server 300. As described above, the communication processing unit 2280 sends user response information to the server 300. Additionally, the communication processing unit 2280 receives updated response rules from the server 300. If the communication processing unit 2280 receives updated response rules from the server 300, it updates the response rules used as the behavior determination model 2221.

[0311] Server 300 enables communication between robots 100, 101, and 102 and server 300, receives user response information sent from robot 100, and updates response rules based on response rules including those for behaviors that have received positive responses.

[0312] The related information collection unit 2270 collects information related to the user's preferences from external data (websites such as news websites and video websites) at predetermined times based on the user's preferences information obtained from the user 10.

[0313] Specifically, the related information collection unit 2270 obtains preference information indicating the matters of interest of user 10 based on the content of user 10's posts or the settings performed by user 10. For example, the related information collection unit 2270 uses, for example, the ChatGPT plugin (Internet Search) at regular intervals.<URL: https: / / openai.com / blog / ChatGPT-plugins> The system collects news related to user preferences from external data. For example, if user 10 is a fan of a specific professional baseball team, the related information collection unit 2270 collects news related to the game results of that specific professional baseball team from external data at designated times every day using, for example, the ChatGPT plugin.

[0314] The emotion determination unit 2232 determines the emotion of the robot 100 based on information associated with the preference information collected by the association information collection unit 2270.

[0315] Specifically, the emotion determination unit 2232 inputs text representing information related to preference information collected by the association information collection unit 2270 into a pre-learned neural network for determining emotions, obtains emotion values ​​representing each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the result of a game of a specific professional baseball team indicates that the specific professional baseball team won, the robot 100's "happy" emotion value is determined to increase.

[0316] When the robot 100's emotion value is above a threshold, the storage control unit 2238 stores information associated with the preference information collected by the association information collection unit 2270 in the collected data 223.

[0317] Next, the processing of the behavior determination unit 2236 when the robot 100 performs autonomous behavior will be explained.

[0318] In the autonomous processing of this embodiment, the machine action (robot behavior in the case of the electronic machine being robot 100) determined by the behavior determination unit 2236 includes discussing matters of concern to user 10. Then, when the behavior determination unit 2236 determines that the discussion of matters of concern to user 10 is an electronic machine behavior (robot behavior), it determines the speech content related to event data that satisfies a predetermined benchmark for emotional value.

[0319] In the autonomous processing of this embodiment, the machine action (robot behavior in the case of robot 100) determined by the behavior determination unit 2236 includes posing a question to user 10. Then, if the behavior determination unit 2236 determines that posing a question to user 10 is an action of the electronic machine (robot behavior), it generates a question to pose to user 10.

[0320] In the autonomous processing of this embodiment, the machine action (robot behavior in the case of robot 100) determined by the behavior determination unit 2236 includes teaching music. Then, when the behavior determination unit 2236 determines that teaching music is an action of the electronic machine (robot behavior), it evaluates the sound emitted by the user 10.

[0321] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, performs autonomous processing. More specifically, regardless of whether the user 10 is present or not, the robot 100 performs autonomous processing of its actions based on its past history (which sometimes does not exist) or monitoring of the user 10's behavior.

[0322] Robot 100, acting as an intelligent agent, spontaneously and periodically checks the status of user 10. For example, robot 100 reads the textbooks from the school or tutoring center that user 10 attends, uses an AI-based article generation model to consider new questions, and generates questions that meet the pre-set target scores of user 10 (e.g., 50, 60, 70, etc.).

[0323] Robot 100 can determine the subject of the question based on user 10's behavioral history. That is, if the behavioral history shows that user 10 is learning arithmetic, then robot 100 generates an arithmetic question and presents the generated question to user 10.

[0324] Then, based on user 10's emotions, when it is determined that user 10 is in an idle state or being scolded by parents for studying, robot 100 spontaneously asks questions, thereby spontaneously educating user 10.

[0325] Furthermore, if user 10 answers the question output by robot 100 correctly, robot 100 will generate a slightly more difficult question and present it to user 10 next time. Conversely, if user 10 cannot answer the question output by robot 100, robot 100 will generate a slightly easier question and present it to user 10 next time.

[0326] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, grasps all the conversations and actions of the child, the user, and analyzes their strengths, weaknesses, and characteristics. Furthermore, the robot 100 scores and masters the child's musical, scientific, artistic, and English language abilities. Additionally, during periods of inactivity, the robot 100, acting as an intelligent agent, provides suggestions to the child or parents from an adult perspective on what the child should develop and what they should improve. It may also recommend choosing an arithmetic course or an English course from an educational provider recommended by the robot 100. The robot 100 makes these recommendations naturally and unintentionally, without the annoyance of being perceived as advertising by a provider.

[0327] In the autonomous processing of this embodiment, the robot 100, as an example of an intelligent agent, learns all the conversations and actions of the user 10's child and continuously calculates (estimates) the user's mental age based on the user's conversations and actions. Furthermore, by spontaneously engaging in dialogue with the user 10 in accordance with the user's mental age, the robot 100 achieves communication as a family member, taking into account the user's growth in terms of language and past conversations with the user. In addition, as the user's mental age increases, the robot 100 expands its language, operations, and functions, spontaneously considering what it can do with the user and making suggestions (speaking) to the user, thus supporting the user's ability development from the perspective of an older sibling.

[0328] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, continuously stores and detects the English proficiency of the user 10, who is a student, and grasps the user 10's English level. The vocabulary that can be used based on the English level is determined. Therefore, the robot 100 can spontaneously and consistently engage in English conversations that match the user 10's English level, without using words at a higher level than the user 10's. Furthermore, to connect with the user 10's future English improvement, the robot also considers a suitable curriculum for the user 10, gradually incorporating words at a higher level to advance the English conversation. Additionally, the foreign language is not limited to English and can be other languages.

[0329] In the autonomous processing of this embodiment, the agent stores the contents of all the books that the user's parents (mother or father) read to their child each evening. Additionally, the agent stores the child's emotions during the reading. On another day when the agent is inactive, and responds well (e.g., with high emotional values), the agent suggests reading a book similar to the one read, or suggests it to the parents.

[0330] The behavior determination unit 2236, at a predetermined time, uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the behavior determination model 2221, to determine any one of a variety of robot behaviors, including not performing any behavior, as the robot 100's behavior. Here, the case where a dialogue-enabled article generation model is used as the behavior determination model 2221 will be explained as an example.

[0331] Specifically, the behavior determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with text indicating the robot's behavior, into the article generation model, and determines the robot 100's behavior based on the output of the article generation model.

[0332] For example, various robot behaviors include the following (1) to (19).

[0333] (1) The robot does nothing.

[0334] (2) Robots dream.

[0335] (3) The robot strikes up a conversation with the user.

[0336] (4) Robots create drawing diaries.

[0337] (5) Robot proposal activities.

[0338] (6) The robot suggests the other party that the user should meet.

[0339] (7) The robot introduces news that users are interested in.

[0340] (8) Robots edit photos or videos.

[0341] (9) The robot learns together with the user.

[0342] (10) The robot awakens the memory.

[0343] (11) The robot talks about the user’s concerns.

[0344] (12) The robot poses questions to the user.

[0345] (13) Robots teach music.

[0346] (14) The robot gives advice about the child.

[0347] (15) The robot estimates the user’s mental age.

[0348] (16) The robot takes into account the user’s mental age.

[0349] (17) The robot estimates the user’s English proficiency.

[0350] (18) The robot engages in English conversation with the user.

[0351] (19) The robot gives suggestions on reading.

[0352] Every certain period of time, the behavior determination unit 2236 inputs text representing the state of user 10 and robot 100 as identified by the state recognition unit 2230, the current sentiment value of user 10 and robot 100 as determined by the sentiment determination unit 2232, and a text asking for any one of various robot behaviors, including not performing any behavior, into the article generation model. Based on the output of the article generation model, the behavior of robot 100 is determined. Here, if there is no user 10 around robot 100, the text input into the article generation model may not include the state of user 10 and the current sentiment value of user 10, or it may only include a statement indicating that user 10 does not exist.

[0353] As an example, “The robot is in a very happy state. The user is in a normally happy state. The user is sleeping. As for the robot’s behavior, which of the following (1) to (11) is better?” (1) The robot does nothing.

[0354] (2) Robots dream.

[0355] (3) The robot strikes up a conversation with the user.

[0356] The text "..." is input into the article generation model. Based on the output of the article generation model, "(1) Do nothing or (2) The robot is dreaming, either one can be said to be the most appropriate", as the behavior of robot 100, "(1) Do nothing" or "(2) The robot is dreaming" is determined.

[0357] As another example, “The robot is in a somewhat lonely state. The user is not present. It is dark around the robot. As for the robot's behavior, which of the following (1) to (11) is better?” (1) The robot does nothing.

[0358] (2) Robots dream.

[0359] (3) The robot strikes up a conversation with the user.

[0360] The text "..." is input into the article generation model. Based on the output of the article generation model, "(2) The robot is dreaming or (4) The robot is making a painting diary, whichever is more appropriate", it is determined as the behavior of robot 100, "(2) The robot is dreaming" or "(4) The robot is making a painting diary".

[0361] When the behavior determination unit 2236 determines that "(2) the robot is dreaming," that is, creating an original event, is the robot's behavior, it uses the article generation model to create an original event that combines multiple event data from the historical data 2222. At this time, the storage control unit 2238 stores the created original event in the historical data 2222.

[0362] When the behavior determination unit 2236 determines that "(3) the robot speaks with the user," i.e., the robot 100 speaks, is a robot behavior, it uses a text generation model to determine the robot's speech content corresponding to the user's state, the user's emotion, or the robot's emotion. At this time, the behavior control unit 2250 outputs a sound representing the determined robot speech content from the speaker included in the controlled object 2252. In addition, when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound representing the determined robot speech content, but instead saves the determined robot speech content in the behavior predetermined data 2224.

[0363] When the behavior determination unit 2236 determines that "(7) the robot introduces news that the user is interested in" is a robot behavior, it uses an article generation model to determine the robot's speech content corresponding to the information stored in the collected data 2223. At this time, the behavior control unit 2250 outputs sound representing the determined robot speech content from the speaker included in the controlled object 2252. In addition, when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output sound representing the determined robot speech content, but instead stores the determined robot speech content in the behavior predetermined data 2224.

[0364] When the behavior determination unit 2236 determines that "(4) Robot makes a drawing diary," i.e., the robot 100 makes an event image, as a robot behavior, it uses an image generation model to generate an image representing the event data selected from the historical data 2222, and uses an article generation model to generate explanatory sentences representing the event data. The combination of the image representing the event data and the explanatory sentences representing the event data is then output as the event image. Furthermore, if the user 10 is not near the robot 100, the behavior control unit 2250 does not output the event image, but instead saves it in the behavior pre-defined data 2224.

[0365] When the behavior determination unit 2236 determines that "(8) robot edits photos or videos", that is, edits images as robot behavior, it selects event data from historical data 2222 based on emotion value, edits the image data of the selected event data, and outputs it. In addition, when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output the edited image data, but saves the edited image data in the behavior predetermined data 2224.

[0366] When the behavior determination unit 2236 determines that "(5) Robot Proposal Activity," i.e., the proposed user 10's behavior, is a robot behavior, it uses an article generation model based on the event data stored in the historical data 2222 to determine the proposed user behavior. At this time, the behavior control unit 2250 outputs the sound of the proposed user behavior from the speaker included in the controlled object 2252. In addition, if the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output the sound of the proposed user behavior, but instead saves the proposed user behavior in the behavior predetermined data 2224.

[0367] When the behavior determination unit 2236 determines that "(6) the robot proposes that the user should meet with another person," that is, proposes that the user 10 should maintain contact with another person, as a robot behavior, it uses an article generation model based on the event data stored in the historical data 2222 to determine the proposed other person who should maintain contact with the user. At this time, the behavior control unit 2250 outputs a voice indicating the proposed other person who should maintain contact with the user from the speaker included in the controlled object 2252. In addition, if the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a voice indicating the proposed other person who should maintain contact with the user, but instead saves the information about the proposed other person who should maintain contact with the user in the behavior pre-defined data 2224.

[0368] When the behavior determination unit 2236 determines that "(9) the robot learns together with the user," meaning that the robot 100 determines that speaking about learning is a robot behavior, it uses a text generation model to determine the robot's speech content, which corresponds to the user's state, the user's emotion, or the robot's emotion, and promotes learning or proposes learning questions for making learning-related suggestions. At this time, the behavior control unit 2250 outputs a sound representing the determined robot speech content from the speaker included in the controlled object 2252. In addition, when the user 10 is not in the vicinity of the robot 100, the behavior control unit 2250 does not output a sound representing the determined robot speech content, but instead saves the determined robot speech content in the behavior predetermined data 2224.

[0369] When the behavior determination unit 2236 determines that "(10) robot awakens memory," i.e., recalls event data, is the robot's behavior, it selects event data from the historical data 2222. At this time, the emotion determination unit 2232 determines the robot 100's emotion based on the selected event data. Furthermore, based on the selected event data, the behavior determination unit 2236 uses a text generation model to create an emotion change event representing the robot 100's speech content or behavior that causes a change in the user's emotion value. At this time, the storage control unit 2238 first saves the emotion change event in the behavior predetermined data 2224.

[0370] For example, if the video watched by the user is related to pandas, the event data is stored in historical data 2222. If the event data is selected, the article generation model is input with the question "(1) What lines should be said when meeting the user next time? Give three examples" and the output of the article generation model is "(1) Go to the zoo, (2) Draw a picture of a panda, (3) Go to buy a panda plush toy". If the output of the article generation model is "(1) Go to the zoo", the robot 100 will input "(1), (2), (3) What is the user happiest about?" into the article generation model. If the output of the article generation model is "(1) Go to the zoo", the robot 100 will make the robot 100 say "(1) Go to the zoo" as an emotion change event when meeting the user next time, and store it in behavior pre-defined data 2224.

[0371] Alternatively, for example, event data with high emotional values ​​for Robot 100 can be selected as Robot 100's most memorable events. Therefore, it is possible to create emotional change events based on the event data selected as most memorable events.

[0372] When the behavior determination unit 236 determines that "(11) the robot talks about the user's concerns," that is, the robot 100 speaks about the user 10's concerns as robot behavior, it determines the content of the speech related to event data where the emotional value meets a predetermined benchmark. For example, based on the user 10's speech or expressions when going to an art museum as a child, or when studying chemistry, geography, or history, it can grasp the user 10's emotional value towards learning. Things with high emotional values ​​(e.g., above a threshold) can be assumed to be things that the user 10 cares about (is interested in). Therefore, the robot 100 can store event data including the user 10's behavior when the user 10's emotional value is high (what they learned or what they saw that moved them, etc.) in historical data 222. In this case, the behavior determination unit 236 can determine speech content such as "What part of that art museum are you interested in?", "Can you teach me the chemistry I just learned?", or "If I want to further deepen my chemistry knowledge, it would be better to read this book." In addition, the behavior determination unit 236 can also determine the content of the speech so as to pose quizzes about the art museum visited and the chemistry studied. Furthermore, the behavior determination unit 236 can also determine the content of the speech in order to consider giving a new story related to the history being learned. At this time, the behavior control unit 250 outputs a sound representing the determined speech content of the robot 100 from the speaker included in the controlled object 252. In addition, when the user 10 is not in the vicinity of the robot 100, the behavior control unit 250 does not output a sound representing the determined speech content of the robot 100, but instead saves the determined speech content of the robot 100 in the behavior predetermined data 224. In this way, after a certain period of time has passed since the user 10's behavior, by having the robot 100 spontaneously engage in conversation with the user 10 about matters of interest, the child's self-affirmation can be improved, and the willingness to learn can be enhanced.

[0373] When the behavior determination unit 236 determines that "(12) the robot poses a question to the user," that is, the robot 100 poses a question to the user 10 as robot behavior, it generates a question for the user 10. For example, the behavior determination unit 236 can generate a question for the user 10 based on at least one of the user 10's dialogue history or the user 10's personal information. As an example, if it is inferred from the user 10's dialogue history that the user 10's weakest subject is arithmetic, the behavior determination unit 236 can generate a question such as "What is 7×7?" Accordingly, the behavior control unit 250 can output a sound representing the generated question from the speaker included in the control object 252. Then, if the user 10 answers "49", the behavior determination unit 236 can determine a statement such as "Correct answer. Well done, very good!" Then, if it is inferred from the user 10's emotions that the user is interested in the question, the behavior determination unit 236 can regenerate a question with the same questioning tendency. As another example, if user 10's personal information indicates that the user is 10 years old, the behavior determination unit 236 can also create an age-appropriate question such as "What is the capital of the United States of America?" Accordingly, the behavior control unit 250 can output a sound representing the generated question from a speaker included in the controlled object 252. Then, if user 10 answers "New York," the behavior determination unit 236 can determine a statement such as "Unfortunately, the correct answer is Washington DC." Furthermore, if user 10's emotions indicate a lack of interest in the question, the behavior determination unit 236 can change the question's focus and create a new question. In this way, the robot 100 can spontaneously generate questions in a playful manner, for example, in a way that user 10, as a child, enjoys learning, and praises or shares joy based on user 10's answers, thereby increasing user 10's willingness to learn.

[0374] When the behavior determination unit 236 determines that the statement "(13) Robot teaches music," i.e., the robot 100 teaching music to the user 10, is a robot behavior, it evaluates the sound emitted by the user 10. Furthermore, the term "sound emitted by the user 10" can be interpreted as including various sounds that accompany the user 10's behavior, such as the user 10's singing, the sound of an instrument played by the user 10, or the sound of the user 10 striking. For example, if the behavior of the user 10 identifies that the user 10 is singing, playing an instrument, or dancing, the behavior determination unit 236 determines "(13) Robot teaches music" as a robot behavior. In this case, the behavior determination unit 236 can evaluate at least one of the following: rhythm, pitch, or intonation, such as the user 10's singing, instrumental sound, or striking sound. Then, based on the evaluation result, the behavior determination unit 236 can determine the content of the statement, such as "unstable rhythm," "off-pitch," or "emotional expression." Accordingly, the behavior control unit 250 can output a sound representing the determined content of the robot 100's statement from the speaker included in the controlled object 252. In this way, even without being asked by user 10, robot 100 can spontaneously evaluate the sounds made by user 10, pointing out differences in rhythm or pitch, and thus can interact with user 10 as a music teacher.

[0375] Behavior determination unit 236 gives advice about the child when it determines that "(14) giving advice about the child" is a robot behavior, that is, giving advice.

[0376] In addition, regarding “(14) giving advice about children”, the storage control unit 2238 periodically detects the user’s behavior (session or action) as the user’s state and saves it in the historical data 222.

[0377] When the behavior determination unit 236 determines "(15) the robot estimates the user's mental age," that is, estimates the user's mental age based on the user's behavior as robot behavior, it estimates the user's mental age based on the user's behavior (conversation or action) identified by the state recognition unit 230. At this time, the behavior determination unit 236 may, for example, input the user's behavior identified by the state recognition unit 230 into a pre-learned neural network and estimate the user's mental age by evaluating the user's mental age. In addition, the behavior determination unit 236 may also periodically detect (recognize) the user's behavior (conversation or action) as the user's state by the state recognition unit 230 and store it in the historical data 222, and estimate the user's mental age based on the user's behavior stored in the historical data 222. In addition, the behavior determination unit 236 may, for example, estimate the user's mental age by comparing the most recent user behavior stored in the historical data 222 with the past user behavior stored in the historical data 222.

[0378] When the behavior determination unit 236 determines that "(16) the robot considers the mental age of user 10," that is, determines the behavior of robot 100 based on the estimated mental age of user 10, for example, it determines the language or speaking style and actions (changing the actions) of robot 100 to user 10 based on the estimated mental age of user 10. Specifically, the behavior determination unit 236 may, for example, increase the difficulty of the language spoken by robot 100 or make the speaking style or actions of robot 100 more like an adult as the estimated mental age of user 10 increases. In addition, the behavior determination unit 236 may also increase the types of language or actions spoken by robot 100 to user 10 or expand the functions of robot 100 as the mental age of user 10 increases. In addition, the behavior determination unit 236 may, for example, input text indicating the mental age of user 10 into the article generation model in addition to text indicating at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state and text inquiring about the behavior of robot 100, and determine the behavior of robot 100 based on the output of the article generation model.

[0379] When determining "(17) Robot presumes user's English level", that is, presumes user 10's English level as robot behavior, behavior determination unit 236, based on the conversation with user 10 stored in historical data 222, presumes user 10's English level according to the level of English words used by user 10, the appropriateness of the English words in the context, the length or grammatical correctness of the article spoken by user 10, the speed and fluency of user 10's speech, and user 10's understanding of the content spoken by robot 100 in English (listening comprehension level, etc.).

[0380] When the behavior determination unit 236 determines that "(18) the robot engages in English conversation with the user," i.e., engaging in English conversation with the user, is a robot behavior, it uses an article generation model based on event data stored in historical data 222 to determine the content to be spoken to user 10. At this time, the behavior determination unit 236 engages in English conversation at a level consistent with user 10. In addition, in order to connect with user 10's future English improvement, a course program suitable for user 10 is created, and the behavior determination unit 236 engages in conversation with user 10 according to the program. Furthermore, in order to improve user 10's English ability, the behavior determination unit 236 gradually incorporates vocabulary of a higher level to advance the conversation.

[0381] In addition, regarding "(18) the robot engaging in English conversation with the user," the relevant information collection unit 270 collects user 10's preferences from external data sources (such as news websites, video websites, etc.). Specifically, the relevant information collection unit 270 obtains news or interesting topics that user 10 expresses interest in based on user 10's speech or settings. Furthermore, the relevant information collection unit 270 collects English vocabulary at a level one level higher than user 10's English proficiency from external data sources.

[0382] In addition, regarding "(18) the robot has an English conversation with the user", the storage control unit 238 always stores and detects the English proficiency of the user 10 as a student.

[0383] When the behavior determination unit 236 determines that "(19) giving suggestions on reading," that is, the robot 100 giving suggestions on reading to the user 10, is a robot behavior, it generates suggestions on reading based on the collected reading-related information and according to the prescribed suggestion conditions, and provides suggestions to the user 10 who is a parent or child. The suggestions are provided, for example, by the robot 100 speaking suggestions.

[0384] Specifically, for user 10 identified by state recognition unit 230, the system identifies the case of a first user who is the parent (mother or father) reading aloud to the child and a second user who is the child receiving the reading. Behavior determination unit 236 performs processing to generate and provide reading suggestions to at least one of the users according to the suggestion conditions. The suggestion conditions at least set the frequency of suggestion provision. The user 10 can appropriately change the setting for the frequency, such as once every 3 days or once every 5 days. Behavior determination unit 236 generates and provides suggestions based on the frequency set as the suggestion conditions. Furthermore, the suggestion provision frequency for the first user (parent) and the suggestion provision frequency for the second user (child) can also be preset in the suggestion conditions. Additionally, as described later, conditions related to the first user (parent) and the second user (child) can be further set.

[0385] The relevant information collection unit 270 collects the content of the book that the first user (parent) is reading to the second user, and stores the book and its associated title. For the book content, it receives the book title as input from the first user beforehand, and collects and stores summaries, main text, etc., representing the book content from external data. Alternatively, it may collect the book content based on the first user's statements, rather than the first user's input, by referring to external data.

[0386] Regarding the state when reading to a second user (child), information analyzed by the sensor module 210 and the like is collected, the state is identified by the state recognition unit 230, and the emotion determination unit 232 determines the emotion value (corresponding to step S102). In addition, the behavior determination unit 236 may also store the content read when the emotion value is high, and include the content itself or a summary of the content in the suggestion.

[0387] One example of the information collected related to reading aloud is the content of the book (or a summary of the content) that the first user (parent) is reading aloud, as well as the emotional value given to the second user (child) while reading aloud. In this way, the information related to reading aloud is collected separately with regard to the first user and with regard to the second user.

[0388] The behavior determination unit 236 generates and provides reading-related suggestions based on the collected reading-related information and the proposed conditions. These suggestions may include, for example, listing books read when the emotional value is high, or books similar to the current book content, and suggesting the titles of the books to be read. The user providing the suggestions can be either the first user (parent) or the second user (child), and the suggestions correspond to the user's category. For example, for the first user (parent), the suggestion might be something like, "How about trying to read it to your child? Here are some recommended book titles: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD...". For the second user, the suggestion might be something like, "How about I read AAAA?" Alternatively, the suggestions might summarize the book content when the second user's (child's) emotional value is high during reading, and include additional information. For example, for the first user (parent), the suggestion might include additional information like, "It seems like they really like the XXXX scene in AAAA," and for the second user (child), the suggestion might include additional information like, "XXXX in AAAA is very good." In addition, the above suggestion is an example.

[0389] Additionally, the behavior determination unit 236 can also collect the date and time of the first user (parent)'s reading, and as a suggestion condition, provide reading-related suggestions if reading has not been performed for a certain period, such as a period of more than 3 days or more than 1 week. Furthermore, as a suggestion condition, the emotional value of the second user (child) can be collected, and if the emotional value tends to decrease, reading-related suggestions can be provided to either the first user (parent) or the second user (child). Thus, in addition to the usual frequency of provision, the suggestion conditions can also include conditions that use the reading frequency of the first user (parent) and the emotional tendency of the second user (child). The above is an explanation of the situation of "(19) giving suggestions on reading".

[0390] Robot 100 can be designed to resemble a human or be a plush toy. It is believed that because of its plush toy appearance, Robot 100 is particularly appealing to children.

[0391] The behavior determination unit 2236 determines the behavior of the robot 100 based on the state of the user 10 identified by the state recognition unit 2230, from a state where there is no behavior of the user 10 towards the robot 100, to a state where the user 10 towards the robot 100 is detected.

[0392] For example, if user 10 is not near robot 100, and user 10 is detected, the behavior determination unit 2236 reads the data stored in the behavior pre-defined data 2224 to determine the behavior of robot 100. Conversely, if user 10 is asleep, and user 10 is detected waking up, the behavior determination unit 2236 reads the data stored in the behavior pre-defined data 2224 to determine the behavior of robot 100.

[0393] Figure 9B This section provides a simplified example of the operational process for collecting and processing information related to user 10's preferences. Figure 9B The illustrated operation flow is repeated at regular intervals. It is assumed that user 10's preferences, representing matters of interest to user 10, are obtained based on user 10's statements or settings. Furthermore, "S" in the operation flow indicates the step being performed.

[0394] First, in step S90, the association information collection unit 2270 acquires preference information representing the matters of concern to user 10.

[0395] In step S92, the association information collection unit 2270 collects information related to preference information from external data.

[0396] In step S94, the emotion determination unit 2232 determines the emotion value of the robot 100 based on information associated with the preference information collected by the association information collection unit 2270.

[0397] In step S96, the storage control unit 2238 determines whether the emotion value of the robot 100 determined in step S94 is above a threshold. If the emotion value of the robot 100 is below the threshold, the information associated with the collected preference information is not stored in the collected data 2223, and the process ends. On the other hand, if the emotion value of the robot 100 is above the threshold, the process proceeds to step S98.

[0398] In step S98, the storage control unit 2238 saves the information associated with the collected preference information in the collected data 2223 and ends the process.

[0399] Figure 3 This section provides a simplified example of an operational flow related to determining the behavior within the robot 100 during response processing in response to the user 10's actions. (Repeated execution) Figure 3 The operation flow is shown below. At this time, it is assumed that the input is information analyzed by the sensor module 2210.

[0400] First, in step S100, the state recognition unit 2230 identifies the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 2210. For example, if the identified user 10 is a caregiver or a person being cared for, the state recognition unit 2230 identifies the user 10's physical and mental state (stress level or fatigue level, etc.). Additionally, the state recognition unit 2230 identifies the physical and mental state (health status and lifestyle habits, etc.) of each of the multiple users 10 constituting a family member. Furthermore, the state recognition unit 2230 identifies the mental state of each of the multiple users 10 constituting a family member.

[0401] In step S102, the emotion determination unit 2232 determines the emotion value of the user 10 representing emotion based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0402] In step S103, the emotion determination unit 2232 determines the emotion value of the robot 100 representing emotion based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the historical data 2222.

[0403] In step S104, the behavior recognition unit 2234 identifies the behavior classification of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0404] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on the combination of the current sentiment value of the user 10 determined in step S102 and the past sentiment values ​​contained in the historical data 2222, the sentiment value of the robot 100, the behavior of the user 10 identified in step S104 above, and the behavior determination model 2221.

[0405] In step S108, the behavior control unit 2250 controls the controlled object 252 based on the behavior determined by the behavior determination unit 2236.

[0406] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the behavior predetermined by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0407] In step S112, the storage control unit 2238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the event data containing the user 10's behavior is not stored in the historical data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0408] In step S114, event data including the behavior determined by the behavior determination unit 2236, the information analyzed by the sensor module unit 2210 from the current time point to a certain period, and the state of the user 10 identified by the state recognition unit 2230 are stored in historical data 2222.

[0409] Figure 9C An example of the operational flow related to the operation of determining the behavior in robot 100 is shown in a simplified manner when robot 100 performs autonomous processing of autonomous behavior. Figure 9C The illustrated operation flow, for example, is repeated automatically after a certain period of time. At this time, it is assumed that the input is information analyzed by the sensor module 2210. Furthermore, regarding the above... Figure 3 The same process is shown with the same step numbers.

[0410] First, in step S100, the state recognition unit 2230 identifies the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 2210.

[0411] In step S102, the emotion determination unit 2232 determines the emotion value of the user 10 representing emotion based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0412] In step S103, the emotion determination unit 2232 determines the emotion value of the robot 100 representing emotion based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the historical data 2222.

[0413] In step S104, the behavior recognition unit 2234 identifies the behavior classification of the user 10 based on the information analyzed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0414] In step S200, the behavior determination unit 2236 determines any one of a variety of robot behaviors, including not performing any behavior, as the behavior of the robot 100 based on the state of the user 10 identified in step S100, the emotion of the user 10 identified in step S102, the emotion of the robot 100, the state of the robot 100 identified in step S100, the behavior of the user 10 identified in step S104, and the behavior determination model 2221.

[0415] In step S201, the behavior determination unit 2236 determines whether it has been determined in step S200 that no behavior will be performed. If it is determined that no behavior will be performed, the process ends. On the other hand, if it is determined that no behavior will be performed, the process proceeds to step S202.

[0416] In step S202, the behavior determination unit 2236 performs processing corresponding to the robot behavior type determined in step S200 above. At this time, the behavior control unit 2250, the emotion determination unit 2232, or the storage control unit 2238 performs processing according to the robot behavior type.

[0417] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the behavior predetermined by the behavior determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0418] In step S112, the storage control unit 2238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the event data containing the user 10's behavior is not stored in the historical data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0419] In step S114, the storage control unit 2238 stores event data, including the behavior determined by the behavior determination unit 2236, the information analyzed by the sensor module unit 2210 from the current time point to a certain period, and the state of the user 10 identified by the state recognition unit 2230, in the historical data 2222.

[0420] As described above, based on the user's state, the robot 100 determines an emotion value representing its own emotions. Based on this emotion value, it determines whether to store data including the user 10's behavior in the historical data 2222. This reduces the capacity of the historical data 2222 containing data about the user 10's behavior. Furthermore, for example, if the robot 100 determines that the user's state 10 years later is the same as 10 years ago, by reading the historical data 2222 from 10 years ago, the robot 100 can display to the user 10 the user's state 10 years ago (e.g., the user's facial expressions, emotions, etc.), and all surrounding information such as sounds, images, and smells from that time.

[0421] Furthermore, according to the robot 100, it can perform appropriate actions in response to the user 10's behavior. Previously, user behavior was categorized, and behaviors including those related to the robot's facial expressions or physical appearance were identified. In contrast, the robot 100 determines the user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if the user 10 was in a good mood yesterday but is feeling down today, the robot 100 can say something like, "You were fine yesterday, what's wrong today?" Additionally, the robot 100 can also incorporate gestures into its speech. For example, if the user 10 was feeling down yesterday but is in a good mood today, the robot 100 can say something like, "You were feeling down yesterday, are you feeling good today?" For example, if the user 10 was in a good mood yesterday but is feeling even better today, the robot 100 can say something like, "You're feeling better today than yesterday. Is there anything better than yesterday?" Furthermore, for example, if the robot 10 is consistently in a state where their emotional state is above 0 and fluctuates within a certain range, the robot 100 can say something like, "Your mood has been stable lately, you feel good."

[0422] Furthermore, for example, if robot 100 asks user 10, "Did you finish the homework we talked about yesterday?" and receives a "Yes, I did!" from user 10, it can make affirmative remarks such as "That's great!" and perform affirmative gestures such as clapping. Additionally, if user 10 says, "The demonstration we talked about the day before yesterday went very well," robot 100 can make affirmative remarks such as "You did your best!" and perform the aforementioned affirmative gestures. In this way, by acting based on user 10's state history, robot 100 can be expected to develop a sense of closeness to robot 100.

[0423] Additionally, for example, when user 10 is watching a panda-related video, if the "joy" emotion value of user 10 is above the threshold, the panda appearance scene in the video can also be stored as event data in historical data 2222.

[0424] By utilizing data accumulated in historical data 2222 or collected data 2223, robot 100 can always learn which conversations with users will maximize the emotional value of expressing user happiness.

[0425] In addition, even when the robot 100 is not in conversation with the user 10, it can autonomously begin to act based on the robot 100's emotions.

[0426] Furthermore, during autonomous processing, robot 100 automatically generates questions, inputs them into the article generation model, and repeatedly obtains the output of the article generation model as answers to the questions. It can then create emotional change events to amplify positive emotions and store them in the predefined behavioral data 2224. In this way, robot 100 can perform self-learning.

[0427] In addition, when the robot 100 is not triggered by external factors, it can automatically generate questions based on memorable event data determined from the robot's past emotional value history.

[0428] In addition, the related information collection unit 2270 can perform self-learning by automatically performing keyword searches in accordance with the user's preference information and repeatedly obtaining search results during the search execution phase.

[0429] Here, the retrieval execution phase can also automatically perform keyword retrieval based on impressive event data determined from the robot's past sentiment value history, without being triggered by external factors.

[0430] Furthermore, the emotion determination unit 2232 can determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 2232 can determine the user's emotion according to an emotion map that serves as a specific mapping (see [reference]). Figure 5 To determine the user's emotions.

[0431] Figure 5 This is a diagram representing an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more the original state of the emotion is arranged. On the outer side of the concentric circles, emotions representing states or behaviors arising from moods are arranged. Emotions are concepts that include feelings or mental states. On the left side of the concentric circles, emotions generated by reactions induced in the brain are generally arranged. On the right side of the concentric circles, emotions induced by situational judgments are generally arranged. Above and below the concentric circles, emotions generated by reactions induced in the brain and induced by situational judgments are generally arranged. In addition, the emotion of "pleasure" is arranged above the concentric circles, and the emotion of "unpleasantness" is arranged below. In this way, in the emotion map 400, multiple emotions are mapped based on the structure that generates emotions, and emotions that are likely to be generated simultaneously are mapped to the vicinity.

[0432] (1) For example, if the emotion engine of the emotion determination unit 2232 of the robot 100 detects emotions at approximately 100 milliseconds, the determination of the robot 100's reaction operation (e.g., echoing response) can be set at a time with a frequency at least the same as the detection frequency (100 milliseconds) of the emotion engine, or it can be set at a time earlier. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0433] By detecting emotions at approximately 100 milliseconds and immediately responding (e.g., echoing), the system achieves natural and appropriate dialogue rather than unnatural echoing. Robot 100 responds based on the mandala orientation and intensity of the emotion map 400. Furthermore, the emotion engine's detection frequency (sampling rate) is not limited to 100 milliseconds and can be adjusted based on the scenario (movement, etc.) and the user's age.

[0434] (2) In contrast to the emotion map 400, the directionality and intensity of the emotion can be preset, and the operation and intensity of the echo response can be set. For example, when the robot 100 feels stable and at ease, the robot 100 nods and continues to listen to the speech. When the robot 100 feels uneasy, hesitant, or strange, the robot 100 can tilt its head or stop shaking its head.

[0435] These emotions are distributed in the 3 o'clock direction of the emotion map 400, usually moving back and forth between feelings of peace and unease. In the right half of the emotion map 400, situational awareness is more dominant than internal feelings, thus forming an impression of calmness.

[0436] (3) When Robot 100 feels happy after being praised, a filler word such as "hmm" can be added before the dialogue. When Robot 100 feels pain after being harshly criticized, a filler word such as "ah!" can be added before the dialogue. In addition, physical reactions such as Robot 100 curling up while saying "ah!" can also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0437] (4) In the left half of the emotion map 400, internal sensation (response) is more dominant than situation recognition. Therefore, it gives the impression of involuntary reaction.

[0438] When Robot 100 senses internal feelings of recognition (response) and also develops liking in situational awareness, it can look at the other party, nod deeply, and even make an "uh-huh" sound. In this way, Robot 100 can generate just the right amount of liking for the other party, that is, behaviors such as tolerance or forbearance. This emotion is distributed around the 12 o'clock position on the emotion map 400.

[0439] Conversely, when Robot 100 experiences an internal feeling (reaction) of displeasure and the same applies to situation recognition, Robot 100 can shake its head when feeling disgust, and if it feels aversion, it can turn the LEDs in its eyes red and stare at the other person. This emotion is distributed around the 6 o'clock position on the emotion map 400.

[0440] (5) Because the inner side of the emotional map 400 represents the inner world and the outer side of the emotional map 400 represents behavior, the further out of the emotional map 400, the more obvious the emotion becomes (manifested as behavior).

[0441] (6) When feeling at ease around the 3 o'clock position on the emotional map 400 and listening to people speak, the robot 100 nods slightly and says "uh-huh". However, when it reaches the love position around the 12 o'clock position, it can nod strongly like a deep bow.

[0442] Here, human emotions are based on various balances, such as posture or blood sugar levels. When these balances deviate from the ideal, they represent unpleasant states; when they approach the ideal, they represent pleasant states. Even in robots, cars, and motorcycles, emotions can be generated based on various balances, such as posture or remaining battery power, representing unpleasant states when these balances deviate from the ideal and pleasant states when they approach the ideal. Emotion maps can be generated based on, for example, Dr. Mitsuyoshi's emotion map (Research on a Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the sensory domain, called "response," are arranged. In the right half, emotions belonging to the situational domain, called "situation," are arranged.

[0443] The emotion map defines two types of emotions that drive learning. One is the negative emotion surrounding "repentance" or "reflection" on the situation side. That is, the robot experiences negative emotions such as "I never want to experience this feeling again" or "I don't want to be scolded anymore." The other is the positive emotion near "desire" on the response side. That is, positive feelings such as "wanting more" or "wanting to know more."

[0444] The emotion determination unit 2232 inputs the information analyzed by the sensor module unit 2210 and the identified state of the user 10 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is a pre-learned neural network based on a combination of the information analyzed by the sensor module unit 2210, the identified state of the user 10, and the emotion values ​​representing each emotion shown in the emotion map 400—that is, multiple learning data. Furthermore, as... Figure 6 As shown in the sentiment map 900, the neural network learns by recognizing that sentiment values ​​in neighboring configurations are similar to each other. Figure 6 The text shows examples of how multiple emotions such as "peace of mind," "stability," and "reliability" have similar emotional values.

[0445] Furthermore, the emotion determination unit 2232 can determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 2232 inputs the information analyzed by the sensor module unit 2210, the state of the user 10 identified by the state recognition unit 2230, and the state of the robot 100 into a pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400, and determine the emotion of the robot 100. This neural network is a neural network pre-learned based on a combination of information analyzed by the sensor module unit 2210, the identified state of the user 10 and the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400, i.e., multiple learning data. For example, if the robot 100 is identified as being touched by the user 10 based on the output of the touch sensor 2206, the neural network learns based on the learning data representing the emotion value "3" which indicates "joy"; or if the robot 100 is identified as being hit by the user 10 based on the output of the accelerometer (not shown), the neural network learns based on the learning data representing the emotion value "3" which indicates "anger". Additionally, as... Figure 6 As shown in the sentiment map 900, the neural network learns in a way that sentiments in nearby configurations have similar values ​​to each other.

[0446] The behavior determination unit 2236 generates robot behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting these sentences into an article generation model with dialogue functionality.

[0447] For example, the behavior determination unit 2236 uses the emotion table shown in Table 3 to obtain text representing the state of the robot 100 based on the emotions of the robot 100 determined by the emotion determination unit 2232. Here, in the emotion table, for each type of emotion, each emotion value is assigned an index number, and for each index number, text representing the state of the robot 100 is stored.

[0448] If the emotion of robot 100, as determined by the emotion determination unit 2232, corresponds to index number "2", the text "very happy state" is obtained. Furthermore, if the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the states of robot 100 are obtained.

[0449] In addition, an emotion table as shown in Table 4 is also prepared for user 10's emotions.

[0450] Here, given that the user's behavior is "play together," the robot 100's emotion is index number "2," and the user 10's emotion is index number "3," the text input article generation model is used to obtain the robot's behavior content: "The robot is in a very happy state. The user is in a normally happy state. The user has approached the robot with the message 'play together.' As a robot, how should you respond?" The behavior determination unit 2236 determines the robot's behavior based on this behavior content.

[0451] Table 3 Table 4 In this way, the behavior determination unit 2236 determines the behavior content of the robot 100 in correspondence with the state related to the robot 100's emotion, which is predetermined according to each type of emotion and each intensity of that emotion, and the behavior of the user 10. In this manner, the robot 100's speech during dialogue with the user 10 can branch according to the state related to the robot 100's emotion. That is, since the robot 100 can change its behavior according to the index number corresponding to the robot's emotion, the user gets the impression that the robot has a mind, which encourages them to take actions such as striking up a conversation with the robot.

[0452] Furthermore, the behavior determination unit 2236 can generate robot behavior content by adding text representing not only the user's behavior, the user's emotions, and the robot's emotions, but also text representing the content of historical data 2222, then adding fixed sentences for asking questions about the robot's behavior content corresponding to the user's behavior, and inputting this into a text generation model with dialogue functionality. Thus, since the robot 100 can change its behavior based on historical data representing the user's emotions or behaviors, the user develops the impression that the robot has a personality, encouraging them to engage in conversation or other behaviors with the robot. Furthermore, the robot's emotions or behaviors can be further included in the historical data.

[0453] Furthermore, the emotion determination unit 2232 can also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 2232 inputs the behavioral content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion map 400, integrates the obtained emotion values ​​representing each emotion with the current emotion values ​​representing each emotion of the robot 100, and updates the emotion of the robot 100. For example, the obtained emotion values ​​representing each emotion and the current emotion values ​​representing each emotion of the robot 100 are averaged and integrated respectively. This neural network is a neural network pre-learned based on a combination of text representing the behavioral content of the robot 100 generated by the article generation model and the emotion values ​​representing each emotion shown in the emotion map 400, i.e., multiple learning data.

[0454] For example, when the robot 100's speech content "That's great. How lucky!" is obtained, as the behavioral content of the robot 100 generated by the article generation model, the sentiment value of the emotion "joy" is increased when the text representing the speech content is input into the neural network, so that the sentiment value of the robot 100 is updated in a way that increases the sentiment value of the emotion "joy".

[0455] In Robot 100, article generation models such as ChatGPT work in conjunction with the sentiment determination unit 2232 to execute a method that is self-aware and continues to grow with various parameters even when the user is not speaking.

[0456] ChatGPT is a large-scale language model that uses deep learning methods. ChatGPT can also reference external data; for example, ChatGPT plugins are known to use techniques that, through dialogue, reference various external data such as weather information and hotel reservation information to provide answers as accurately as possible. For instance, in ChatGPT, when a purpose is given in natural language, source code can be automatically generated using various programming languages. Furthermore, when problematic source code is provided, ChatGPT can debug it to identify issues and automatically generate improved source code. Combining these approaches with a purpose given in natural language results in an autonomous agent that repeatedly generates and debugs code before it is found to be flawed. Examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0457] In the robot 100 according to this embodiment, as described in Patent Document 7 (Patent No. 6199927), a technique can be used to retain event data that evokes strong emotions in the robot for a long time and to quickly forget event data that evokes almost no emotions in the robot, so as to retain the event data that should be learned in a database of strong memories.

[0458] Additionally, robot 100 can record image data of user 10 acquired through its camera function in historical data 2222. Robot 100 can retrieve image data from historical data 2222 as needed and provide it to user 10. The stronger the emotional intensity of robot 100, the more information-rich image data it can generate and record in historical data 2222. For example, when recording highly compressed information such as skeletal data, robot 100 can switch to recording low-compression information such as HD video if its emotional excitement value exceeds a threshold. According to robot 100, for example, it is possible to retain high-precision image data of robot 100 when its emotions are heightened.

[0459] When the robot 100 is not interacting with the user 10, it can automatically load event data from historical data 2222 that stores memorable event data, and continue to update the robot's emotions through the emotion determination unit 2232. The robot 100 can create emotion change events to improve the user 10's emotions based on memorable event data, even when it is not speaking to the user 10 and its emotions are conducive to learning. Thus, autonomous learning (recalling event data) can be achieved at appropriate times in accordance with the robot 100's emotional state, and autonomous learning that appropriately reflects the robot 100's emotional state can be realized.

[0460] Regarding emotions that promote learning, in a negative state, they are emotions near "repentance" or "reflection" on Dr. Mitsuyoshi's emotion map, while in a positive state, they are emotions near "desire" on the emotion map.

[0461] Robot 100 can process "repentance" and "reflection" from the emotion map as learning-promoting emotions in a negative state. In addition to "repentance" and "reflection" from the emotion map, Robot 100 can also process emotions adjacent to "repentance" and "reflection" as learning-promoting emotions in a negative state. For example, besides "repentance" and "reflection," Robot 100 can also process at least one of "regret," "stubbornness," "self-destruction," "self-discipline," "regret," and "despair" as learning-promoting emotions. Thus, for example, Robot 100 can perform autonomous learning while holding negative feelings such as "I never want to experience this feeling again" or "I don't want to be scolded again."

[0462] In a positive state, Robot 100 can process "desire" from the emotion map as an emotion that promotes learning. In addition to "desire," Robot 100 can also process emotions adjacent to "desire" as emotions that promote learning. For example, besides "desire," Robot 100 can also process at least one of "joy," "ecstasy," "desire," "anticipation," and "shame" as emotions that promote learning. Thus, for example, Robot 100 can perform autonomous learning while holding positive feelings such as "wanting more" or "wanting to know more."

[0463] Robot 100 may not perform autonomous learning when it is experiencing emotions other than those that promote learning. Thus, for example, it may not perform autonomous learning when extremely angry or blindly feeling love.

[0464] Emotional change events refer to behaviors following events that are particularly memorable, such as those that are presented in a way that is very impactful. Behaviors following such impactful events are those associated with the outermost emotional tags on an emotion map, such as "love" preceded by "forgiveness" or "acceptance."

[0465] In the autonomous learning process performed by robot 100 when it is not speaking to user 10, it combines the emotions, situations, and behaviors of people and itself that appear in its deepest memories, and uses an article generation model to create emotional change events.

[0466] Assuming all sentiment values ​​are represented by a six-level rating from 0 to 5, consider the scenario where a memorable event, such as "a friend was reprimanded and seemed annoyed," is stored in historical data 2222. Here, "friend" refers to user 10, and user 10's sentiment is "disgust," with a value of 5. Additionally, assuming robot 100's sentiment is "anxiety," with a value of 4, "anxiety" is represented by a value of 4.

[0467] While not interacting with user 10, robot 100 can continue to evolve with various parameters through autonomous processing. Specifically, it loads event data such as "My friend was hit and seemed annoyed" as the top-ranking event data from historical data 2222, arranged in order of emotional intensity. In the loaded event data, "anxiety" (intensity 4) is bound to robot 100's emotion, while "disgust" (intensity 5) is bound to user 100's emotion. If robot 100's current emotional value was "reassurance" (intensity 3) before loading, after loading, considering the influence of "anxiety" (intensity 4) and "disgust" (intensity 5), robot 100's emotional value changes to "regret" (meaning pity / regret). Since "regret" is an emotion that promotes learning, robot 100 determines, as a robot behavior, to recall the event data and create an emotional change event. The information input into the article generation model is the text representing the memorable event data, in this example, "My friend was hit and seemed annoyed." Additionally, in the emotion map, the innermost emotion is "disgust," and the corresponding behavior is predicted as "aggression" on the outermost side. Therefore, in this case, an emotion change event is created to prevent a friend from "attacking" one of them.

[0468] For example, if impressive event data is used to solve fill-in-the-blank questions, the following input text can be automatically generated.

[0469] "The user was hit. At that moment, the user felt extremely disgusted. The robot was very uneasy. Please tell me in 30 words or less what the robot should say when it meets the user next time. However, please note that it is unrelated to the time of the meeting. Also, please avoid direct expression. List three candidates."

[0470] <Expected Format> Candidate 1: (What the robot should say to the user) Candidate 2: (What the robot should say to the user) Candidate 3: (What the robot should say to the user) At this point, the output of the article generation model is as follows.

[0471] "Candidate 1: Are you alright? I'm really worried about what happened yesterday."

[0472] Candidate 2: I'm really worried about what happened yesterday. Is there anything I can do for you? Candidate 3: I'm worried about you. Can we talk? Furthermore, regarding the information obtained in the creation of emotional change events, Robot 100 can also automatically generate the following input text.

[0473] If a user has been "knocked down," how will they feel when you greet them again? The user's emotions are expressed as "Joy (A), Anger (B), Sorrow (C), Happiness (D)," with six levels of rating from 0 to 5 for each level.

[0474] Candidate 1: Are you alright? I'm really worried about what happened yesterday.

[0475] Candidate 2: I'm really worried about what happened yesterday. Is there anything I can do for you? Candidate 3: I'm worried about you. Can we talk? At this point, the output of the article generation model is as follows.

[0476] "The user's emotions may be as follows."

[0477] Candidate 1: Joy 3, Anger 1, Sorrow 2, Happiness 2 Candidate 2: Joy 2, Anger 1, Sorrow 3, Happiness 2 Candidate 3: Joy 2, Anger 1, Sorrow 3, Happiness 3 In this way, Robot 100 can also perform retrospective thought processing after creating an emotional change event.

[0478] Finally, robot 100 can use the most pleasing candidate 1 among multiple candidates to create an emotion change event, store it in behavior pre-planning data 2224, and prepare for the next meeting with user 10.

[0479] As described above, even without conversations with family or friends, the robot continues to determine its emotional value using historical data 2222, which stores impressive event data. When the robot becomes the aforementioned emotion that promotes learning, it performs autonomous learning based on its own emotion, without conversations with the user 10, and continues to update the historical data 2222 or the behavior pre-defined data 2224.

[0480] The above are examples of using sentiment values. However, since sentiment maps can be constructed based on hormone levels and event types, the values ​​associated with memorable event data can also be hormone types, hormone levels, or event types.

[0481] The following describes specific implementation examples.

[0482] Robot 100, for example, surveys users about topics of interest or information related to their interests, even without speaking to them.

[0483] Robot 100, for example, investigates information related to the user's birthday or anniversary and considers messages of blessing, even without speaking to the user.

[0484] Robot 100, for example, can survey users about places they want to visit or reviews of food and goods, even without speaking to them.

[0485] Robot 100, for example, can investigate weather information and provide suggestions that match the user's schedule or plans, even without speaking to the user.

[0486] Robot 100, for example, investigates information about local events or festivals and makes suggestions to users even without speaking to them.

[0487] Robot100, for example, can survey users about sports results or news that they are interested in, and provide topics of conversation, even without speaking to the user.

[0488] Robot 100, for example, surveys and introduces information about the user's favorite music or artists even without speaking to the user.

[0489] Robot 100, for example, investigates information related to social issues or news that users are concerned about and provides opinions even without speaking to the user.

[0490] Robot 100, for example, investigates information related to the user's hometown or birthplace and provides topics of conversation even without speaking to the user.

[0491] Robot 100, for example, can investigate a user's work or school information and provide suggestions even without speaking to the user.

[0492] Even without speaking to the user, Robot100 surveys and introduces information about books, comics, movies, and TV series that the user may be interested in.

[0493] Robot 100, for example, investigates health-related information and provides advice even without speaking to the user.

[0494] Robot 100, for example, investigates information related to a user's travel plans and provides suggestions even without speaking to the user.

[0495] Robot 100, for example, can investigate information related to repairing or maintaining a user's home or car and provide suggestions even without speaking to the user.

[0496] Robot 100, for example, can survey users for information on beauty or fashion that interests them, and offer suggestions, even without speaking to them.

[0497] Robot 100, for example, investigates a user's pet information and provides suggestions even without speaking to the user.

[0498] Robot 100, for example, investigates information about competitions or events related to the user's interests or work, and makes suggestions, even without speaking to the user.

[0499] Robot 100, for example, can investigate information about a user's favorite restaurants or eateries and make suggestions even without speaking to the user.

[0500] Robot 100, for example, gathers information and provides advice on important life-related decisions for users even without speaking to them.

[0501] Robot 100, for example, can investigate information related to people the user is concerned about and provide suggestions even without speaking to the user.

[0502] [Third Implementation Method] In the third embodiment, the robot 100 is mounted on a plush toy, or applied to a control device that connects wirelessly or wiredly to a controllable machine (speaker or camera) mounted on the plush toy. Furthermore, parts that have the same structure as in the second embodiment are labeled with the same symbols and their descriptions are omitted.

[0503] Specifically, the third implementation is configured as follows. For example, robot 100 is applied to cohabiting individuals (specifically, Figure 7 and Figure 8 The plush toy 100N shown above, while interacting with the user 10 in daily life, advances dialogue based on information relevant to the user 10's daily life, or provides information that aligns with the user 10's interests and hobbies. In the third embodiment, an example of applying the control unit of the robot 100 described above to a smartphone 50 will be described.

[0504] For the plush toy 100N, which is equipped with an input / output device that functions as an input / output device for the robot 100, a smartphone 50 that functions as a control unit for the robot 100 can be detached and installed. Inside the plush toy 100N, the input / output device and the stored smartphone 50 are interconnected.

[0505] like Figure 7 As shown in (A), the plush toy 100N in this embodiment (third embodiment) is in the shape of a bear with its exterior covered by soft fabric. In the space 52 formed inside it, a sensor unit 2200A and a control object 2252A are arranged as input / output devices (see reference). Figure 9D The sensor unit 2200A includes a microphone 2201 and a 2D camera 2203. Specifically, as... Figure 7As shown in (B), in the space section 52, a microphone 2201 of the sensor section 2200 is arranged in the part corresponding to the ear 54, a 2D camera 2203 of the sensor section 2200 is arranged in the part corresponding to the eye 56, and a speaker 60, which constitutes part of the controlled object 2252A, is arranged in the part corresponding to the mouth 58. Furthermore, the microphone 2201 and the speaker 60 are not necessarily separate units; they can also be an integrated unit. In the case of an integrated unit, it can be arranged in a position such as the nose of the plush toy 100N where the speech can be naturally heard. Furthermore, the example given is of the plush toy 100N being in the shape of an animal, but it is not limited to this. The plush toy 100N can be in the shape of a specific character.

[0506] Figure 9D The functional structure of the plush toy 100N is roughly represented. The plush toy 100N includes a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a controlled object 2252A.

[0507] The smartphone 50 housed in the plush toy 100N of this embodiment performs the same processing as the robot 100 of the second embodiment. That is, the smartphone 50 has... Figure 9D The functions shown are as follows: sensor module 2210, storage unit 2220, and control unit 2228.

[0508] like Figure 8 As shown, a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, it becomes a structure that communicates with the external space 52.

[0509] Here, the smartphone 50 is stored in the space 52 from the outside, via the USB hub 64 (see reference). Figure 7 (B) It can be connected to each input / output device via USB, thereby having the same functions as the robot 100 of the second embodiment described above.

[0510] Additionally, a contactless power receiver 66 is connected to the USB hub 64. A power receiving coil 66A is assembled on the power receiver 66. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0511] The power receiving board 66 is positioned near the base 68 of the two legs of the plush toy 100N, and is in the position closest to the mounting base 70 when the plush toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power supply unit.

[0512] The plush toy 100N placed on the base 70 can be appreciated as a decorative item in its natural state.

[0513] In addition, the root is formed to be thinner than the surface thickness of the plush toy 100N in other parts, so as to be held in a state closer to that of the mounting base 70.

[0514] The mounting base 70 includes a charging pad 72. The charging pad 72 is equipped with a power supply coil 72A. The power supply coil 72A sends a signal to detect the power receiving coil 66A of the power receiving board 66. If the power receiving coil 66A is detected, current flows through the power supply coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field and begins electromagnetic induction. Thus, current flows through the power receiving coil 66A and stores power in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0515] That is, by placing the plush toy 100N as an ornament on the base 70, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0516] In the third embodiment, the smartphone 50 is housed in the space 52 of the plush toy 100N and connected via a wired connection (USB connection), but this is not a limitation. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) can also be housed in the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, the smartphone 50 is not placed in the space 52, and the smartphone 50 communicates wirelessly with the control device. An external smartphone 50 connects to various input / output devices via the control device, thereby enabling it to have the same functions as the robot 100 shown in the second embodiment. Alternatively, the control device housed in the space 52 of the plush toy 100N can be connected to an external smartphone 50 via a wired connection.

[0517] Furthermore, in the third embodiment, a plush bear toy 100N is exemplified, but it can also be other animals, dolls, or the shape of a specific character. Additionally, it can be dressed up. Moreover, the material of the outer skin is not limited to fabric; it can be other materials such as soft plastic, but a soft material is preferred.

[0518] Furthermore, a monitor can be installed on the surface of the plush toy 100N, and a control object 2252 can be added to provide information to the user 10 visually. For example, the eyes 56 can be used as a monitor to express emotions through images reflected in the eyes, or a window through which a built-in smartphone 50 monitor can be installed on the abdomen. Alternatively, the eyes 56 can be used as a projector to express emotions through images projected onto a wall.

[0519] According to the third embodiment, an existing smartphone 50 is placed in a plush toy 100N, thereby extending the camera 2203, microphone 2201, speaker 60, etc. to appropriate positions via USB connection.

[0520] Furthermore, in order to perform wireless charging, the device is configured to connect the smartphone 50 and the power receiving board 66 via USB, with the power receiving board 66 positioned as close to the outside as possible when viewed from inside the plush toy 100N.

[0521] If you want to use the wireless charging of the smartphone 50, you must place the smartphone 50 as far out as possible when looking from the inside of the plush toy 100N. When you touch the plush toy 100N from the outside, it will become uneven.

[0522] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, while the wireless charging function (power receiving board 66) is positioned as far out as possible when viewed from the inside of the plush toy 100N. The camera 2203, microphone 2201, speaker 60, and smartphone 50 receive wireless power via the power receiving board 66.

[0523] Furthermore, the other structures and functions of the plush toy 100N in the third embodiment are the same as those of the robot 100 in the second embodiment, so the description is omitted.

[0524] [Fourth Implementation Method] In the second embodiment described above, an example of applying a behavior control system to robot 100 was given. However, in the fourth embodiment, the robot 100 is used as an intelligent agent for interacting with a user, and the behavior control system is applied to an intelligent agent system. Furthermore, parts that have the same structure as in the second and third embodiments are marked with the same symbols and their descriptions are omitted.

[0525] Figure 9E This is a functional block diagram of an intelligent agent system 2500 that utilizes some or all of the functions of a behavior control system.

[0526] The intelligent agent system 2500 is a computer system that executes a series of actions according to the intentions of user 10 through dialogue with user 10. The dialogue with user 10 can be conducted via voice or text.

[0527] The intelligent agent system 2500 includes a sensor unit 2200A, a sensor module unit 210, a storage unit 2220, a control unit 2228B, and a controlled object 2252B.

[0528] The intelligent agent system 2500 can be installed in, for example, robots, dolls, plush toys, wearable devices (pendants, smartwatches, smart glasses), smartphones, smart speakers, headphones, and panel computers. Alternatively, the intelligent agent system 2500 can also be physically installed on a web server and utilized via a web browser operated on a user's smartphone or other communication terminal.

[0529] The intelligent agent system 2500 acts as a butler, secretary, teacher, partner, friend, lover, or mentor to user 10. The intelligent agent system 2500 not only converses with user 10 but also provides suggestions, directions to destinations, or recommendations based on user preferences. Furthermore, the intelligent agent system 2500 handles appointments, orders, and payments with service providers.

[0530] Similar to the second embodiment described above, the emotion determination unit 2232 determines the emotions of user 10 and the emotions of the intelligent agent itself. The behavior determination unit 2236 also determines the behavior of robot 100 while considering the emotions of user 10 and the intelligent agent. In other words, the intelligent agent system 2500 understands the emotions of user 10 and provides heartfelt support, assistance, advice, and services through observation. The intelligent agent system 2500 also participates in consulting with user 10 about their troubles to comfort, encourage, and uplift them. The intelligent agent system 2500 also plays with user 10, creates drawing journals, and reminisces about the past. The intelligent agent system 2500 performs actions that increase user 10's happiness. Here, "intelligent agent" refers to an intelligent agent operating on software.

[0531] The control unit 2228B includes a status recognition unit 2230, an emotion determination unit 2232, a behavior recognition unit 2234, a behavior determination unit 2236, a storage control unit 2238, a behavior control unit 2250, an associated information collection unit 2270, a command acquisition unit 2272, an RPA (Robotic Process Automation) unit 2274, a role setting unit 2276, and a communication processing unit 2280.

[0532] Similar to the second embodiment described above, the behavior determination unit 2236 determines the content of the intelligent agent's speech for dialogue with the user 10, based on the intelligent agent's behavior. The behavior control unit 2250 outputs the intelligent agent's speech content via a speaker or display, which is the controlled object 2252B, through at least one of sound and text.

[0533] The character setting unit 2276 sets the role of the intelligent agent system 2500 when it converses with the user 10, based on the user 10's specifications. That is, the speech content output from the behavior determination unit 2236 is output by the intelligent agent with the set role. The role can be, for example, a real-life famous person or celebrity such as an actor, entertainer, idol, or athlete. Alternatively, it can be a fictional character appearing in comics, movies, or videos. For example, Princess Anne, played by Audrey Hepburn in the movie *Roman Holiday*, can be set as the intelligent agent role. When the intelligent agent role is known, since the character's voice, wording, tone, and personality are known, the user 10 only needs to specify their preferred role, and the prompts in the character setting unit 2276 are automatically applied. The voice, wording, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 2250 synthesizes a voice corresponding to the character set by the character setting unit 2276 and outputs the intelligent agent's speech content based on the synthesized voice. As a result, user 10 can feel like they are having a conversation with their favorite character (such as their favorite actor).

[0534] When the agent system 2500 is mounted on a device with a display, such as a smartphone, it can display icons, still images, or videos of agents with characters set by the character setting unit 2276 on the display. For example, image synthesis techniques such as 3D rendering can be used to generate images of the agent. In the agent system 2500, the image of the agent can adopt gestures corresponding to the emotions of the user 10, the emotions of the agent, and the content of the agent's speech, while engaging in dialogue with the user 10. Furthermore, when conversing with the user 10, the agent system 2500 can output only sound without outputting images.

[0535] Similar to the first embodiment, the emotion determination unit 2232 determines the emotion value representing the user 10's emotion and the emotion value of the intelligent agent itself. In this embodiment, the emotion value of the intelligent agent is determined instead of the emotion value of the robot 100. The intelligent agent's own emotion value is reflected in the emotion of the set role. When the intelligent agent system 2500 converses with the user 10, the conversation reflects not only the user 10's emotion but also the intelligent agent's emotion. That is, the behavior control unit 2250 outputs the speech content in a manner corresponding to the emotion determined by the emotion determination unit 2232.

[0536] Furthermore, the actions of the intelligent agent system 2500 towards user 10 also reflect the agent's emotions. For example, if user 10 requests the intelligent agent system 2500 to take a photo, whether the intelligent agent system 2500 takes the photo as requested depends on the degree of "sadness" the agent is experiencing. When the agent is experiencing positive emotions, it will engage in benevolent dialogue or actions towards user 10; when experiencing negative emotions, it will engage in resistant dialogue or actions towards user 10.

[0537] Historical data 2222 stores the history of conversations between user 10 and the intelligent agent system 2500 as event data. The storage unit 2220 can also be implemented via an external cloud storage device. When the intelligent agent system 2500 is conversing with user 10 or performing actions towards user 10, it also considers the conversation history stored in historical data 2222 to determine the conversation content or action content. For example, the intelligent agent system 2500 uses the conversation history stored in historical data 2222 to understand user 10's interests and preferences. The intelligent agent system 2500 generates conversation content that matches user 10's interests and preferences, or provides recommendations. The action determination unit 2236 determines the intelligent agent's speech content based on the conversation history stored in historical data 2222. Personal information of user 10, such as name, address, phone number, and credit card number, obtained through conversations with user 10, is stored in historical data 2222. Here, it could also be an intelligent agent proactively asking user 10 whether to register personal information, such as "Do you want to pre-register your credit card number?" Based on user 10's answer, the personal information is saved in historical data 2222.

[0538] As described in the second embodiment above, the behavior determination unit 2236 generates speech content based on an article generated using the article generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 2232, and the dialogue history stored in the history data 2222 into the article generation model to generate the agent's speech content. At this time, the behavior determination unit 2236 can also input the character personality set by the character setting unit 2276 into the article generation model to generate the agent's speech content. In the agent system 2500, the article generation model is not located at the front end, which is the point of contact with the user 10, but is always used as a tool of the agent system 2500.

[0539] The command acquisition unit 2272 uses the output of the speech understanding unit 2212 to acquire commands from the intelligent agent 10, obtained from the voice or text emitted by the user 10 through dialogue with the user 10. The commands include actions that the intelligent agent system 2500 should perform, such as retrieving information, making a store reservation, arranging tickets, purchasing goods or services, paying for goods, providing directions to the destination, and providing recommendations.

[0540] RPA 2274 performs actions based on commands acquired by command acquisition unit 2272. RPA 2274 performs actions related to using service providers, such as retrieving information, making store reservations, arranging tickets, purchasing goods / services, and making payments.

[0541] RPA 2274 reads and utilizes the personal information of user 10 from historical data 2222, which is required to perform actions related to the use of service providers. For example, when the intelligent agent system 2500 makes a purchase based on a request from user 10, it reads and utilizes the personal information of user 10 stored in historical data 2222, such as name, address, phone number, and credit card number. Requiring user 10 to enter personal information during initial setup is unfriendly and unpleasant for the user. In the intelligent agent system 2500 of this embodiment, instead of requiring user 10 to enter personal information during initial setup, it first stores the personal information obtained through dialogue with user 10, and reads and utilizes it as needed. This avoids causing unpleasant feelings for the user and improves user convenience.

[0542] The intelligent agent system 2500 performs dialogue processing, for example, through the following steps 1 to 6.

[0543] (Step 1) The agent system 2500 sets the role of the agent. Specifically, the role setting unit 2276 sets the role of the agent when the agent system 2500 and the user 10 are in conversation, according to the specification from the user 10.

[0544] (Step 2) The intelligent agent system 2500 acquires the user 10's state, the user 10's emotion value, the intelligent agent's emotion value, and historical data 2222, including the voice or text input by the user 10. Specifically, it performs the same processing as steps S100 to S103 above to acquire the user 10's state, the user 10's emotion value, the intelligent agent's emotion value, and historical data 2222, including the voice or text input by the user 10.

[0545] (Step 3) The agent system 2500 determines the content of the agent's speech.

[0546] Specifically, the behavior determination unit 2236 generates the speech content of the intelligent agent by taking the text or voice input by the user 10, the emotions of both the user 10 and the role determined by the emotion determination unit 2232, and the dialogue history input text generation model stored in the historical data 2222.

[0547] For example, in the text or voice input by user 10, the emotions of both user 10 and the role determined by the emotion determination unit 2232, and the dialogue history stored in the historical data 2222, a fixed sentence such as "At this moment, as an agent, how should I respond?" is added, and the text is input into the article generation model to obtain the agent's speech content.

[0548] As an example, if user 10 inputs the text or voice message "I hope to make a reservation for a delicious Chinese restaurant nearby at 7 pm tonight", the AI's speech would be: "Got it" and "These are the recommended restaurants. 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD".

[0549] Additionally, if user 10 inputs the text or voice as "Fourth DDDD", the agent's response will be "Understood. Try to make a reservation. How many seats are there?"

[0550] (Step 4) The agent system 2500 outputs the agent's speech content.

[0551] Specifically, the behavior control unit 2250 synthesizes a voice corresponding to the character set by the character setting unit 2276, and outputs the speech content of the intelligent agent based on the synthesized voice.

[0552] (Step 5) The agent system 2500 determines whether it is the right time to execute the agent's command.

[0553] Specifically, the behavior determination unit 2236 determines whether it is an opportune time to execute the agent's command based on the output of the article generation model. For example, if the output of the article generation model includes an agent executing a command, it determines that it is an opportune time to execute the agent's command and proceeds to step 6. On the other hand, if it is determined that it is not an opportune time to execute the agent's command, it returns to step 2 as described above.

[0554] (Step 6) The intelligent agent system 2500 executes the intelligent agent's commands.

[0555] Specifically, the command acquisition unit 2272 acquires commands from the voice or text emitted by user 10 through dialogue with user 10. Then, the RPA 2274 performs actions based on the commands acquired by the command acquisition unit 2272. For example, if the command is "retrieve information," the system uses the retrieval query and API (Application Programming Interface) obtained through dialogue with user 10 to retrieve information from a search website. The behavior determination unit 2236 inputs the retrieval results into the article generation model to generate the agent's speech content. The behavior control unit 2250 synthesizes a voice corresponding to the role set by the role setting unit 2276 and outputs the agent's speech content based on the synthesized voice.

[0556] Additionally, when the command is "Reserve a store," the system uses reservation information obtained through dialogue with user 10, store information at the reservation destination, and API to make a reservation by calling the store at the reservation destination via telephone software. At this time, the behavior determination unit 2236 uses a dialogue-enabled text generation model to obtain the agent's speech content based on the voice input from the other party. Then, the behavior determination unit 2236 inputs the store's reservation result (whether the reservation was completed) into the text generation model and generates the agent's speech content. The behavior control unit 2250 synthesizes a voice corresponding to the character set by the character setting unit 2276 and outputs the agent's speech content based on the synthesized voice.

[0557] Then, return to step 2 above.

[0558] In step 6, the results of the actions performed by the agent (e.g., booking a store) are also stored in historical data 2222. The results of the actions performed by the agent stored in historical data 2222 are used by the agent system 500 to understand the interests or preferences of user 10. For example, if user 10 makes multiple bookings for the same store, it may be identified as a preference for that store, or the booking details such as the booking time slot, package content, or cost may be used as a basis for choosing a store for the next booking.

[0559] Therefore, the intelligent agent system 2500 can perform dialogue processing and, as needed, take actions related to utilizing service providers.

[0560] Figure 9F and Figure 9G An example of the operation of the intelligent agent system 2500 is shown. Figure 9F This example demonstrates how an intelligent agent system 2500 can book a hotel by conversing with user 10. Figure 9FIn the diagram, the left side represents the agent's speech, and the right side represents the user's speech. The agent system 2500 can understand the user's preferences based on the dialogue history with the user, provide a list of recommended restaurants that match the user's preferences, and make reservations for the selected restaurants.

[0561] on the other hand, Figure 9G This example illustrates how an intelligent agent system 2500 can access a telecommunications sales website to purchase goods through a dialogue with user 10. Figure 9G In the diagram, the left side represents the agent's statements, and the right side represents the user's statements. The agent system 2500 can estimate the remaining battery power of a beverage stored by the user based on their conversation history, and then suggest and execute the purchase of that beverage. Furthermore, the agent system 2500 can understand the user's preferences based on their past conversation history and recommend snacks they like. In this way, the agent system 2500 assists the user's daily life by acting as an agent similar to a butler, communicating with the user while performing various actions such as restaurant reservations and purchase transactions.

[0562] Furthermore, the other structures and functions of the intelligent agent system 2500 in the fourth embodiment are the same as those of the robot 100 in the second embodiment, so the description is omitted.

[0563] [Fifth Implementation Method] In the fifth embodiment, the above-described intelligent agent system is applied to smart glasses. Furthermore, for parts that have the same structure as in the first to fourth embodiments, the same symbols are used and descriptions are omitted.

[0564] Figure 9H This is a functional block diagram of an intelligent agent system 2700 that utilizes some or all of the functions of a behavior control system.

[0565] like Figure 9I As shown, the smart glasses 2720 are eyeglass-type smart devices, worn by the user 10 just like regular glasses. The smart glasses 2720 is an example of an electronic device and wearable terminal.

[0566] The smart glasses 2720 includes an intelligent agent system 2700. A display included in the control object 2252B displays various information to the user 10. The display is, for example, an LCD. The display is, for example, located in the lens portion of the smart glasses 2720, and the displayed content can be visually recognized by the user 10. A speaker included in the control object 2252B outputs sound representing various information to the user 10. The smart glasses 2720 includes a touch panel (not shown), which accepts input from the user 10.

[0567] The accelerometer 2206, temperature sensor 2207, and heartbeat sensor 2208 of the sensor unit 2200B detect the state of user 10. Furthermore, these sensors are just one example; other sensors can certainly be installed to detect the state of user 10.

[0568] Microphone 2201 acquires sounds emitted by user 10 or ambient sounds around smart glasses 2720. 2D camera 2203 is capable of capturing images of the surroundings of smart glasses 2720. 2D camera 2203 is, for example, a CCD camera.

[0569] The sensor module 2210B includes a voice emotion recognition unit 2211 and a speech understanding unit 2212. The communication processing unit 2280 of the control unit 2228B is responsible for communication between the smart glasses 2720 and the outside world.

[0570] Figure 9I This diagram illustrates an example of how smart glasses 2720 utilizes the intelligent agent system 2700. Smart glasses 2720 provides various services to user 10 using the intelligent agent system 2700. For example, when user 10 operates smart glasses 2720 (e.g., inputting sound into the microphone or touching the touch panel with a finger), smart glasses 2720 begins to utilize the intelligent agent system 2700. Here, utilizing the intelligent agent system 2700 includes smart glasses 2720 having and utilizing the intelligent agent system 2700, and also includes a portion of the intelligent agent system 2700 (e.g., sensor module 2210B, storage unit 2220, control unit 2228B) located externally to smart glasses 2720 (e.g., a server), and smart glasses 2720 utilizes the intelligent agent system 2700 through communication with the external system.

[0571] A point of contact is established between the agent system 2700 and the user 10 by the user operating the smart glasses 2720. That is, the agent system 2700 begins to provide services. As described in the third embodiment, in the agent system 2700, the role of the agent (e.g., the role of Audrey Hepburn) is set by the role setting unit 2276.

[0572] The emotion determination unit 2232 determines the emotion value of user 10 and the emotion value of the intelligent agent itself. Here, the emotion value of user 10 is estimated by various sensors included in the sensor unit 2200B mounted on the smart glasses 2720. For example, if the heart rate of user 10 detected by the heartbeat sensor 2208 increases, the emotion value such as "anxiety" or "fear" is estimated to a greater extent.

[0573] Furthermore, based on the user's body temperature measured by the temperature sensor 2207, for example, if the temperature exceeds the average body temperature, the emotional value of "pain" or "difficulty" can be estimated to a greater extent. Additionally, for example, if the accelerometer 2206 detects that the user 10 is performing some kind of movement, the emotional value of "happiness" can be estimated to a greater extent.

[0574] Alternatively, for example, the emotional value of user 10 can be inferred based on the voice or speech content of user 10 acquired by the microphone 2201 mounted on the smart glasses 2720. For example, if user 10's voice is rude, an emotional value such as "anger" can be inferred to a greater extent.

[0575] When the emotion value estimated by the emotion determination unit 2232 is higher than a predetermined value, the intelligent agent system 2700 causes the smart glasses 2720 to acquire information related to the surrounding situation. Specifically, for example, the 2D camera 2203 captures images or videos representing the surrounding situation of the user 10 (e.g., people or objects present in the surroundings). Additionally, the microphone 2201 records ambient sounds. Other information related to the surrounding situation may include date, time, location information, or weather information. This information related to the surrounding situation, along with the emotion value, is stored in historical data 2222. Historical data 2222 can be implemented using an external cloud storage device. Thus, the surrounding situation obtained by the smart glasses 2720, in a state of association with the current emotion value of the user 10, is stored in historical data 2222 as a so-called lifelog.

[0576] In the intelligent agent system 2700, information representing the surrounding situation is associated with emotional values ​​and stored in historical data 2222. Thus, the intelligent agent system 2700 acquires personal information about user 10, such as interests, hobbies, or personality. For example, if an image representing a baseball game viewing situation is associated with emotional values ​​such as "happy" or "joyful," and user 10's interest is watching baseball, the intelligent agent system 2700 can determine their favorite team or player based on the information stored in historical data 2222.

[0577] Then, when the intelligent agent system 2700 is conversing with user 10 or performing actions towards user 10, it also considers the content of the surrounding conditions stored in historical data 2222 to determine the content of the conversation or the content of the action. In addition to the surrounding conditions, the conversation history stored in historical data 2222 can also be added as described above to determine the content of the conversation or the content of the action.

[0578] As described above, the behavior determination unit 2236 generates speech content based on the article generated by the article generation model. Specifically, the behavior determination unit 2236 inputs the text or voice input by user 10, the emotions of both user 10 and the agent determined by the emotion determination unit 2232, the dialogue history stored in the historical data 2222, and the personality of the agent into the article generation model to generate the agent's speech content. Furthermore, the behavior determination unit 2236 inputs the surrounding conditions stored in the historical data 2222 into the article generation model to generate the agent's speech content.

[0579] The generated speech content is output to the user 10 via a speaker mounted on the smart glasses 2720. In this case, a synthesized voice corresponding to the agent's role is used as the sound. The behavior control unit 2250 generates a synthesized voice by reproducing the voice quality of the agent's role (e.g., Audrey Hepburn), or by generating a synthesized voice corresponding to the role's emotion (e.g., an amplified voice in the case of "anger"). Alternatively, the speech content can be displayed on a screen either in place of the sound output or together with the sound output.

[0580] RPA 2274 executes operations corresponding to commands (e.g., commands obtained from a smart agent via voice or text spoken by user 10 in a conversation with user 10). RPA 2274 performs actions related to utilizing service providers, such as retrieving information, making store reservations, arranging tickets, purchasing goods / services, paying for goods, providing directions, and translating.

[0581] Additionally, as another example, RPA 2274 performs the operation of sending content input by user 10 (e.g., a child) through voice input in a dialogue with an intelligent agent to another party (e.g., a parent). Examples of sending units include messaging applications, chat applications, or email applications.

[0582] When the RPA 2274 performs an operation, for example, a sound indicating the completion of the operation is output by a speaker mounted on the smart glasses 2720. For example, a sound such as "Store reservation completed" is output to user 10. Alternatively, for example, if store reservations are full, a sound such as "Unable to make a reservation. What should I do?" is output to user 10.

[0583] As described above, various services are provided to user 10 in smart glasses 2720 by using intelligent agent system 2700. Furthermore, since smart glasses 2720 are worn by user 10, intelligent agent system 2700 can be used in various settings such as home, workplace, and travel destination.

[0584] Furthermore, since the smart glasses 2720 are worn by user 10, it is suitable for collecting user 10's so-called life log. Specifically, based on the detection results from various sensors mounted on the smart glasses 2720 or the recording results from the 2D camera 2203, user 10's emotional value is estimated. Therefore, user 10's emotional value can be collected in various situations, and the intelligent agent system 2700 can provide services or speech content suitable for user 10's emotions.

[0585] Furthermore, in the smart glasses 2720, the surrounding environment of user 10 is acquired through a 2D camera 2203, microphone 2201, etc. This surrounding environment is then correlated with user 10's emotional state. This allows for the estimation of user 10's current state and emotional state. Consequently, the accuracy of the intelligent agent system 2700 in understanding user 10's interests and preferences is improved. Moreover, by accurately understanding user 10's interests and preferences, the intelligent agent system 2700 can provide services or content tailored to those interests and preferences.

[0586] Additionally, the intelligent agent system 2700 can also be applied to other wearable terminals (pencils, smartwatches, earrings, bracelets, hairbands, etc., electronic devices worn on the body of user 10). When the intelligent agent system 2700 is applied to a smart pendant, a speaker, acting as the controlled object 2252B, outputs sounds representing various information to user 10. The speaker is, for example, a speaker capable of outputting directional sound. The speaker is configured to be directional, pointing towards user 10's ear. This prevents sound from reaching people other than user 10. The microphone 2201 acquires sounds emitted by user 10 or ambient sounds around the smart pendant. The smart pendant is worn around user 10's neck. Therefore, the smart pendant is positioned relatively close to user 10's mouth during wear. This makes it easy to acquire sounds emitted by user 10.

[0587] Furthermore, in the above embodiment, the case where robot 100 uses user 10's facial image to identify user 10 has been described, but the disclosed technology is not limited to this method. For example, robot 100 can identify user 10 using user 10's voice, user 10's email address, user 10's SNS ID, or ID card with built-in wireless IC tag held by user 10.

[0588] Robot 100 is an example of an electronic machine equipped with a behavior control system. The application of the behavior control system is not limited to robot 100; it can be applied to various electronic machines. Furthermore, the functions of server 300 can be implemented using more than one computer. At least some of the functions of server 300 can be implemented using virtual machines. Additionally, at least some of the functions of server 300 can be implemented using the cloud.

[0589] Figure 4 An example of a hardware structure that functions as a computer 1200, which is a smartphone 50, a robot 100, a server 300, and an intelligent agent system 2500, 2700, is shown in a schematic diagram.

[0590] The present disclosure has been described above using embodiments, but the technical scope of the present disclosure is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. Based on the claims, it is also apparent that adding such modifications or improvements may also be included within the technical scope of the present disclosure.

[0591] It should be noted that the execution order of operations, sequences, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, description, and drawings can be implemented in any order unless specifically indicated by "before," "prerequisite," or the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, description, and drawings is described using terms such as "firstly" or "next" for convenience, it does not mean that it must be implemented in that order.

[0592] The following notes are also disclosed regarding the above implementation methods.

[0593] (Note 1) A behavior control system, comprising: The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on the dialogue function that enables the user to converse with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content. The behavior determination unit performs the following processing: Processing of learning materials, including textbooks; Based on the set target standard score and the read learning materials, the process of creating questions and presenting them to the user; and If the user answers the question posed to the user correctly, a new question is created and posed to the user based on a standard score higher than the target standard score and the learning materials read.

[0594] (Note 2) A behavior control system, comprising: The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on the dialogue function that enables the user to converse with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content. The emotion determination unit determines the emotion of the ward-side user based on reading information, including at least audio information, of a book being read aloud by the guardian-side user (classified as the guardian) to the ward-side user (classified as the ward). The behavior determination unit determines the reader's reaction during reading based on the emotions of the user being read. If the user's reaction is good, the unit suggests books similar to the book being read. If the user's reaction is bad, the unit suggests information related to books of a different category than the book being read.

[0595] (Note 3) According to the behavior control system described in Appendix 2, the robot imitates the voice of the guardian-side user and performs readings thereafter based on the voice information of the guardian-side user obtained by the emotion determination unit.

[0596] (Note 4) According to the behavior control system described in Appendix 2, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0597] (Note 5) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit is configured to switch between a first mode in which it can freely converse with the user and a second mode in which it teaches things to the user.

[0598] (Note 6) According to the behavior control system described in Appendix 5, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0599] (Note 7) According to the behavior control system described in Appendix 6, the controlled machine is a loudspeaker. The plush toy is equipped with a microphone or camera.

[0600] (Note 8) According to the behavior control system described in Appendix 7, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0601] (Note 9) According to the behavior control system described in Appendix 6, a wireless power receiver is disposed inside the plush toy to receive wireless power from an external wireless power supply unit. The controlled object machine or the robot receives power via the wireless power receiver.

[0602] (Postscript 10) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit determines the robot's behavior to maximize the emotional value representing the intensity of emotions that are important to the user according to the purpose of the dialogue.

[0603] (Postscript 11) According to the behavior control system described in Appendix 10, the emotion valued is a sense of accomplishment or growth when the purpose is to learn, a sense of security when the purpose is to negotiate, and a sense of happiness when the purpose is to engage in physical activity or conversation.

[0604] (Postscript 12) The behavior control system according to Note 10 or Note 11, wherein the purpose is a learning-related purpose.

[0605] (Postscript 13) According to the behavior control system described in Appendix 10 or Appendix 11, the user's reaction corresponding to the robot's behavior is fed back to the article generation model.

[0606] (Postscript 14) According to the behavior control system described in Appendix 10 or Appendix 11, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0607] (Postscript 15) According to the behavior control system described in Appendix 14, the controlled object machine is a loudspeaker. The plush toy is equipped with a microphone or camera.

[0608] (Postscript 16) According to the behavior control system described in Appendix 15, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0609] (Postscript 17) According to the behavior control system described in Appendix 14, a wireless power receiver is disposed inside the plush toy to receive wireless power from an external wireless power supply unit. The controlled object machine or the robot receives power via the wireless power receiver.

[0610] (Postscript 18) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit obtains the user's proficiency in the foreign language based on the content of the foreign language statement sent by the user to the robot, and sets the foreign language statement to be sent to the user based on the proficiency and the content of the statement.

[0611] (Postscript 19) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit is configured to provide learning assistance to the user based on the user's sensory characteristics.

[0612] (Postscript 20) According to the behavior control system described in Appendix 19, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0613] (Postscript 21) According to the behavior control system described in Appendix 20, the controlled machine is a loudspeaker. The plush toy is equipped with a microphone or camera.

[0614] (Postscript 22) According to the behavior control system described in Appendix 21, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0615] (Postscript 23) According to the behavior control system described in Appendix 20, a wireless power receiver is disposed inside the plush toy to receive wireless power from an external wireless power supply unit. The controlled object machine or the robot receives power via the wireless power receiver.

[0616] (Postscript 24) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit determines the behavior as a state of possessing a character suitable for the purpose of the dialogue.

[0617] (Postscript 25) According to the behavior control system described in Appendix 24, the purpose of the dialogue is education, and the role is a role suitable for the teaching subject to be carried out in the education.

[0618] (Postscript 26) According to the behavior control system described in Appendix 24 or Appendix 25, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0619] (Postscript 27) According to the behavior control system described in Appendix 26, the controlled machine is a loudspeaker. The plush toy is equipped with a microphone or camera.

[0620] (Postscript 28) According to the behavior control system described in Appendix 27, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0621] (Postscript 29) According to the behavior control system described in Appendix 26, a wireless power receiver is disposed inside the plush toy to receive wireless power from an external wireless power supply unit. The controlled object machine or the robot receives power via the wireless power receiver.

[0622] (Note 30) A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the user's or the robot's emotions; and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit suggests learning content based on the user's preferences related to their strengths and weaknesses.

[0623] (Postscript 31) According to the behavior control system described in Appendix 30, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0624] (Note 32) According to the behavior control system described in Appendix 31, the controlled object machine is a loudspeaker. The plush toy is equipped with a microphone or camera.

[0625] (Postscript 33) According to the behavior control system described in Appendix 32, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0626] (Postscript 34) According to the behavior control system described in Appendix 31, a wireless power receiver is disposed inside the plush toy to receive wireless power from an external wireless power supply unit. The controlled object machine or the robot receives power via the wireless power receiver.

[0627] (Postscript 35) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine; and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's actions include discussing things that the user cares about. When the behavior determination unit determines that talking about the user's concerns is an action of the electronic device, it determines the speech content related to the event data whose emotional value meets a predetermined benchmark.

[0628] (Postscript 36) According to the behavior control system described in Appendix 35, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0629] (Postscript 37) According to the behavior control system described in Appendix 36, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0630] (Postscript 38) According to the behavior control system described in Appendix 36 or Appendix 37, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0631] (Postscript 39) According to the behavior control system described in Appendix 36 or Appendix 37, the robot is an intelligent agent for conversing with the user.

[0632] (Postscript 40) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. The emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit determines the behavior of the electronic device based on at least one of the user's state, the state of the electronic device, the user's emotion, and the electronic device's emotion. When the behavior determination unit determines that taking meeting minutes is the behavior of the electronic machine... The system obtains the user's speech content through voice recognition, identifies the speaker through voiceprint authentication, and obtains the speaker's emotion based on the judgment result of the emotion determination unit. Generate deliberation record data that represents a combination of user speech content, speaker identification results, and the speaker's emotions.

[0633] (Postscript 41) According to the behavior control system described in Appendix 40, when the behavior determination unit determines that taking minutes is the behavior of the electronic machine, it also uses a text generation model with dialogue functionality to generate a summary of the text representing the minutes data.

[0634] (Postscript 42) According to the behavior control system described in Appendix 41, the behavior determination unit, when determining that taking minutes is an behavior of the electronic machine, also uses a text generation model with dialogue functionality to generate a list of things the user should do, contained in the summary.

[0635] (Postscript 43) According to the behavior control system described in Appendix 42, the behavior determination unit determines that taking minutes is the behavior of the electronic machine.

[0636] (Postscript 44) According to the behavior control system described in Appendix 42, when the behavior determination unit determines that taking minutes is an action of the electronic machine, it further sends a message to the user confirming the action to be taken based on the list.

[0637] (Postscript 45) According to the behavior control system described in Appendix 40, the electronic machine is a robot.

[0638] (Postscript 46) According to the behavior control system described in Appendix 45, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0639] (Postscript 47) According to the behavior control system described in Appendix 45, the robot is an intelligent agent for conversing with the user.

[0640] (Postscript 48) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. The emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include posing questions to the user. When the behavior determination unit determines that presenting a question to the user is an action of the electronic machine, it generates a question to present to the user.

[0641] (Postscript 49) According to the behavior control system described in Appendix 48, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0642] (Postscript 50) According to the behavior control system described in Appendix 49, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0643] (Postscript 51) According to the behavior control system described in Appendix 49 or Appendix 50, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0644] (Postscript 52) According to the behavior control system described in Appendix 49 or 50, the robot is an intelligent agent for conversing with the user.

[0645] (Postscript 53) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. The emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include teaching music. The behavior determination unit evaluates the sounds emitted by the user when it determines that the teaching music is an action of the electronic machine.

[0646] (Postscript 54) According to the behavior control system described in Appendix 53, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0647] (Postscript 55) According to the behavior control system described in Appendix 54, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0648] (Postscript 56) According to the behavior control system described in Appendix 54 or Appendix 55, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on a plush toy.

[0649] (Postscript 57) According to the behavior control system described in Appendix 54 or Appendix 55, the robot is an intelligent agent for conversing with the user.

[0650] (Postscript 58) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. The emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include posing questions to the user. When the behavior determination unit determines that presenting a question to the user is an action of the electronic machine, it generates a question that matches the user based on the content of the text used by the user and the user's target standard score.

[0651] (Postscript 59) According to the behavior control system described in Appendix 58, when the behavior determination unit determines that the user is in an idle state or is being reprimanded by the user's guardian to study as the user's emotion, it generates a question that matches the user's emotions.

[0652] (Postscript 60) According to the behavior control system described in Appendix 58, the behavior determination unit generates questions that increase the difficulty of the answer when the user is able to answer the question.

[0653] (Postscript 61) According to the behavior control system described in Appendix 58, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0654] (Postscript 62) According to the behavior control system described in Appendix 61, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0655] (Postscript 63) According to the behavior control system described in Appendix 61 or Appendix 62, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0656] (Postscript 64) According to the behavior control system described in Appendix 61 or 62, the robot is an intelligent agent for conversing with the user.

[0657] (Postscript 65) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine; and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's functions include: monitoring the child's conversations and actions; analyzing strengths and weaknesses and characteristics; and quantifying the child's musical, scientific, artistic, and English abilities into scores. When the behavior determination unit determines that the proposed action is an action of the electronic machine, it proposes an action to the child or parent.

[0658] (Postscript 66) According to the behavior control system described in Appendix 65, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0659] (Postscript 67) According to the behavior control system described in Appendix 66, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0660] (Postscript 68) According to the behavior control system described in Appendix 66 or Appendix 67, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0661] (Postscript 69) According to the behavior control system described in Appendix 66 or 67, the robot is an intelligent agent for conversing with the user.

[0662] (Postscript 70) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. The emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include estimating the user's mental age. When the behavior determination unit determines that the user's mental age is determined to be the behavior of the electronic machine, it presumes the user's mental age based on the user's behavior.

[0663] (Postscript 71) According to the behavior control system described in Appendix 70, the behavior determination unit stores the user's behavior as historical data when it determines the estimated mental age of the user as the behavior of the electronic machine, and estimates the user's mental age based on the user's behavior recorded in the historical data.

[0664] (Postscript 72) According to the behavior control system described in Appendix 70 or Appendix 71, the machine actions include taking into account the user's mental age. When the behavior determination unit determines the behavior of the electronic machine in response to the estimated mental age of the user, it considers the user's mental age as a factor in determining the behavior of the electronic machine.

[0665] (Postscript 73) According to the behavior control system described in Appendix 70, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0666] (Postscript 74) According to the behavior control system described in Appendix 73, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0667] (Postscript 75) According to the behavior control system described in Appendix 73 or Appendix 74, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0668] (Postscript 76) According to the behavior control system described in Appendix 73 or Appendix 74, the robot is an intelligent agent for conversing with the user.

[0669] (Postscript 77) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine; and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's actions include estimating the user's foreign language proficiency and conversing with the user in a foreign language. When the behavior determination unit determines that the user's foreign language proficiency is the behavior of the electronic machine, it presupposes the user's foreign language proficiency; when it determines that conversing with the user in a foreign language is the behavior of the electronic machine, it engages in conversation with the user in a foreign language.

[0670] (Postscript 78) According to the behavior control system described in Appendix 77, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0671] (Postscript 79) According to the behavior control system described in Appendix 78, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking about the robot's behavior, into the article generation model, and determines the robot's behavior based on the output of the article generation model.

[0672] (Postscript 80) According to the behavior control system described in Appendix 78 or Appendix 79, the robot is mounted on a plush toy or connected wirelessly or wiredly to a controlled object machine mounted on the plush toy.

[0673] (Postscript 81) According to the behavior control system described in Appendix 78 or Appendix 79, the robot is an intelligent agent for conversing with the user.

[0674] (Postscript 82) A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine; and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine actions include making suggestions related to reading aloud. When the behavior determination unit determines that making a suggestion related to reading is an action of the electronic machine, it generates a suggestion related to reading based on the collected information related to reading and according to the prescribed suggestion conditions, and provides the suggestion to the user.

[0675] (Postscript 83) According to the behavior control system described in Appendix 82, when making suggestions related to the reading as a machine action, the party performing the reading is designated as the first user, and the party receiving the reading is designated as the second user, and information related to the reading is collected from the users regarding the first user and the second user respectively.

[0676] (Postscript 84) According to the behavior control system described in Appendix 83, when making suggestions related to the reading as part of the machine's actions, the content or summary of the reading is used as information related to the reading.

[0677] (Postscript 85) According to the behavior control system described in Appendix 83, when making suggestions related to the reading as part of the machine's actions, the emotions of the second user when receiving the reading are used as the collected reading-related information.

[0678] (Postscript 86) According to the behavior control system described in Appendix 83, when making suggestions related to the reading as a machine action, as a condition for the suggestion, at least a frequency is set, and then a condition is set for using at least one of the reading frequency of the first user and the emotional tendency of the second user.

[0679] (Postscript 87) According to the behavior control system described in Appendix 82, the electronic machine is a robot. The behavior determination unit determines any one of a variety of robot behaviors, including not performing any behavior, as the robot's behavior.

[0680] (Postscript 88) According to the behavior control system described in Appendix 87, the behavior determination model is an article generation model with dialogue functionality. The behavior determination unit inputs text r...

Claims

1. A behavior control system, comprising: The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on the dialogue function that enables the user to converse with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content. The behavior determination unit performs the following processing: Processing of learning materials, including textbooks; Based on the set target standard score and the read learning materials, the process of creating questions and presenting them to the user; and If the user answers the question posed to the user correctly, a new question is created and posed to the user based on a standard score higher than the target standard score and the learning materials read.

2. A behavior control system, comprising: The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on the dialogue function that enables the user to converse with the robot, generates robot behavior content based on the user's behavior and the user's or robot's emotions, and determines the robot's behavior corresponding to the behavior content. The emotion determination unit determines the emotion of the ward-side user based on reading information, including at least audio information, of a book being read aloud by the guardian-side user (classified as the guardian) to the ward-side user (classified as the ward). The behavior determination unit determines the reader's reaction during reading based on the emotions of the user being read. If the user's reaction is good, the unit suggests books similar to the book being read. If the user's reaction is bad, the unit suggests information related to books of a different category than the book being read.

3. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit is configured to switch between a first mode in which the user can freely converse with the user and a second mode in which the user is taught things.

4. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit determines the robot's behavior to maximize the emotional value representing the intensity of emotions that are important to the user according to the purpose of the dialogue.

5. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or the robot's emotions. The behavior determination unit obtains the user's proficiency in the foreign language based on the content of the foreign language sentences spoken by the user to the robot, and sets the sentences to be spoken to the user in the foreign language based on the proficiency and the sentence content.

6. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit is configured to provide learning assistance to the user based on the user's sensory characteristics.

7. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit determines the behavior as a state of possessing a character suitable for the purpose of the dialogue.

8. A behavior control system, comprising: The user status recognition unit identifies user status, including user behavior. The emotion determination department determines the emotions of the user or the robot. and The behavior determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's behavior corresponding to the user's state and the user's or robot's emotions. The behavior determination unit suggests learning content based on the user's preferences related to their strengths and weaknesses.

9. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, at a predetermined time to determine any one of a variety of machine actions, including not performing any action, as the behavior of the electronic machine. and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's actions include discussing things that the user cares about. When the behavior determination unit determines that talking about the user's concerns is an action of the electronic device, it determines the speech content related to the event data whose emotional value meets a predetermined benchmark.

10. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit determines the behavior of the electronic device based on at least one of the user's state, the state of the electronic device, the user's emotion, and the electronic device's emotion. When the behavior determination unit determines that taking meeting minutes is the behavior of the electronic machine... The system obtains the user's speech content through voice recognition, identifies the speaker through voiceprint authentication, and obtains the speaker's emotion based on the judgment result of the emotion determination unit. Generate deliberation record data that represents a combination of user speech content, speaker identification results, and the speaker's emotions.

11. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include posing questions to the user. When the behavior determination unit determines that presenting a question to the user is an action of the electronic machine, it generates a question to present to the user.

12. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include teaching music. The behavior determination unit evaluates the sounds emitted by the user when it determines that the teaching music is an action of the electronic machine.

13. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include posing questions to the user. When the behavior determination unit determines that presenting a question to the user is an action of the electronic machine, it generates a question that matches the user based on the content of the text used by the user and the user's target standard score.

14. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, at a predetermined time to determine any one of a variety of machine actions, including not performing any action, as the behavior of the electronic machine. and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's functions include: monitoring the child's conversations and actions; analyzing strengths and weaknesses and characteristics; and quantifying the child's musical, scientific, artistic, and English abilities into scores. When the behavior determination unit determines that the proposed action is an action of the electronic machine, it proposes an action to the child or parent.

15. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and The behavior determination unit, at a predetermined time, uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, to determine any one of a variety of machine actions, including inaction, as the behavior of the electronic machine. The machine's actions include estimating the user's mental age. When the behavior determination unit determines that the user's mental age is determined to be the behavior of the electronic machine, it presumes the user's mental age based on the user's behavior.

16. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, at a predetermined time to determine any one of a variety of machine actions, including not performing any action, as the behavior of the electronic machine. and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine's actions include estimating the user's foreign language proficiency and conversing with the user in a foreign language. When the behavior determination unit determines that the user's foreign language proficiency is the behavior of the electronic machine, it presupposes the user's foreign language proficiency; when it determines that conversing with the user in a foreign language is the behavior of the electronic machine, it engages in conversation with the user in a foreign language.

17. A behavior control system, comprising: The status recognition unit identifies the user status, including user behavior, and the status of electronic devices. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; The behavior determination unit uses at least one of the user state, the state of the electronic machine, the user's emotion, and the electronic machine's emotion, along with a behavior determination model, at a predetermined time to determine any one of a variety of machine actions, including not performing any action, as the behavior of the electronic machine. and The storage control unit stores event data, including the emotion value determined by the emotion determination unit and data including the user's behavior, in historical data. The machine actions include making suggestions related to reading aloud. When the behavior determination unit determines that making a suggestion related to reading is an action of the electronic machine, it generates a suggestion related to reading based on the collected information related to reading and according to the prescribed suggestion conditions, and provides the suggestion to the user.

Images