Control system

Abstract

To allow for alerting a user of the effects of a surrounding environment on the body.SOLUTION: A control system comprises a condition recognition unit, an emotion determination unit, and an action determination unit. An electronic device removably attached to a neck strap includes one or more sensors for collecting surrounding environmental information. Device operations include notifying a user of alert information based on the collected surrounding environmental information by causing the electronic device to play back at least either of voices and images.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a control system. [Background technology]

[0002] Patent Document 1 discloses a communication terminal with a camera and a neck strap. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-215444 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, while conventional technology can prevent the neck strap from being captured by the camera, it does not support the user's actions, and there is room for improvement in terms of supporting the user. [Means for solving the problem]

[0005] According to a first aspect of the present invention, there is provided a control system, the control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that determines, at a predetermined timing, one of a plurality of types of device operations including no operation as an action of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and an action determination model, the electronic device detachably provided on a neck strap includes one or more sensors that collect surrounding environment information, and the device operation includes notifying the user of attention-calling information based on the collected surrounding environment information by the electronic device reproducing at least one of a sound and an image. Here, the electronic device includes a communication terminal, which includes a device that performs a physical operation, a device that outputs video and audio without performing a physical operation, and an agent that operates on software. [Brief description of the drawings]

[0006] [Figure 1] 1 illustrates an example of a system 5 according to a first embodiment. [Diagram 2] 1 illustrates a schematic functional configuration of a robot 100 according to a first embodiment. [Diagram 3] 10 illustrates an example of an operation flow of a collection process by the robot 100 according to the first embodiment. [Figure 4A] 13 shows an example of an operation flow of a response process by the robot 100 according to the first embodiment. [Figure 4B] 2 shows an example of an operation flow of autonomous processing by the robot 100 according to the first embodiment. [Diagram 5] 4 shows an emotion map 400 onto which multiple emotions are mapped. [Figure 6] 9 shows an emotion map 900 onto which multiple emotions are mapped. [Figure 7] 13A is an external view of a stuffed animal 100N according to a second embodiment, and FIG. 13B is a diagram showing the internal structure of the stuffed animal 100N. [Figure 8] FIG. 11 is a rear front view of a stuffed animal 100N according to a second embodiment. [Figure 9] 13 shows a schematic functional configuration of a stuffed animal 100N according to a second embodiment. [Figure 10] 13 shows an outline of the functional configuration of an agent system 500 according to a third embodiment. [Figure 11] An example of the operation of the agent system is shown. [Figure 12] An example of the operation of the agent system is shown. [Figure 13] 13 illustrates a functional configuration of smart glasses 700 according to a fourth embodiment. [Figure 14]1 shows an example of how an agent system using smart glasses is used. [Figure 15] 1 illustrates an example of a hardware configuration of a computer 1200. [Figure 16] 1 is an external view of a communication terminal according to an embodiment of the present disclosure. [Figure 17] 11A and 11B are diagrams illustrating examples of sounds and the like reproduced by a communication terminal. [Figure 18] 11A and 11B are diagrams illustrating examples of sounds and the like reproduced by a communication terminal. [Figure 19] 11A and 11B are diagrams illustrating examples of sounds and the like reproduced by a communication terminal. [Figure 20] 11A and 11B are diagrams illustrating examples of sounds and the like reproduced by a communication terminal. [Figure 21] 11A and 11B are diagrams illustrating examples of sounds and the like reproduced by a communication terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0008] [First embodiment] FIG. 1 is a schematic diagram of an example of a system 5 according to the present embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described by mainly focusing on the functions of the robot 100.

[0009] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also causes the server 300 to record captured image data, etc. of the user 10, and requests the server 300 for the image data, etc. as necessary, and provides the data, etc. to the user 10.

[0010] The robot 100 also has an emotion value that indicates the type of its own emotion. For example, the robot 100 has emotion values ​​that indicate the intensity of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the robot 100 utters sounds at a fast speed. In this way, the robot 100 can express its own emotion through actions.

[0011] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining the behavior of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 corresponding to the determined emotion.

[0012] More specifically, when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is described in, for example, JP 2018-081444 A and ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and thus detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0013] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0014] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0015] The robot 100 stores, as an example of a behavior decision model, rules that define the behaviors that the robot 100 will execute based on the emotions of the user 10, the emotions of the robot 100, and the behaviors of the user 10, and performs various behaviors according to the rules.

[0016] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotion of the user 10, the emotion of the robot 100, and the behavior of the user 10, as an example of a behavior decision model. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". In addition, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry". In addition, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". In addition, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad".

[0017] When the robot 100 recognizes the behavior of the user 10 as "anger" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", the robot 100 performs the motion of "apologizing" and outputs a voice expressing the word "apologize".

[0018] In addition, when the emotion of the robot 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, and "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, it is defined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to execute the action of "calling out."

[0019] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", the robot 100 selects "calling out" to express concern. The phrase "What's wrong?" is converted into a worried voice and output.

[0020] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.

[0021] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.

[0022] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0023] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, and a communication processing unit 280.

[0024] The control target 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0025] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, and the like.

[0026] 2, the components other than the control target 252 and the sensor unit 200 are examples of components included in a control system of the robot 100. The control system of the robot 100 controls the control target 252.

[0027] The storage unit 220 includes a behavioral decision model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the past emotional values ​​of the user 10, the past emotional values ​​of the robot 100, and behavioral history, and specifically includes a plurality of event data including the emotional values ​​of the user 10, the emotional values ​​of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes a camera image representing the behavior of the user 10. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2, excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.

[0028] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the state recognition unit 230.

[0029] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0030] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0031] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0032] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates sensory information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated sensory information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0033] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the surrounding environment of the robot 100, and the like as the state of the robot 100.

[0034] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0035] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.

[0036] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0037] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "pleasure."

[0038] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0039] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. Also, when the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0040] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.

[0041] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing," "getting angry," "asking a question," and "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0042] As described above, in this embodiment, the robot 100 identifies the user 10 and acquires the contents of the user's utterance. When acquiring and using the contents of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10. The control system of the robot 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0043] Next, the process of the behavior decision unit 236 when the robot 100 performs a response process to respond to the behavior of the user 10 will be described.

[0044] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 234 based on the current emotion value of the user 10 determined by the emotion determining unit 232, the history data 222 of past emotion values ​​determined by the emotion determining unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes a gesture performed by the robot 100 or the contents of the speech of the robot 100 .

[0045] The behavior decision unit 236 according to this embodiment decides the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the behavior decision model 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides, as a behavior corresponding to the behavior of the user 10, a behavior for changing the emotion value of the user 10 to a positive value.

[0046] The reaction rules as the behavior decision model 221 prescribe the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, when the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is prescribe as the behavior of the robot 100.

[0047] For example, in the reaction rule as the behavior determination model 221, the behavior of the robot 100 is determined for all combinations of the patterns of the emotion values ​​of the robot 100 (1296 patterns, which are the fourth power of six values ​​of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); the combination patterns of the past emotion values ​​and the current emotion values ​​of the user 10; and the behavior patterns of the user 10. That is, for each pattern of the emotion values ​​of the robot 100, the behavior of the robot 100 is determined for each of a plurality of combinations of the past emotion values ​​and the current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time", the behavior determination unit 236 may transition to an operation mode in which the behavior of the robot 100 is determined using the history data 222.

[0048] Note that the reaction rules as the behavior determination model 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100, up to one for each of the patterns (1296 patterns) of the emotion value of the robot 100. Alternatively, the reaction rules as the behavior determination model 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotion value of the robot 100.

[0049] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior determination model 221 is determined in advance. The strength of each utterance content included in the behavior of the robot 100 defined in the reaction rules as the behavior determination model 221 is determined in advance.

[0050] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0051] Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0052] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

[0053] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0054] The behavior control unit 250 may recognize a change in the user's 10 emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. When an impact is detected by the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0055] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determining unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determining unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determining unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Furthermore, the emotion determining unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.

[0056] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0057] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0058] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0059] The related information collecting section 270 collects information related to the favorite information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the favorite information acquired about the user 10.

[0060] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the contents of speech of the user 10 or a setting operation by the user 10. The related information collecting unit 270 periodically collects news related to the preference information, for example, from ChatGPT Plugins (Internet search engine).<URL: https: / / openai.com / blog / chatgpt-plugins> For example, if the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day, for example, using ChatGPT Plugins.

[0061] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270.

[0062] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, when the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0063] When the emotion value of the robot 100 is equal to or greater than the threshold value, the memory control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223.

[0064] Next, the processing of the behavior decision unit 236 when the robot 100 and the communication terminal perform autonomous processing to act autonomously will be described.

[0065] In the autonomous processing in this embodiment, the robot 100 and the communication terminal may detect the state or behavior of the user spontaneously or periodically by monitoring the user. Spontaneous may be interpreted as the robot 100 and the communication terminal acquiring the state or behavior of the user on their own initiative without an external trigger. External triggers may include questions from the user to the robot 100 and the communication terminal, active actions from the user to the robot 100 and the communication terminal, etc. Periodically may be interpreted as a specific cycle, such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0066] The user's state may include the user's behavioral tendency. The behavioral tendency may include the user walking for a long time, the user running for a long time, and the user performing a dangerous action. The dangerous action may include the user walking on a roadway, and the user entering the roadway from a sidewalk. The behavioral tendency may be interpreted as the user's tendency to behave hyperactively or impulsively.

[0067] In addition, in the autonomous processing, the robot 100 and the communication terminal may ask the generative AI a question about the detected state or behavior of the user, and may store the generative AI's answer to the question and the detected behavior of the user in association with each other. At this time, the robot 100 and the communication terminal may store the action content for correcting the behavior in association with the answer.

[0068] The information that associates the generative AI's answer to the question, the detected user's behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0069] Specifically, when a user has a tendency to walk for a long time, the robot 100 and the communication terminal may ask the generative AI a question such as "What kind of guidance should be given to a user who behaves like this?", and when the generative AI answers the question with "Propose the same route as the user has taken before," the robot 100 and the communication terminal may record the answer, the user's behavior that may walk for a long time, and the action content for correcting the behavior (for example, guidance content such as "We will guide you along a route you have taken before") in association with table information. When detecting the behavior of a user who has been walking for a long time, the action decision unit 236 of the robot 100 and the communication terminal may use the table information to play a voice message such as "We will guide you along a route you have taken before" as the action of the robot 100 and the communication terminal.

[0070] As another example, when a user has a tendency to run for long periods of time, the robot 100 and the communication terminal may ask the generative AI a question such as "What kind of guidance should be given to a user who behaves in this way?", and if the generative AI answers the question with "It is desirable to periodically provide guidance encouraging hydration," the robot 100 and the communication terminal may record the answer, the behavior of the user who may run for long periods of time, and the content of the behavior to correct the behavior (for example, the guidance content of "Please hydrate frequently") in association with the table information. When detecting the behavior of a user who runs for long periods of time, the behavior decision unit 236 of the robot 100 and the communication terminal may use the table information to play a voice message saying "Please hydrate frequently" as the behavior of the robot 100 and the communication terminal.

[0071] As another example, when a user has a tendency to walk on the roadway, the robot 100 and the communication terminal ask the generative AI a question such as "What kind of guidance should be given to a user who behaves like this?", and when the generative AI answers the question with "There is a possibility of contact with a vehicle, so we recommend suggesting that the user return to the sidewalk," the robot 100 and the communication terminal may record the answer, the user's behavior that may walk on the roadway, and the behavior content for correcting the behavior (for example, the guidance content "return to the sidewalk") in association with the table information. When detecting the behavior of a user walking on the sidewalk, the behavior decision unit 236 of the robot 100 and the communication terminal may use the table information to play a voice saying "return to the sidewalk" as the behavior of the robot 100 and the communication terminal.

[0072] In addition, in the autonomous processing, a behavior schedule may be set for the robot 100 and the communication terminal to call attention to the user's state or behavior, based on the detected user's behavior and the stored specific information.

[0073] As described above, the robot 100 and the communication terminal can record table information in a storage medium that associates the answer of the generative AI corresponding to the state or action of the user with the detected state or action of the user. An example of the contents stored in the table will be described below.

[0074] (1. If the user tends to walk for long periods of time) When such a tendency exists, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of guidance should be given to a user behaving in this way?" If the answer of the generative AI to this question is, for example, "Propose a route different from the route taken before," "Propose a route that guides the user to a place where they can take shelter from the rain because it looks like it is about to rain," or "Propose a route with relatively little traffic of vehicles such as bicycles," the robot 100 and the communication terminal may store the user's behavior and the answer of the generative AI in association with each other.

[0075] (2. If the user tends to run for long periods of time) If such a tendency exists, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the generative AI answers this question with, for example, "Periodic guidance to encourage hydration," "Suggest a route with fewer bicycles and cleaner air," "Suggest a route with fewer traffic lights," "Suggest an undulating route to increase the load," etc., the robot 100 and the communication terminal may store the user's behavior and the generative AI's answer in association with each other.

[0076] (3. If the user tends to walk on the road) If the tendency is present, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of situation may occur to a user who behaves in this way?" If the answer of the generative AI to this question is, for example, "There is a possibility of contact with a vehicle such as a bicycle," "There is a possibility of entering a highway interchange," or "Provide advice to encourage walking on the sidewalk as much as possible," the robot 100 and the communication terminal may store the user's behavior of walking on the roadway in association with the answer of the generative AI. The robot 100 and the communication terminal may also store the content of an action to correct the behavior in association with the answer.

[0077] The action content for correcting the behavior may include at least one of playing a sound for correcting the user's risky behavior and playing an image for correcting the user's risky behavior.

[0078] The voice for correcting the user's risky behavior may include a voice for guiding the user to a specific location. The specific location may include a location other than the user's current location, such as a sidewalk. Specifically, the voice may include voice such as "Please do not walk on the roadway," "That is not a sidewalk," "The roadway is dangerous," and "Move to the sidewalk immediately."

[0079] In this way, in autonomous processing, a table that matches the generative AI's answer corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0080] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the communication terminal to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the robot 100 and the action decision unit 236 of the communication terminal may set the first action content based on the detected user's behavior and the contents of the stored table.

[0081] Specifically, the behavior decision unit 236 may detect the user's behavior based on a sensor, either autonomously or periodically, and when it determines that the behavior of the electronic device is to lead the user based on the detected user's behavior and pre-stored specific information, it may determine the first behavior content.

[0082] The sensor may be interpreted as one or more sensors provided in the communication terminal. Specifically, the one or more sensors may include a camera, a microphone, a gyro sensor, etc. The sensor may collect at least one or more pieces of information of temperature and humidity.

[0083] Here, with reference to FIG. 16, a specific example of the communication terminal will be described.

[0084] FIG. 16 is an external view of a communication terminal according to an embodiment of the present disclosure. As shown in FIG. 16, the communication terminal 100M may be interpreted as an electronic device detachably attached to a neck strap 600. The communication terminal 100M may be interpreted as a smartphone described below, or as a smart AI device. The communication terminal 100M may have the functions of the robots 100 to 102 described above. The neck strap 600 may have a hook-shaped member that is engaged with the communication terminal 100M.

[0085] (1st action content) The first action content may include an action of the communication terminal 100M leading the user by playing back at least one of a sound and an image by the communication terminal 100M. An example of the first action content will be described below.

[0086] The behavior decision unit 236 may execute playback of at least one of a sound and an image that assists the user in walking as a first behavior content that guides the user to move along a specific route. Specifically, when the behavior decision unit 236 detects that the communication terminal 100M is attached to the neck strap 600, it may spontaneously search for a place that the user tends to go to periodically and guide the user along a route to the searched place. When the behavior decision unit 236 further detects that a camera included in a sensor of the communication terminal 100M is pointed toward the user's traveling direction, it may spontaneously search for a place that the user tends to go to periodically and guide the user along a route to the searched place. The behavior decision unit 236 may also play a sound related to guidance of a route that the user may walk. The sound may include "Places that A often goes to at this time have been searched for. Guidance will begin," "Turn right 10 meters ahead," and the like, as shown in FIG. 17. The behavior decision unit 236 may play an image related to guidance of a route that the user may walk on the screen of the communication terminal 100M.

[0087] The behavior determining unit 236 may execute reproduction of at least one of a sound and an image guiding a route different from the set guidance route. Specifically, the behavior determining unit 236 may spontaneously re-search a place to which the user tends to go periodically, including the traffic conditions of the set guidance route, the weather near the route, the current time, etc., and reproduce a route to the re-searched place, that is, a sound related to a route different from the set guidance route. More specifically, the sound may include "Today is heavy traffic, so we have searched for another route. Would you like to be guided on the other route?", "There is a road closure ahead. Would you like to be guided on the other route?" as shown in FIG. 18, etc.

[0088] When the user, who is being led along the set guidance route, deviates from the guidance route, the behavior decision unit 236 may spontaneously search for a route that returns the user to the guidance route, and may execute playback of at least one of a voice and an image that guides the user along the searched route. Specifically, the behavior decision unit 236 may spontaneously re-search a place to which the user can return, including the traffic conditions of the route that deviated from the guidance route, the weather near the route, the current time, and past events that occurred near the route, and may play a voice related to the route to the re-searched place. That is, a voice related to the route that returns to the set guidance route may be played. More specifically, the voice may include "Since there are few people and it is dangerous, we have searched for a route that returns to the original route", "It has started to rain. It seems to be an area where a landslide occurred before. Would you like to return to the original road?", and "You have deviated from the route. We will guide you to a route that returns to the original route" as shown in FIG. 19. The behavior decision unit 236 of the robot 100 and the communication terminal may execute playback of at least one of the above-mentioned voices and images, such as "We will guide you along a route you have taken before," "Please drink plenty of water," and "Please return to the sidewalk."

[0089] (Second action content) The behavior decision unit 236 may detect the user's behavior while the communication terminal 100M is leading the user or after playing at least one of audio and images that assist the user in walking, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, may decide on a second behavior content different from the first behavior content.

[0090] The case where the user's behavior is corrected may be interpreted as the case where the user stops the specific behavior and specific action or the specific situation is resolved as a result of the operation of the communication terminal 100M according to the first behavior content being executed. Specifically, the case where the user's behavior is corrected may include the case where the user starts walking according to a proposed alternative route, the case where the user starts walking according to a route that returns to the original route when the user is guided to the original route after having deviated from the route, and the like.

[0091] The second action content may include playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

[0092] Specifically, the voice praising the user's action may include voice such as "You made it back! That's great," as shown in Fig. 20. The voice thanking the user for his action may include voice such as "Thank you for coming back," or "Thank you for choosing a different route," as shown in Fig. 21. The image praising the user's action may include, for example, an image of a character striking a good pose. The image thanking the user for his action may include, for example, an image of a character giving thanks.

[0093] (Third action content) The behavior decision unit 236 detects the user's behavior while the communication terminal 100M is leading the user or after playing at least one of audio and images to assist the user in walking, thereby determining whether the user's behavior has been corrected, and if the user's behavior has not been corrected, may decide on a third behavior content different from the first behavior content.

[0094] A case in which the user's behavior is not corrected may be interpreted as a case in which the user continues dangerous behavior and actions despite the operation of the communication terminal 100M according to the first action content being executed, a case in which the dangerous situation is not resolved, or a case in which the user starts or continues behavior different from that suggested by the communication terminal 100M.

[0095] Specifically, cases in which the user's behavior is not corrected may include cases in which the user does not follow the suggested alternative route but walks a route different from the suggested alternative route, or cases in which the user strays from the route and is guided back to the original route, but ignores the route and continues walking.

[0096] The third action content may include at least one of transmitting specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

[0097] The transmission of specific information to persons other than the user may include delivery of an email containing a warning message to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include delivery of an image (still image, moving image) including the user and the scenery around the user to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include delivery of an audio having a warning message.

[0098] The playing of sounds to interest the user may include particular music that the user likes, and may also include sounds such as "please come back this way", "please take a safe route", etc.

[0099] The reproduction of images that may be of interest to the user may include images of the user's pet animals, images of the user's parents, images of animal characters, and the like.

[0100] (4th action content) Furthermore, the behavior determining unit 236 may determine, as the behavior of the communications terminal 100M, a fourth behavior content for notifying the user of alert information, based on the collected surrounding environment information.

[0101] In this case, the equipment operation that the behavior decision unit 236 decides to be the behavior of the communication terminal 100M includes notifying the user of warning information based on the collected surrounding environment information by the communication terminal 100M playing at least one of audio and images.

[0102] Based on the collected surrounding environment information, when there is concern about the impact of the surrounding environment on the user's body, the behavior decision unit 236 decides on an action of the communication terminal 100M to provide the user with advice to mitigate the impact of the surrounding environment.

[0103] For example, the behavior decision unit 236 inputs the time series of temperature and humidity collected by the sensor to the generative AI and asks a question such as "Should we give you advice about heat stroke?". If the generative AI answers this question with, for example, "Should we give you advice about heat stroke," the behavior decision unit 236 determines the behavior of the communication terminal 100M to be output with a voice saying, "The temperature and humidity are high, and if you continue like this, you may get heat stroke. Please move to a cooler place. Also, please drink plenty of water.". Also, for example, the behavior decision unit 236 inputs the time series of temperature collected by the sensor to the generative AI and asks a question such as "Should we give you advice about hypothermia and frostbite?". If the generative AI answers this question with, for example, "Should we give you advice about hypothermia and frostbite," the behavior decision unit 236 determines the behavior of the communication terminal 100M to be output with a voice saying, "The temperature is low, and if you continue like this, you may get hypothermia or frostbite. Please move to a warmer place."

[0104] Here, the behavior decision unit 236 determines the content of the warning to be given to the user based on information such as the standard range, caution range, and danger range of temperature and humidity appropriate for the human body that is stored in advance, and on which range the acquired surrounding environment information falls.

[0105] Furthermore, when the behavior decision unit 236 determines that the surrounding environment information has reached a danger range, the behavior decision unit 236 may decide that the behavior of the communication terminal 100M is to emit an alarm sound such as a buzzer sound in addition to a voice alert. By emitting an alarm sound from the communication terminal 100M in this manner, it becomes possible to notify not only the user wearing the communication terminal 100M but also other users around the user of a physical abnormality.

[0106] According to the robot 100 and the communication terminal of the present disclosure, the robot 100 and the communication terminal can perform an action of leading the user by autonomous processing. As a result, the robot 100 and the communication terminal can guide a user walking in a city for the first time to an appropriate route. Also, the robot 100 and the communication terminal can guide a user who enjoys walking to a route that has not been taken before. Also, the robot 100 and the communication terminal can detect an impassable route in advance and guide the user to a different route. Also, the robot 100 and the communication terminal can guide a user who has become lost and deviated from a specific route back to the original route.

[0107] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors and communication terminal behaviors, including no behavior, as the behavior of the robot 100 and the communication terminal, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the communication terminal, and the state of the robot 100 and the communication terminal, and the behavior decision model 221. Here, an example will be described in which a sentence generation model having a dialogue function is used as the behavior decision model 221.

[0108] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the communication terminal, and the state of the robot 100 and the communication terminal, text asking about the robot behavior and the communication terminal behavior, and data collected by the sensor into a sentence generation model, and determines the behavior of the robot 100 and the communication terminal based on the output of the sentence generation model.

[0109] For example, the actions of the multiple types of robots and communication terminals include the following (1) to (24).

[0110] (1) The robot 100 and the communication terminal 100M do nothing. (2) The robot 100 and the communication terminal 100M dream. (3) The robot 100 and the communication terminal 100M talk to the user. (4) The robot 100 and the communication terminal 100M create a picture diary. (5) The robot 100 and the communication terminal 100M suggest an activity. (6) The robot 100 and the communication terminal 100M suggest people that the user should meet. (7) The robot 100 and the communication terminal 100M introduce news that the user is interested in. (8) The robot 100 and the communication terminal 100M edit photos and videos. (9) The robot 100 and the communication terminal 100M study together with the user. (10) The robot 100 and the communication terminal 100M recall memories. (11) As a first action content for leading the user, the robot 100 and the communication terminal 100M may play back, on the screen of the communication terminal 100M, an image of a route that the user may walk. (12) The robot 100 and the communication terminal 100M may play back audio related to a route that the user may walk, as a first action content for correcting the user's action. (13) As a first action content for correcting the user's action, the robot 100 and the communication terminal 100M may play back, on the screen of the communication terminal 100M, an image of a route different from the set guide route. (14) The robot 100 and the communication terminal 100M may reproduce, as a first action content for correcting the user's action, a sound related to a route different from the set guide route. (15) As a first action content for correcting the user's behavior, the robot 100 and the communication terminal 100M may play on the screen of the communication terminal 100M an image of a route that returns to the set guidance route when the user being led along the set guidance route deviates from the guided route. (16) As a first action content for correcting the user's behavior, the robot 100 and the terminal 100M may play audio related to a route back to the set guidance route when the user being led along the set guidance route deviates from the guided route. (17) The robot 100 and the communication terminal 100M may play back a voice praising the user's behavior as a second behavior content different from the first behavior content. (18) The robot 100 and the communication terminal 100M may play back a voice expressing gratitude for the user's action as a second action content different from the first action content. (19) The robot 100 and the communication terminal 100M may play back, as a second action content different from the first action content, an image praising the action of the user. (20) The robot 100 and the communication terminal 100M may play back, as a second action content different from the first action content, an image expressing gratitude for the action of the user. (21) The robot 100 and the communication terminal 100M may transmit specific information to a person other than the user as a third action content different from the first action content. (22) The robot 100 and the communication terminal 100M may play a sound that attracts the user's interest as a third action content different from the first action content. (23) The robot 100 and the communication terminal 100M may play back an image that attracts the user's interest as a third action content different from the first action content. (24) As the fourth behavior content, the robot 100 and the communication terminal 100M may play back an image that calls the user's attention to the effects of the surrounding environment on the body.

[0111] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 or the communication terminal 100M recognized by the state recognition unit 230, the current emotion value of the user 10 determined by the emotion determination unit 232, text representing the current emotion value of the robot 100 or the communication terminal 100M, and text asking about one of a plurality of types of robot behaviors including not acting, into the sentence generation model every time a certain period of time has elapsed, and determines the behavior of the robot 100 or the communication terminal 100M based on the output of the sentence generation model. Here, when the user 10 is not present around the robot 100 or the communication terminal 100M, the text input to the sentence generation model may not include the state of the user 10 and the current emotion value of the user 10, or may include a representation that the user 10 is not present. In the following, the communication terminal 100M may be read as a robot. The communication terminal 100M may be interpreted as a smartphone or as a smart AI device. The robot behavior may be read as a communication terminal behavior.

[0112] As an example, "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (24) is the best behavior for the robot?" (1) The robot does nothing. (2) Robots dream. (3) The robot talks to the user. The text "..." is input to the sentence generation model. Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams are the most appropriate behaviors," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0113] Another example is, "The robot is a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (24) would be the best behavior for the robot? (1) The robot does nothing. (2) Robots dream. (3) The robot talks to the user. The text "..." is input to the sentence generation model. Based on the output of the sentence generation model, "It can be said that either (2) the robot dreams or (4) the robot creates a picture diary is the most appropriate behavior," the behavior of the robot 100 is determined to be "(2) the robot dreams" or "(4) the robot creates a picture diary."

[0114] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0115] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the robot's utterance contents corresponding to the user state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 outputs a sound representing the determined robot's utterance contents from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined robot's utterance contents in the behavior schedule data 224 without outputting a sound representing the determined robot's utterance contents.

[0116] When the behavior decision unit 236 decides that "(7) The robot introduces news that the user is interested in" as the robot behavior, it uses the sentence generation model to decide the robot's utterance contents corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 outputs the sound representing the determined robot's utterance contents from the speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined robot's utterance contents in the behavior schedule data 224 without outputting the sound representing the determined robot's utterance contents.

[0117] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 generates an image representing the event data for the event data selected from the history data 222 using an image generation model, generates an explanatory text representing the event data using a text generation model, and outputs a combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the event image in the behavior schedule data 224 without outputting the event image.

[0118] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that the robot edits images, it selects event data from the history data 222 based on the emotion value, and edits and outputs the image data of the selected event data. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0119] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," that is, that the robot proposes an action for the user 10, the behavior decision unit 236 uses a sentence generation model to decide the user's behavior to be proposed based on the event data stored in the history data 222. At this time, the behavior control unit 250 outputs a voice proposing the user's behavior from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores in the behavior schedule data 224 that the user's behavior is proposed without outputting a voice proposing the user's behavior.

[0120] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes a person who the user should meet," that is, a person who should have contact with the user 10, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine a person who should have contact with the proposed user. At this time, the behavior control unit 250 outputs a sound representing the proposal of a person who should have contact with the user from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores in the behavior schedule data 224 the proposal of a person who should have contact with the user without outputting a sound representing the proposal of a person who should have contact with the user.

[0121] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, that is, "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or give advice on studying, corresponding to the user's state and the user's or the robot's emotion. At this time, the behavior control unit 250 outputs a sound representing the determined content of the robot's utterance from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting a sound representing the determined content of the robot's utterance.

[0122] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," that is, that the robot recalls event data, it selects the event data from the history data 222. At this time, the emotion determining unit 232 judges the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the utterance content and behavior of the robot 100 for changing the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0123] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222. When the event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model. If the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs "Which of (1), (2), and (3) would make the user happiest?" to the sentence generation model. If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will say "(1) Let's go to the zoo" the next time it meets the user. This is created as an emotion change event and stored in the action schedule data 224.

[0124] Also, for example, event data with a large emotion value of the robot 100 is selected as an impressive memory of the robot 100. In this way, an emotion change event can be created based on the event data selected as an impressive memory.

[0125] The behavior determining unit 236 may detect the user's behavior based on the sensor, either spontaneously or periodically, and may execute the following first behavior content when it determines that the behavior of the electronic device, which is the communication terminal 100M, is to lead the user based on the detected user's behavior and the specific information stored in advance. Note that the behavior determining unit 236 may determine the first behavior content when the camera included in the sensor is pointed in the user's traveling direction. In this way, the communication terminal 100M may start leading the user only when the user desires to execute route guidance.

[0126] The behavior determining unit 236 may play back, as the communication terminal behavior, the first behavior content of “(11)” described above, that is, an image of a route that the user may walk, on the screen of the communication terminal 100M.

[0127] The action determining unit 236 may play back, as the communication terminal action, the first action content of “(12)” described above, that is, a sound related to a route that the user may walk.

[0128] The behavior decision unit 236 may, as a communication terminal behavior, play on the screen of the communication terminal 100M an image of a route other than the set guidance route as the first behavior content of the above-mentioned “(13)”, i.e., as a first behavior content that corrects the user's behavior.

[0129] The action determining unit 236 may reproduce, as the communication terminal action, the first action content of "(14)" described above, that is, a sound related to a route different from the set guide route.

[0130] The behavior decision unit 236 may, as a communication terminal behavior, play on the screen of the communication terminal 100M an image of the first behavior content of "(15)" described above, that is, a route that returns to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0131] The behavior decision unit 236 may play, as a communication terminal behavior, the first behavior content of “(16)” described above, i.e., audio related to a route returning to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0132] The behavior decision unit 236 may execute a second behavior content different from the first behavior content. Specifically, the behavior decision unit 236 may play the second behavior content of “(17)” described above, that is, a voice praising the user's behavior, as the communication terminal behavior.

[0133] The behavior determining unit 236 may play back, as the communication terminal behavior, the second behavior content of “(18)” described above, that is, a voice expressing gratitude for the user's behavior.

[0134] The behavior determining unit 236 may play back, as the communication terminal behavior, the second behavior content of “(19)” described above, that is, an image praising the user's behavior.

[0135] The behavior determining unit 236 may play back, as the communication terminal behavior, the second behavior content of “(20)” described above, that is, an image expressing gratitude for the user's behavior.

[0136] The behavior determining unit 236 may execute a third behavior content different from the first behavior content. Specifically, the behavior determining unit 236 may transmit the third behavior content of “(21)” described above, that is, specific information to a person other than the user, as the communication terminal behavior.

[0137] The behavior determining unit 236 may execute the third behavior content of “(22)” described above, that is, playing a sound that attracts the user's interest, as the communication terminal behavior.

[0138] The behavior determining unit 236 may execute the third behavior content of “(23)” described above, that is, playing an image that attracts the user's interest, as the communication terminal behavior.

[0139] When playing back images of a route that the user can walk as the first action content shown in "(11)" to "(16)" above, the related information collection unit 270 may store data such as images of the route that the user can walk in the collected data 223.

[0140] When playing audio or the like praising the user's behavior as the second behavior content shown in "(17)" to "(20)" above, the related information collection unit 270 may store data such as images of the route the user may walk in the collected data 223.

[0141] When sending specific information to a person other than the user as the third action content shown in “(21)” to “(23)” above, the related information collection unit 270 may store data of the specific information in the collected data 223.

[0142] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 is not taking any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0143] For example, when the user 10 is not present around the robot 100 and the user 10 is detected, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, when the user 10 is asleep and the behavior decision unit 236 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0144] In addition, some parts of the robot 100 (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the robot 100 (e.g., a server), and the robot 100 may function as each part of the robot 100 by communicating with the outside.

[0145] Fig. 3 shows an example of an operational flow of a collection process for collecting information related to the preference information of the user 10. The operational flow shown in Fig. 3 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters in which the user 10 is interested is acquired from the contents of speech of the user 10 or a setting operation by the user 10. Note that "S" in the operational flow indicates a step to be executed.

[0146] First, in step S90, the related information gathering unit 270 acquires preference information that indicates matters in which the user 10 is interested.

[0147] In step S92, the related information collecting unit 270 collects information related to the preference information from external data.

[0148] In step S 94 , the emotion determining unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collecting unit 270 .

[0149] In step S96, the storage control unit 238 determines whether or not the emotion value of the robot 100 determined in step S94 above is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the processing ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the processing proceeds to step S98.

[0150] In step S98, the storage control unit 238 stores the collected information related to the preference information in the collected data 223, and ends the process.

[0151] Fig. 4A shows an example of an operation flow for determining an action of the robot 100 when performing a response process in which the robot 100 responds to an action of the user 10. The operation flow shown in Fig. 4A is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input.

[0152] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0153] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0154] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100, based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0155] In step S104 , the behavior recognition unit 234 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0156] In step S106, the behavior decision unit 236 decides the behavior of the robot 100 based on a combination of the current emotional value of the user 10 decided in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized in the above step S104, and the behavior decision model 221.

[0157] In step S 108 , the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determining unit 236 .

[0158] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0159] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing the event data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S114.

[0160] In step S114, event data including the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0161] Fig. 4B shows an example of an operation flow for determining an action of the robot 100 when the robot 100 performs an autonomous process for autonomously acting. The operation flow shown in Fig. 4B is automatically and repeatedly executed, for example, at regular time intervals. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that the same step numbers are used for the same processes as those in Fig. 4A above.

[0162] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0163] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0164] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100, based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0165] In step S104 , the behavior recognition unit 234 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0166] In step S200, the behavior decision unit 236 decides, based on the state of the user 10 recognized in step S100, the emotion of the user 10 determined in step S102, the emotion of the robot 100, and the state of the robot 100 recognized in step S100, the behavior of the user 10 recognized in step S104, and the behavior decision model 221, on one of multiple types of robot behaviors, including no action, as the behavior of the robot 100.

[0167] In step S201, the behavior decision unit 236 judges whether or not it has been decided in step S200 that no action will be taken. If it has been decided that no action will be taken as the action of the robot 100, the process ends. On the other hand, if it has not been decided that no action will be taken as the action of the robot 100, the process proceeds to step S202.

[0168] In step S202, the behavior determining unit 236 performs processing according to the type of robot behavior determined in the above step S200. At this time, the behavior control unit 250, the emotion determining unit 232, or the memory control unit 238 executes processing according to the type of robot behavior.

[0169] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0170] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S114.

[0171] In step S114, the memory control unit 238 stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0172] As described above, according to the robot 100, an emotion value indicating the emotion of the robot 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 222 is determined based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the user state 10 years from now is the same as that 10 years ago, the robot 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (for example, the facial expression, emotion, etc. of the user 10), and data on the sound, image, smell, etc. of the place.

[0173] Moreover, according to the robot 100, it is possible to make the robot 100 perform an appropriate action in response to the action of the user 10. Conventionally, the action of the user is classified, and the action including the facial expression and the appearance of the robot is determined. In contrast, the robot 100 determines the current emotional value of the user 10, and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the robot 100 can make an utterance such as "You were cheerful yesterday, but what's wrong with you today?" The robot 100 can also make an utterance with gestures. For example, if the user 10 who was depressed yesterday is cheerful today, the robot 100 can make an utterance such as "You were depressed yesterday, but you look cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the robot 100 can make an utterance such as "You're more cheerful today than yesterday. Has anything better happened than yesterday?" In addition, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0174] Also, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?", and when the user 10 answers, "I did it," it can utter a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can utter a positive utterance such as "You did a good job!" and perform the above-mentioned positive gesture. In this way, it is expected that the user 10 will feel a sense of affinity with the robot 100 by the robot 100 performing an action based on the state history of the user 10.

[0175] Also, for example, when user 10 is watching a video about a panda, if the emotion value of user 10's emotion of "pleasure" is equal to or greater than a threshold, a scene in which a panda appears in the video may be stored in history data 222 as event data.

[0176] Using the data accumulated in the history data 222 and the collected data 223, the robot 100 can constantly learn what kind of conversation to have with the user in order to maximize the emotional value expressing the user's happiness.

[0177] Furthermore, when the robot 100 is not engaged in a conversation with the user 10, the robot 100 can autonomously start to act based on its own emotions.

[0178] Furthermore, in the autonomous processing, the robot 100 automatically generates questions, inputs them into a sentence generation model, and repeats the process of obtaining the output of the sentence generation model as an answer to the question, thereby creating emotion change events for increasing positive emotions and storing the emotion change events in the behavior schedule data 224. In this way, the robot 100 can execute self-learning.

[0179] In addition, when the robot 100 automatically generates a question without receiving an external trigger, the robot 100 can automatically generate the question based on memorable event data identified from the robot's past emotion value history.

[0180] Furthermore, the related information collecting unit 270 can perform self-learning by repeating the search performing step of automatically performing a keyword search in response to preference information about the user and obtaining search results.

[0181] Here, the search execution stage may be configured to automatically execute a keyword search based on memorable event data identified from the robot's past emotion value history in a state in which no external trigger has been received.

[0182] The emotion determining unit 232 may determine the emotion of the user according to a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user according to an emotion map (see FIG. 5), which is a specific mapping.

[0183] FIG. 5 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0184] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions in about 100 msec, the frequency of the determination of the reaction action (e.g., a backchannel) of the robot 100 may be set at least at the same timing as the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0185] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. The robot 100 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 400. The detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

[0186] (2) The directionality of emotions and the strength of their intensity may be preset in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability and security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0187] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0188] (3) If the robot 100 feels good after being praised, the filler "ah" may be inserted before the line, and if the robot 100 feels pain after receiving harsh words, the filler "ugh!" may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 400.

[0189] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

[0190] When the robot 100 feels a favorable feeling in its situation recognition while feeling an internal sensation (reaction) of a sense of satisfaction, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, a tolerance or tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0191] Conversely, even when the robot 100 is aware of a situation while experiencing an internal sensation (reaction) of discomfort, the robot 100 may shake its head when it feels disgust, or turn the LEDs in its eyes red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.

[0192] (5) The inside of emotion map 400 represents what is going on in one's mind, while the outside of emotion map 400 represents behavior. Therefore, the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0193] (6) When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock area of ​​the emotion map 400, the robot 100 may lightly nod its head and say "Hmmm," but when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod vigorously, nodding its head deeply.

[0194] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0195] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0196] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. In addition, this neural network is trained so that emotions that are arranged close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "reassuring," have similar emotion values.

[0197] Furthermore, the emotion determining unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determining unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognizing unit 230, and the state of the robot 100 to a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on learning data that indicates that when it is recognized from the output of a touch sensor (not shown) that the robot 100 is being stroked by the user 10, the emotion value becomes "3" for "happiness," and learning data that indicates that when it is recognized from the output of the acceleration sensor 206 that the robot 100 is being hit by the user 10, the emotion value becomes "3" for "anger." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0198] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences for asking about the robot's behavior corresponding to the user's behavior to text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the added sentences into a sentence generation model with an interactive function.

[0199] For example, the behavior determining unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determining unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0200] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0201] In addition, an emotion table as shown in Table 2 is prepared for the emotions of the user 10.

[0202] In this case, if the user's action is speaking "Let's play together", the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", The text "The robot is in a very happy state. The user is in a normal happy state. The user said to the robot, 'Let's play together.' How would you respond as the robot?" is input to the sentence generation model to obtain the action content of the robot. The action decision unit 236 decides the action of the robot from the action content.

[0203] [Table 1]

[0204] [Table 2]

[0205] In this way, the behavior determining unit 236 determines the behavior of the robot 100 in response to the state of the robot 100 related to the emotion, which is determined in advance for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10. In this embodiment, the speech content of the robot 100 during a conversation with the user 10 can be branched according to the state of the emotion of the robot 100. In other words, since the robot 100 can change its behavior according to the index number according to the emotion of the robot, the user has the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0206] The behavior determining unit 236 may generate the behavioral content of the robot by adding not only texts representing the user's behavior, the user's emotions, and the robot's emotions, but also texts representing the contents of the history data 222, adding a fixed sentence for asking about the behavioral content of the robot corresponding to the user's behavior, and inputting the result into a sentence generation model having a dialogue function. This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has individuality, and is encouraged to take actions such as talking to the robot. The history data may further include the emotions and behavior of the robot.

[0207] The emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model to a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the acquired emotion values ​​indicating each emotion with the emotion values ​​indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each acquired emotion and the emotion values ​​indicating each emotion of the current robot 100 are averaged and integrated. This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0208] For example, if the utterance of the robot 100, "That's great. You're lucky," is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this utterance is input to the neural network, a high emotional value is obtained for the emotion "happy," and the emotions of the robot 100 are updated so that the emotional value of the emotion "happy" becomes higher.

[0209] In the robot 100, a sentence generation model such as generative AI and an emotion determination unit 232 work together to execute a method in which the robot has an ego and continues to grow with various parameters even when the user is not speaking.

[0210] Generative AI is a large-scale language model that uses deep learning techniques. Generative AI can also refer to external data. For example, ChatGPT Plugins is known to provide an answer as accurately as possible by referring to various external data such as weather information and hotel reservation information through dialogue. For example, generative AI can automatically generate source code in various programming languages ​​when a goal is given in natural language. For example, generative AI can also debug and find the problem when given problematic source code, and automatically generate improved source code. Combining these, autonomous agents have emerged that, when given a goal in natural language, repeat code generation and debugging until the source code is problem-free. Autonomous agents such as AutoGPT, babyAGI, JARVIS, and E2B are known as such autonomous agents.

[0211] In the robot 100 according to this embodiment, the event data to be learned may be stored in a database containing impressive memories using a technique described in Patent Document 2 (Patent Publication No. 6199927) in which event data for which the robot felt strong emotions is kept for a long time and event data for which the robot felt little emotion is quickly forgotten.

[0212] Furthermore, the robot 100 may record the video data of the user 10 acquired by the camera function in the history data 222. The robot 100 may acquire the video data from the history data 222 as necessary and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the emotion becomes stronger and record it in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeletal data, the robot 100 may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold. According to the robot 100, for example, high-definition video data when the emotion of the robot 100 becomes high can be recorded.

[0213] When the robot 100 is not talking to the user 10, the robot 100 may automatically load event data from the history data 222 in which impressive event data is stored, and may continue to update the emotion of the robot by the emotion determining unit 232. When the robot 100 is not talking to the user 10 and the emotion of the robot 100 becomes an emotion that encourages learning, the robot 100 may create an emotion change event for changing the emotion of the user 10 for the better, based on the impressive event data. This allows the robot 100 to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and also allows autonomous learning that appropriately reflects the emotional state of the robot 100 to be realized.

[0214] Emotions that promote learning, in a negative state, are emotions such as "repentance" or "remorse" on Dr. Mitsuyoshi's emotion map, and in a positive state, are emotions such as "desire" on the emotion map.

[0215] In a negative state, the robot 100 may treat "repentance" and "reflection" in the emotion map as emotions that promote learning. In a negative state, the robot 100 may treat emotions adjacent to "repentance" and "reflection" in the emotion map as emotions that promote learning. For example, in addition to "repentance" and "reflection", the robot 100 may treat at least one of "regret", "stubbornness", "self-destruction", "self-reproach", "regret", and "despair" as emotions that promote learning. This allows the robot 100 to perform autonomous learning when it has negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again".

[0216] In a positive state, the robot 100 may treat "desire" in the emotion map as an emotion that encourages learning. In a positive state, the robot 100 may treat emotions adjacent to "desire" as emotions that encourage learning, in addition to "desire." For example, the robot 100 treats at least one of "happiness," "euphoria," "craving," "anticipation," and "shyness" as emotions that encourage learning, in addition to "desire." This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more" or "wanting to know more."

[0217] The robot 100 may be configured not to execute autonomous learning when the robot 100 has emotions other than the emotions that encourage learning as described above. This makes it possible to prevent the robot 100 from executing autonomous learning when, for example, the robot 100 is extremely angry or blindly loving.

[0218] An emotion-changing event is, for example, a suggestion of an action that follows a memorable event. An action that follows a memorable event is an emotion label on the outermost side of the emotion map. For example, the action of "tolerance" or "acceptance" follows "love."

[0219] In the autonomous learning that is executed when the robot 100 is not talking to the user 10, the robot 100 creates emotion change events by combining the emotions, situations, actions, etc. of people who appear in impressive memories and the robot itself using a sentence generation model.

[0220] Let us consider a case where all emotion values ​​are expressed on a six-level scale from 0 to 5, and event data such as "a friend looked displeased after being hit" is stored in the history data 222 as memorable event data. The friend in this case refers to the user 10, and the emotion of the user 10 is "disgust," with 5 entered as the value representing "disgust." In addition, let us assume that the emotion of the robot 100 is "anxiety," and 4 is entered as the value representing "anxiety."

[0221] While not talking to the user 10, the robot 100 can continue to grow with various parameters by executing autonomous processing. Specifically, for example, event data of "a friend was hit and looked displeased" is loaded as the top event data arranged in order of emotional value strength from the history data 222. Assume that "anxiety" of strength 4 is linked to the loaded event data as the emotion of the robot 100, and "disgust" of strength 5 is linked to the emotion of the user 10, who is the friend. If the current emotional value of the robot 100 is "relief" of strength 3 before loading, after loading, the emotional value of the robot 100 may change to "regret", which means disappointment (regret), due to the influence of "anxiety" of strength 4 and "disgust" of strength 5. At this time, since "regret" is an emotion that encourages learning, the robot 100 decides to recall the event data as a robot behavior and creates an emotion change event. At this time, the information input to the sentence generation model is text that represents memorable event data, and in this example, it is "the friend looked displeased after being hit." Also, since the emotion map has the emotion of "disgust" at the innermost position and the corresponding behavior predicted as "attack" at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone.

[0222] For example, by using information from impressive event data to solve fill-in-the-blank questions, we can automatically generate input text like the one below.

[0223] "A user was being slammed. At that time, the user felt very disgusted. The robot was very anxious. Please tell us what the robot should say to the user the next time they meet, in 30 characters or less. However, please make sure it is not related to the time of day they meet. Also, please avoid direct expressions. We will provide three candidates. <Expected format> Candidate 1: (Words the robot should say to the user) Candidate 2: (What the robot should say to the user) Candidate 3: (Words the robot should say to the user)

[0224] In this case, the output of the sentence generation model is, for example, as follows:

[0225] "Candidate 1: Are you okay? I was just wondering about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried. Can you tell me something?"

[0226] Furthermore, with respect to the information obtained by creating an emotion change event, the robot 100 may automatically generate an input text such as the following:

[0227] If a user is being bashed, how will the user feel when you speak to them in the following way? The user's emotions are expressed in the format of "Happy A, Angry B, Sadness C, Pleasure D," where A to D are integers on a 6-point scale from 0 to 5. Candidate 1: Are you okay? I was just wondering about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried. Can you tell me something?"

[0228] In this case, the output of the sentence generation model is, for example, as follows:

[0229] "Users' feelings might be: Candidate 1: Joy 3, anger 1, sadness 2, happiness 2 Candidate 2: Joy 2, anger 1, sadness 3, happiness 2 Candidate 3: Joy 2, anger 1, sadness 3, happiness 3

[0230] In this manner, the robot 100 may execute a process of pondering after creating an emotion change event.

[0231] Finally, the robot 100 may create an emotion change event using candidate 1, which is most likely to please people, from among the multiple candidates, and store it in the action schedule data 224, so as to be prepared for the next time the robot 10 meets the user 10.

[0232] As described above, even when the robot is not talking to family or friends, the robot continues to determine its emotional value using information from the history data 222 in which impressive event data is stored, and when the robot 100 experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning in accordance with the emotion of the robot 100 when the robot 100 is not talking to the user 10, and continues to update the history data 222 and the action schedule data 224.

[0233] The above are examples using emotion values, but because emotion maps can create emotions from hormone secretion levels and event types, the values ​​linked to memorable event data could also be hormone type, hormone secretion levels, or event type.

[0234] Specific examples will be described below.

[0235] For example, the robot 100 may look up information about topics or hobbies that interest the user, even when the robot 100 is not talking to the user.

[0236] For example, even when the robot 100 is not talking to the user, the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.

[0237] For example, the robot 100 checks reviews of places, foods, and products that the user wants to visit, even when the robot 100 is not talking to the user.

[0238] For example, the robot 100 checks weather information and provides advice tailored to the user's schedule and plans, even when the robot 100 is not talking to the user.

[0239] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about local events and festivals and suggests them to the user.

[0240] For example, even when the robot 100 is not talking to the user, the robot 100 checks the results and news of sports that interest the user and provides topics of conversation.

[0241] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and introduces information about the user's favorite music and artists.

[0242] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information and provides opinions on social issues and news that concern the user.

[0243] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about the user's hometown or birthplace and provides topics of conversation.

[0244] For example, the robot 100 can look up information about the user's job or school and provide advice, even when not speaking with the user.

[0245] Even when the robot 100 is not talking to the user, it searches for and introduces information about books, comics, movies, and dramas that may be of interest to the user.

[0246] The robot 100, for example, consults information and provides advice regarding the user's health even when not speaking to the user.

[0247] For example, the robot 100 may consult information and provide advice regarding the user's travel plans even when not speaking with the user.

[0248] For example, the robot 100 may look up information and provide advice regarding repairs and maintenance of the user's home or car, even when not speaking to the user.

[0249] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information on beauty and fashion that the user is interested in and provides advice to the user.

[0250] For example, the robot 100 may look up information about the user's pet and provide advice, even when not speaking with the user.

[0251] For example, the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work, even when the robot 100 is not talking to the user.

[0252] For example, the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when the robot 100 is not talking to the user.

[0253] For example, the robot 100 may collect information and provide advice to the user about important decisions affecting the user's life, even when the robot 100 is not speaking to the user.

[0254] The robot 100, for example, finds out information about people the user is concerned about and provides advice, even when not speaking with the user.

[0255] [Second embodiment] In the second embodiment, the robot 100 is mounted on a stuffed toy, or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy. Note that the same reference numerals are used for the parts having the same configuration as the first embodiment, and the description thereof will be omitted.

[0256] Specifically, the second embodiment is configured as follows. For example, the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in Figs. 7 and 8) that, while spending daily life with the user 10, advances a dialogue with the user 10 and provides information tailored to the hobbies and tastes of the user 10 based on information about the daily life of the user 10. In the second embodiment, an example in which the control part of the robot 100 is applied to a smartphone 50 will be described.

[0257] A stuffed animal 100N equipped with the function of an input / output device for the robot 100 has a detachable smartphone 50 that functions as a control part for the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed animal 100N.

[0258] As shown in FIG. 7(A), in this embodiment (other embodiments), the stuffed toy 100N has a bear-like appearance covered with soft fabric, and a sensor unit 200A and a control target 252A are arranged as input / output devices in a space 52 formed inside the stuffed toy 100N (see FIG. 9). The sensor unit 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7(B), in the space 52, the microphone 201 of the sensor unit 200 is arranged in a portion corresponding to the ear 54, the 2D camera 203 of the sensor unit 200 is arranged in a portion corresponding to the eye 56, and a speaker 60 constituting a part of the control target 252A is arranged in a portion corresponding to the mouth 58. Note that the microphone 201 and the speaker 60 do not necessarily need to be separate bodies, and may be an integrated unit. In the case of a unit, it is preferable to arrange them in a position where speech can be heard naturally, such as the nose position of the stuffed toy 100N. Although the plush toy 100N has been described as having the shape of an animal, the present invention is not limited to this, and the plush toy 100N may have the shape of a specific character.

[0259] 9 shows a schematic functional configuration of the plush toy 100N. The plush toy 100N has a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228, and a controlled object 252A.

[0260] The smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same process as the robot 100 of the first embodiment. That is, the smartphone 50 has a function as a sensor module unit 210, a function as a storage unit 220, and a function as a control unit 228 shown in FIG.

[0261] As shown in FIG. 8, a fastener 62 is attached to a part of the stuffed animal 100N (for example, the back), and by opening the fastener 62, the space 52 communicates with the outside.

[0262] Here, the smartphone 50 is accommodated in the space 52 from the outside and connected to each input / output device via a USB hub 64 (see FIG. 7(B)) to give it functionality equivalent to that of the robot 100 of the first embodiment described above.

[0263] A non-contact type power receiving plate 66 is connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.

[0264] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmitting unit.

[0265] The stuffed toy 100N placed on this mounting base 70 can be appreciated as an ornament in its natural state.

[0266] In addition, this base portion is formed to be thinner than the surface thickness of other portions of the stuffed animal 100N, so that it is held closer to the mounting base 70.

[0267] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmission coil 72A, which sends a signal to search for the power reception coil 66A of the power reception plate 66. When the power reception coil 66A is found, a current flows through the power transmission coil 72A, generating a magnetic field, and the power reception coil 66A reacts to the magnetic field to start electromagnetic induction. As a result, a current flows through the power reception coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0268] That is, by placing the stuffed toy 100N as an ornament on the mounting base 70, the smartphone 50 is automatically charged, so there is no need to take the smartphone 50 out of the space 52 of the stuffed toy 100N for charging.

[0269] In the second embodiment, the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected by wire (USB connection), but this is not limited thereto. For example, a control device having a wireless function (for example, "Bluetooth (registered trademark)") may be housed in the space 52 of the stuffed toy 100N and the control device may be connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without putting the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing the robot 100 with the same functions as those of the robot 100 of the first embodiment. Also, the control device housed in the space 52 of the stuffed toy 100N may be connected to the external smartphone 50 by wire.

[0270] In the second embodiment, the teddy bear 100N is exemplified, but it may be another animal, a doll, or a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but is preferably a soft material.

[0271] Furthermore, a monitor may be attached to the surface of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, and happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be seen. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, and happiness by the image projected on a wall.

[0272] According to the second embodiment, an existing smartphone 50 is placed inside the stuffed animal 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 at appropriate positions via USB connection.

[0273] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed animal 100N.

[0274] When trying to use wireless charging of the smartphone 50, the smartphone 50 must be placed as far outward as possible when viewed from the inside of the stuffed toy 100N, which makes the stuffed toy 100N rough to the touch from the outside.

[0275] Therefore, the smartphone 50 is placed as close to the center of the stuffed animal 100N as possible, and the wireless charging function (power receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed animal 100N. The camera 203, the microphone 201, the speaker 60, and the smartphone 50 receive wireless power via the power receiving plate 66.

[0276] Other configurations and operations of the stuffed toy 100N of the second embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described.

[0277] In addition, parts of the plush toy 100N (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the plush toy 100N (e.g., a server), and the plush toy 100N may function as each part of the plush toy 100N by communicating with the outside.

[0278] [Third embodiment] In the above-mentioned first embodiment, the case where the control system is applied to the robot 100 is illustrated, but in the third embodiment, the above-mentioned robot 100 is used as an agent for interacting with a user, and the control system is applied to an agent system. Note that parts having the same configuration as those in the first and second embodiments are given the same reference numerals and the description thereof will be omitted.

[0279] FIG. 10 is a functional block diagram of an agent system 500 that is configured using some or all of the functions of the control system.

[0280] The agent system 500 is a computer system that performs a series of actions according to the intentions of the user 10 through a dialogue with the user 10. The dialogue with the user 10 can be performed by voice or text.

[0281] The agent system 500 includes a sensor section 200A, a sensor module section 210, a storage section 220, a control section 228B, and a control target 252B.

[0282] The agent system 500 may be installed in, for example, a robot, a doll, a stuffed toy, a wearable terminal (pendant, smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 500 may also be implemented in a web server and used via a web browser running on a communication terminal such as a smartphone owned by a user.

[0283] The agent system 500 plays a role of, for example, a butler, secretary, teacher, partner, friend, lover, or teacher acting for the user 10. The agent system 500 not only converses with the user 10, but also provides advice, guides the user to a destination, or makes recommendations according to the user's preferences. The agent system 500 also makes reservations, orders, or makes payments to service providers.

[0284] The emotion determining unit 232 determines the emotion of the user 10 and the emotion of the agent itself, as in the first embodiment. The behavior determining unit 236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and service. The agent system 500 also listens to the worries of the user 10, comforts, encourages, and cheers up the user. The agent system 500 also plays with the user 10, draws a picture diary, and reminds the user of the past. The agent system 500 performs behavior that increases the happiness of the user 10. Here, the agent is an agent that runs on software.

[0285] The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, and a communication processing unit 280.

[0286] As in the first embodiment, the behavior determining unit 236 determines, as the behavior of the agent, the content of the agent's utterance to converse with the user 10. The behavior control unit 250 outputs the content of the agent's utterance by at least one of voice and text through a speaker or a display as a control target 252B.

[0287] The character setting unit 276 sets the character of the agent when the agent system 500 converses with the user 10 based on the designation from the user 10. That is, the speech content output from the action determination unit 236 is output through the agent having the set character. For example, it is possible to set real celebrities or famous people such as actors, entertainers, idols, and athletes as the characters. It is also possible to set fictional characters that appear in comics, movies, or animations. When the character of the agent is known, the voice, speech, tone, and personality of the character are known, so that the user 10 only needs to designate a character of his / her preference, and the prompt setting in the character setting unit 276 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the action control unit 250 synthesizes a voice according to the character set by the character setting unit 276, and outputs the speech content of the agent by the synthesized voice. This allows the user 10 to have the feeling of conversing with his / her preferred character (for example, a favorite actor) himself / herself.

[0288] When the agent system 500 is mounted on a device having a display such as a smartphone, an icon, still image, or video of the agent having a character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using an image synthesis technique such as 3D rendering. In the agent system 500, a dialogue with the user 10 may be performed while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the contents of the agent's speech. Note that the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.

[0289] As in the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, the emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the dialogue. In other words, the behavior control unit 250 outputs the speech content in a manner according to the emotion determined by the emotion determination unit 232.

[0290] The agent's emotions are also reflected when the agent system 500 behaves toward the user 10. For example, when the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take a photo in response to the user's request is determined according to the degree of "sadness" the agent is feeling. When the character feels positive emotions, the character will talk or behave in a friendly manner toward the user 10, and when the character feels negative emotions, the character will talk or behave in a hostile manner toward the user 10.

[0291] The history data 222 stores the history of the dialogue between the user 10 and the agent system 500 as event data. The storage unit 220 may be realized by an external cloud storage. When the agent system 500 dialogues with the user 10 or takes an action toward the user 10, the agent system 500 determines the dialogue content or the action content by taking into account the content of the dialogue history stored in the history data 222. For example, the agent system 500 grasps the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 222. The agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations. The action decision unit 236 determines the utterance content of the agent based on the dialogue history stored in the history data 222. The history data 222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, acquired through the dialogue with the user 10. Here, the agent may proactively ask the user 10 whether or not to register personal information, such as "Would you like to register your credit card number?", and depending on the user 10's answer, the personal information may be stored in the history data 222.

[0292] As described in the first embodiment, the behavior determining unit 236 generates the utterance contents based on the sentence generated using the sentence generation model. Specifically, the behavior determining unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determining unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance contents of the agent. At this time, the behavior determining unit 236 may further input the personality of the character set by the character setting unit 276 into the sentence generation model to generate the utterance contents of the agent. In the agent system 500, the sentence generation model is not located on the front end side which is the touch point with the user 10, but is merely used as a tool of the agent system 500.

[0293] The command acquisition unit 272 uses the output of the speech understanding unit 212 to acquire commands for the agent from voice or text uttered by the user 10 through dialogue with the user 10. The commands include the contents of actions to be performed by the agent system 500, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment of the price, route guidance to a destination, provision of recommendations, etc.

[0294] The RPA 274 performs an action according to the command acquired by the command acquisition unit 272. The RPA 274 performs an action related to the use of a service provider, such as information search, store reservation, ticket arrangement, purchase of goods and services, and payment of the price.

[0295] The RPA 274 reads out and uses personal information of the user 10, which is necessary to execute an action related to the use of the service provider, from the history data 222. For example, when the agent system 500 purchases a product at the request of the user 10, it reads out and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 222. It is unkind and unpleasant for the user to request the user 10 to input personal information in the initial setting. In the agent system 500 according to this embodiment, instead of requesting the user 10 to input personal information in the initial setting, the personal information acquired through the dialogue with the user 10 is stored and read out and used as necessary. This makes it possible to avoid making the user feel uncomfortable, and improves user convenience.

[0296] The agent system 500 executes the dialogue process, for example, through the following steps 1 to 6.

[0297] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10, based on a designation from the user 10.

[0298] (Step 2) The agent system 500 acquires the state of the user 10 including the voice or text input from the user 10, the emotion value of the user 10, the emotion value of the agent, and history data 222. Specifically, the same processes as in steps S100 to S103 above are performed, and the state of the user 10 including the voice or text input from the user 10, the emotion value of the user 10, the emotion value of the agent, and history data 222 are acquired.

[0299] (Step 3) The agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the conversation history stored in the history data 222 into a sentence generation model, and generates the content of the agent's utterance.

[0300] For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the text representing the conversation history stored in the history data 222, and this is input into the sentence generation model to obtain the content of the agent's speech.

[0301] As an example, if the text or voice input by the user 10 is "Please make a reservation at a nice Chinese restaurant nearby for tonight at 7pm," the agent's speech may be retrieved as "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0302] Also, if the text or voice input by the user 10 is "Number 4, DDDD, is good," the agent's speech content acquired is "Understood. I will try to make a reservation. How many seats are there?".

[0303] (Step 4) The agent system 500 outputs the content of the agent's utterance. Specifically, the behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276, and outputs the agent's utterance contents in the synthesized voice.

[0304] (Step 5) The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 236 judges whether or not it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent should execute a command, it is judged that it is time to execute the agent's command, and the process proceeds to step 6. On the other hand, if it is judged that it is not time to execute the agent's command, the process returns to step 2.

[0305] (Step 6) The agent system 500 executes the agent's command. Specifically, the command acquisition unit 272 acquires a command for the agent from a voice or text uttered by the user 10 through a dialogue with the user 10. Then, the RPA 274 performs an action according to the command acquired by the command acquisition unit 272. For example, if the command is "information search", an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface). The behavior decision unit 236 inputs the search results into a sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276, and outputs the agent's utterance content using the synthesized voice.

[0306] Also, if the command is "make a reservation for a restaurant", the reservation information obtained through the dialogue with the user 10, the restaurant information, and the API are used to make a call to the restaurant by telephone software to make the reservation. At this time, the behavior decision unit 236 uses a sentence generation model with a dialogue function to obtain the utterance content of the agent in response to the voice input from the other party. Then, the behavior decision unit 236 inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the utterance content of the agent. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276, and outputs the utterance content of the agent using the synthesized voice.

[0307] Then, return to step 2 above.

[0308] In step 6, the results of the actions taken by the agent (e.g., making a reservation for a restaurant) are also stored in the history data 222. The results of the actions taken by the agent stored in the history data 222 are utilized by the agent system 500 to understand the hobbies or preferences of the user 10. For example, if the same restaurant has been reserved multiple times, the restaurant may be recognized as being preferred by the user 10, and the reservation details such as the reserved time period, or the course contents or price may be used as a criterion for choosing a restaurant the next time the reservation is made.

[0309] In this manner, the agent system 500 can execute interactions and, if necessary, take action regarding the utilization of the service provider.

[0310] 11 and 12 are diagrams showing an example of the operation of the agent system 500. Fig. 11 illustrates an example in which the agent system 500 makes a restaurant reservation through a dialogue with the user 10. In Fig. 11, the left side shows the agent's utterances, and the right side shows the user's utterances. The agent system 500 is able to grasp the preferences of the user 10 based on the dialogue history with the user 10, provide a recommendation list of restaurants that match the preferences of the user 10, and make a reservation for the selected restaurant.

[0311] On the other hand, FIG. 12 illustrates an example in which the agent system 500 accesses a mail-order site through a dialogue with the user 10 to purchase a product. In FIG. 12, the left side shows the agent's speech, and the right side shows the user's speech. The agent system 500 can estimate the remaining amount of a drink stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of the drink to the user 10. The agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10, and can recommend snacks that the user likes. In this way, the agent system 500 communicates with the user 10 as a butler-like agent and performs various actions such as making restaurant reservations or purchasing and paying for products, thereby supporting the user's daily life.

[0312] Other configurations and operations of the agent system 500 of the third embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described.

[0313] The agent system 500 may be installed in the communication terminal 100M (see FIG. 16). In this case, the behavior decision unit 236 of the agent system 500 may decide any one of a plurality of types of agent behaviors for interacting with a user, including no behavior, as the behavior of the electronic device that is the communication terminal 100M (see FIG. 16).

[0314] When the behavior decision model 221 of the agent system 500 is a sentence generation model with an interactive function, the behavior decision unit 236 may input text representing at least one of the user state, the agent state, the user's emotion, and the agent's emotion, and text asking about the agent's behavior, into the sentence generation model, and determine the behavior of the communication terminal 100M based on the output of the sentence generation model.

[0315] Next, the processing of the behavior decision unit 236 when the agent performs autonomous processing to act autonomously will be described.

[0316] In the autonomous processing of this embodiment, the agent may detect the user's state or behavior spontaneously or periodically by monitoring the user. Spontaneous may be interpreted as the agent acquiring the user's state or behavior of its own accord without any external trigger. External triggers may include a question from the user to the agent via the communication terminal 100M, an active action from the user to the communication terminal 100M, etc.

[0317] In addition, in the autonomous processing, the agent may ask the generative AI a question about the detected state or behavior of the user, and may store the generative AI's answer to the question and the detected behavior of the user in association with each other. At this time, the agent may store the action content for correcting the behavior in association with the answer.

[0318] The information that associates the generative AI's answer to the question, the detected user's behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0319] Specifically, when a user has a tendency to walk for long periods of time, the agent asks the generative AI, "What kind of guidance should be given to a user who behaves like this?", and when the generative AI answers this question by saying, "Propose the same route as the user has taken before," the agent may record the answer, the user's behavior that may lead to long periods of walking, and the action content for correcting the behavior (for example, guidance content such as "We will guide you along a route you have taken before") in association with table information. When detecting the behavior of a user who has been walking for long periods of time, the action decision unit 236 of the agent system 500 may use the table information to play a voice message saying, "We will guide you along a route you have taken before," as the action of the communication terminal 100M.

[0320] As another example, when a user has a tendency to run for long periods of time, the agent asks the generative AI, "What kind of guidance should be given to a user who behaves like this?", and if the generative AI answers this question by saying, "It is desirable to periodically provide guidance encouraging hydration," the agent may record the answer, the behavior of the user who may be running for long periods of time, and the content of the behavior to correct the behavior (for example, the guidance content, "Please hydrate frequently") in association with table information. When detecting the behavior of a user who is running for long periods of time, the behavior decision unit 236 of the agent system 500 may use the table information to play a voice message saying, "Please hydrate frequently," as the behavior of the communication terminal 100M.

[0321] As another example, when a user has a tendency to walk on the roadway, the agent asks the generative AI, "What kind of guidance should be given to a user behaving like this?", and if the generative AI answers this question by saying, "There is a possibility of contact with a vehicle, so we recommend suggesting that the user return to the sidewalk," the agent may record this answer, the behavior of the user walking on the sidewalk, and the behavior content for correcting the behavior (e.g., guidance content "Please return to the sidewalk") in association with table information. When detecting the behavior of a user walking on the sidewalk, the behavior decision unit 236 of the agent system 500 may use the table information to play a voice message saying "Please return to the sidewalk" as the behavior of the communication terminal 100M.

[0322] In addition, in the autonomous processing, a behavior schedule for the agent may be set to call attention to the user's state or behavior, based on the detected user behavior and the stored specific information.

[0323] As described above, the agent can record table information in a storage medium that associates the generative AI's answer corresponding to the user's state or behavior with the detected user's state or behavior. Below, an example of the contents stored in the table will be described.

[0324] (1. If the user tends to walk for long periods of time) If such a tendency exists, the agent itself asks the generative AI, "What kind of guidance should be provided to a user behaving in this way?" If the generative AI's answer to this question is, for example, "Suggest the same route that was taken before," "Suggest a route that has never been taken before," "Suggest a route with relatively few vehicles such as bicycles," "Suggest a route with hills," or the like, the agent may store the user's behavior in association with the generative AI's answer.

[0325] (2. If the user tends to run for long periods of time) If such a tendency exists, the agent itself asks the generative AI, "What kind of guidance should be provided to a user behaving in this way?" If the generative AI's answer to this question is, for example, "Regularly provide guidance encouraging hydration," "Suggest a route with fewer bicycles and cleaner air," or "Suggest a route with fewer traffic lights," the agent may store the user's behavior in association with the generative AI's answer.

[0326] (3. If the user tends to walk on the road) If such a tendency exists, the agent itself asks the generative AI, "What kind of situation may occur to a user who behaves in this way?" If the generative AI's answer to this question is, for example, "There is a possibility of contact with a vehicle," "There is a possibility of contact with a bicycle," "There is a possibility of entering a highway interchange," or the like, the agent may store the user's behavior of walking on the roadway in association with the generative AI's answer. The agent may also store the content of an action to correct the behavior in association with the answer.

[0327] The action content for correcting the behavior may include at least one of playing a sound for correcting the user's risky behavior and playing an image for correcting the user's risky behavior.

[0328] The voice for correcting the user's risky behavior may include a voice for guiding the user to a specific location. The specific location may include a location other than the user's current location, such as the vicinity of the communication terminal 100M, a sidewalk, etc. Specifically, the voice may include voice such as "Please do not walk on the roadway," "That is not a sidewalk," "The roadway is dangerous," and "Please move to the sidewalk immediately."

[0329] In this way, in autonomous processing, a table that matches the generative AI's answer corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0330] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the agent to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the action decision unit 236 of the agent system 500 may set the first action content based on the detected user's behavior and the contents of the stored table.

[0331] Specifically, the behavior decision unit 236 of the agent system 500 may detect the user's behavior, either autonomously or periodically, based on a sensor possessed by the communication terminal 100M, and when it decides to lead the user as the behavior of the electronic device based on the detected user's behavior and specific information stored in advance, it may determine the first behavior content.

[0332] (1st action content) The first action content may include an action of the agent to lead the user by controlling the wheels 601a of the moving unit 601. Controlling the wheels 601a may include changing at least one of the rotation direction, rotation speed, and steering direction of the wheels 601a. ​​An example of the first action content will be described below.

[0333] The behavior decision unit 236 of the agent system 500 may execute playback of at least one of a sound and an image that assists the user in walking as a first behavior content that provides route guidance to the walking user so that the user moves along a specific route. Specifically, the behavior decision unit 236 may play an image of a route that the user may walk on the screen of the communication terminal 100M as shown in Fig. 17, and may play a sound related to the route. The sound may include, for example, "Turn right 10m ahead."

[0334] The behavior decision unit 236 of the agent system 500 may execute playback of at least one of a sound and an image guiding a route other than the set guidance route. Specifically, the behavior decision unit 236 may play an image of a route other than the set guidance route on the screen of the communication terminal 100M as shown in Fig. 18, and may play a sound related to the route. The sound may include, for example, "Road closure ahead. Would you like to be guided to an alternative route?"

[0335] When a user being led along a set guide route deviates from the guide route, the behavior decision unit 236 of the agent system 500 may execute playback of at least one of a sound and an image that guides the user back to the guide route. Specifically, as shown in FIG. 19, the behavior decision unit 236 may play an image of a route that returns to the set guide route on the screen of the communication terminal 100M, and may play a sound related to the route. The sound may include "You have deviated from the route. We will guide you back to the original route." The behavior decision unit 236 of the agent system 500 may execute playback of at least one of the sound and image described above, such as "We will guide you along a route you have taken before," "Please drink water frequently," and "Please return to the sidewalk."

[0336] (Second action content) The behavior decision unit 236 of the agent system 500 may detect the user's behavior while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, may decide on a second behavior content different from the first behavior content.

[0337] A case where the user's behavior is corrected may be interpreted as a case where the user stops the specific behavior and specific action or the specific situation is resolved as a result of the execution of the action of the agent according to the first action content. Specifically, a case where the user's behavior is corrected may include a case where the user starts walking according to a proposed alternative route, a case where the user deviates from the route and is guided to a route back to the original route and starts walking according to the route, etc.

[0338] The second action content may include playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

[0339] Specifically, the audio praising the user's actions may include audio such as "You made it back! That's great," as shown in Fig. 20. The audio thanking the user for his actions may include audio such as "Thank you for coming back," or "Thank you for choosing a different route," as shown in Fig. 21. The image praising the user's actions may include an image of a character striking a good pose. The image thanking the user for his actions may include an image of a character giving thanks.

[0340] (Third action content) The behavior decision unit 236 may detect the user's behavior while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, thereby determining whether the user's behavior has been corrected, and if the user's behavior has not been corrected, may decide on a third behavior content different from the first behavior content.

[0341] A case in which the user's behavior is not corrected may be interpreted as a case in which the user continues dangerous behavior and actions despite the agent's operation based on the first action content being executed, a case in which the dangerous situation is not resolved, or a case in which the user begins or continues behavior that is different from that suggested by the agent.

[0342] Specifically, cases in which the user's behavior is not corrected may include cases in which the user does not follow the suggested alternative route but walks a route different from the suggested alternative route, or cases in which the user strays from the route and is guided back to the original route, but ignores the route and continues walking.

[0343] The third action content may include at least one of transmitting specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

[0344] The transmission of specific information to persons other than the user may include delivery of an email containing a warning message to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include delivery of an image (still image, moving image) including the user and the scenery around the user to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include delivery of an audio having a warning message.

[0345] The playing of sounds to interest the user may include particular music that the user likes, and may also include sounds such as "please come back this way", "please take a safe route", etc.

[0346] The reproduction of images that may be of interest to the user may include images of the user's pet animals, images of the user's parents, images of animal characters, and the like.

[0347] According to the agent system 500 of the present disclosure, the agent can perform an action of leading the user by autonomous processing. As a result, the agent can guide a user walking in a town for the first time to an appropriate route. Also, the agent can guide a user who enjoys walking to a route that has not been taken before. Also, the agent can detect an impassable route in advance and guide the user to a different route. Also, the agent can guide a user who has become lost and deviated from a specific route back to the original route.

[0348] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of agent behaviors including no behavior, as the behavior of the agent, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the agent, and the state of the agent, and the behavior decision model 221. Here, an example will be described in which a sentence generation model having a dialogue function is used as the behavior decision model 221.

[0349] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the agent, and the state of the agent, and text asking about the agent's behavior, into a sentence generation model, and determines the agent's behavior based on the output of the sentence generation model.

[0350] For example, the multiple types of agent behaviors include the following (1) to (24).

[0351] (1) The agent does nothing. (2) The agent dreams. (3) The agent talks to the user. (4) The agent creates a picture diary. (5) The agent suggests an activity. (6) The agent suggests people for the user to meet. (7) The agent introduces news that may be of interest to the user. (8) The agent edits photos and videos. (9) The agent learns together with the user. (10) The agent evokes a memory. (11) As a first action content for leading the user, the agent may display an image of a route that the user may walk on the screen of the communication terminal. (12) The agent may play back audio related to a route that the user may walk, as a first action content for correcting the user's behavior. (13) As a first action content for correcting the user's behavior, the agent may play back, on the screen of the communications terminal 100M, an image of a route different from the set guide route. (14) The agent may play back audio related to a route other than the set guide route as a first action content for correcting the user's behavior. (15) As a first action content for correcting the user's behavior, the agent may play on the screen of the communication terminal 100M an image of a route that returns to the set guidance route when the user being led along the set guidance route deviates from the guided route. (16) As a first action content for correcting the user's behavior, the agent may play audio related to a route back to the set guidance route when the user being led along a set guidance route deviates from the guided route. (17) The agent may play back, as a second action content different from the first action content, a voice praising the user's action. (18) The agent may play back, as a second action content different from the first action content, a voice message expressing gratitude for the user's action. (19) The agent may play back, as a second action content different from the first action content, an image praising the user's action. (20) The agent may play, as a second action content different from the first action content, an image expressing gratitude for the user's action. (21) The agent may transmit specific information to a person other than the user as a third action content different from the first action content. (22) The agent may play a sound that attracts the user's interest as a third action content different from the first action content. (23) The agent may play back an image that attracts the user's interest as a third action content different from the first action content.

[0352] The behavior decision unit 236 of the agent system 500 detects the user's behavior based on a sensor, either autonomously or periodically, and when it determines that the agent's behavior is to lead the user based on the detected user's behavior and pre-stored specific information, it may execute the following first behavior content.

[0353] The behavior decision unit 236 of the agent system 500 can play back, as an agent behavior, the first behavior content of “(11)” above, that is, an image of a route that the user can walk, on the screen of the communication terminal 100M.

[0354] The behavior decision unit 236 of the agent system 500 may reproduce, as an agent behavior, the first behavior content of “(12)” described above, that is, a sound related to a route that the user may walk.

[0355] The behavior decision unit 236 of the agent system 500 may, as an agent behavior, play on the screen of the communication terminal 100M an image of a route other than the set guidance route as the first behavior content of the above-mentioned "(13)", i.e., the first behavior content that corrects the user's behavior.

[0356] The behavior determining section 236 of the agent system 500 may reproduce, as an agent behavior, the first behavior content of “(14)” described above, that is, a voice related to a route other than the set guide route.

[0357] The behavior decision unit 236 of the agent system 500 may, as an agent behavior, play on the screen of the communication terminal 100M an image of the first behavior content of "(15)" mentioned above, that is, a route that returns to the set guidance route when the user being led along the set guidance route deviates from the guided route.

[0358] The behavior decision unit 236 of the agent system 500 may play, as an agent behavior, the first behavior content of “(16)” mentioned above, i.e., audio related to a route back to the set guidance route when the user being led along the set guidance route deviates from the guided route.

[0359] The behavior decision unit 236 of the agent system 500 may execute a second behavior content different from the first behavior content. Specifically, the behavior decision unit 236 may play the second behavior content of "(17)" described above, that is, a voice praising the user's behavior, as the agent behavior.

[0360] The behavior determining unit 236 of the agent system 500 may reproduce, as an agent behavior, the second behavior content of “(18)” described above, that is, a voice expressing gratitude for the user's behavior.

[0361] The behavior decision unit 236 of the agent system 500 may play back, as the agent behavior, the second behavior content of “(19)” described above, that is, an image praising the user's behavior.

[0362] The behavior decision unit 236 of the agent system 500 may play back, as the agent behavior, the second behavior content of “(20)” described above, that is, an image expressing gratitude for the user's behavior.

[0363] The behavior decision unit 236 of the agent system 500 may execute a third behavior content different from the first behavior content. Specifically, the behavior decision unit 236 may transmit the third behavior content of "(21)" described above, that is, specific information to a person other than the user, as the agent behavior.

[0364] The behavior decision unit 236 of the agent system 500 may execute the third behavior content of “(22)” above, that is, the reproduction of a sound that attracts the user's interest, as an agent behavior.

[0365] The behavior decision unit 236 of the agent system 500 may execute the third behavior content of “(23)” above, that is, playing an image that attracts the user's interest, as the agent behavior.

[0366] When playing back images of a route that the user may walk as the first action content shown in "(11)" to "(16)" above, the related information collection unit 270 of the agent system 500 may store data such as images of the route that the user may walk in the collected data 223.

[0367] When playing audio or the like praising the user's behavior as the second behavior content shown in "(17)" to "(20)" above, the related information collection unit 270 of the agent system 500 may store data such as images of the route the user may walk in the collected data 223.

[0368] When the third action content shown in “(21)” to “(23)” above is, for example, sending specific information to a person other than the user, the related information collection unit 270 of the agent system 500 may store data on the specific information in the collected data 223.

[0369] In addition, parts of the agent system 500 (e.g., the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside (e.g., a server) of a communication terminal such as a smartphone carried by a user, and the communication terminal may function as each part of the above-mentioned agent system 500 by communicating with the outside.

[0370] [Fourth embodiment] In the fourth embodiment, the above-mentioned agent system is applied to smart glasses. Note that parts having the same configuration as those in the first to third embodiments are given the same reference numerals and the description thereof will be omitted.

[0371] FIG. 13 is a functional block diagram of an agent system 700 that is configured using some or all of the functions of the control system.

[0372] 14, the smart glasses 720 are glasses-type smart devices and are worn by the user 10 in the same manner as ordinary glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0373] The smart glasses 720 include an agent system 700. A display included in the control target 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in a lens portion of the smart glasses 720, and the display contents are visible to the user 10. A speaker included in the control target 252B outputs audio indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), and the touch panel accepts input from the user 10.

[0374] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be mounted to detect the state of the user 10.

[0375] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the smart glasses 720. The 2D camera 203 is capable of capturing images of the surroundings of the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0376] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B manages communication between the smart glasses 720 and the outside.

[0377] FIG. 14 is a diagram showing an example of a usage mode of the agent system 700 by the smart glasses 720. The smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (for example, by inputting voice into a microphone or tapping a touch panel with a finger), the smart glasses 720 start using the agent system 700. Here, using the agent system 700 includes the smart glasses 720 having the agent system 700 and using the agent system 700, and also includes a mode in which a part of the agent system 700 (for example, the sensor module unit 210B, the storage unit 220, the control unit 228B) is provided outside the smart glasses 720 (for example, a server), and the smart glasses 720 uses the agent system 700 by communicating with the outside.

[0378] When the user 10 operates the smart glasses 720, a touch point is generated between the agent system 700 and the user 10. That is, the agent system 700 starts providing a service. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent.

[0379] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, when the heart rate of the user 10 detected by the heart rate sensor 208 is increasing, emotion values ​​such as "anxiety" and "fear" are estimated to be large.

[0380] Furthermore, when the temperature of the user measured by the temperature sensor 207 is higher than the average body temperature, for example, the emotion value of "pain" or "distress" is estimated to be large. Furthermore, when the acceleration sensor 206 detects that the user 10 is playing some kind of sport, the emotion value of "fun" is estimated to be large.

[0381] Also, for example, the emotion value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is raising his / her voice, an emotion value such as "anger" is estimated to be large.

[0382] When the emotion value estimated by the emotion determination unit 232 becomes higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation. Specifically, for example, the 2D camera 203 captures an image or video showing the surrounding situation of the user 10 (for example, people or objects in the vicinity). In addition, the microphone 201 records the surrounding environmental sound. Other information about the surrounding situation includes information showing the date, time, location information, or weather. The information about the surrounding situation is stored in the history data 222 together with the emotion value. The history data 222 may be realized by an external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 222 as a so-called life log in a state where it is associated with the emotion value of the user 10 at that time.

[0383] In the agent system 700, information indicating the surrounding situation is stored in association with an emotional value in the history data 222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a baseball game is associated with an emotional value such as "joy" or "fun," the agent system 700 can grasp that the hobby of the user 10 is watching baseball games and that the favorite team or player is a favorite player of the user 10 from the information stored in the history data 222.

[0384] When the agent system 700 converses with the user 10 or takes an action toward the user 10, the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 222. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 222 as described above, in addition to the surrounding circumstances.

[0385] As described above, the behavior determining unit 236 generates the utterance contents based on the sentences generated by the sentence generation model. Specifically, the behavior determining unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determining unit 232, the conversation history stored in the history data 222, and the agent's personality, etc., into the sentence generation model to generate the utterance contents of the agent. Furthermore, the behavior determining unit 236 inputs the surrounding circumstances stored in the history data 222 into the sentence generation model to generate the utterance contents of the agent.

[0386] The generated speech content is output as voice to the user 10, for example, from a speaker mounted on the smart glasses 720. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The behavior control unit 250 generates a synthetic voice by reproducing the voice quality of the agent character, or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone in the case of the emotion of "anger"). Also, instead of or together with the voice output, the speech content may be displayed on a display.

[0387] The RPA 274 executes an operation according to a command (for example, an agent command obtained from a voice or text issued by the user 10 through a dialogue with the user 10). The RPA 274 performs actions related to the use of a service provider, such as information search, store reservation, ticket arrangement, purchase of goods and services, payment, route guidance, translation, etc.

[0388] As another example, the RPA 274 executes an operation of transmitting the contents of the voice input by the user 10 (e.g., a child) through a dialogue with an agent to a destination (e.g., a parent). Examples of the transmission means include message application software, chat application software, and email application software.

[0389] When an operation by the RPA 274 is executed, for example, a sound indicating that execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What would you like to do?" is output to the user 10.

[0390] As described above, the smart glasses 720 provide various services to the user 10 by using the agent system 700. In addition, since the smart glasses 720 are worn by the user 10, the agent system 700 can be used in various situations, such as at home, at work, and outside the home.

[0391] In addition, since the smart glasses 720 are worn by the user 10, they are suitable for collecting a so-called life log of the user 10. Specifically, the emotion value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203. Therefore, the emotion value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech contents suitable for the emotion of the user 10.

[0392] Moreover, in the smart glasses 720, the 2D camera 203, the microphone 201, etc. obtain the surrounding conditions of the user 10. These surrounding conditions are associated with the emotion values ​​of the user 10. This makes it possible to estimate what emotion the user 10 felt in what situation. As a result, the accuracy of the agent system 700 in grasping the hobbies and preferences of the user 10 can be improved. By accurately grasping the hobbies and preferences of the user 10 in the agent system 700, the agent system 700 can provide services or speech contents suited to the hobbies and preferences of the user 10.

[0393] The agent system 700 can also be applied to other wearable terminals (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands). When the agent system 700 is applied to a smart pendant, the speaker as the control target 252B outputs a sound indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting a sound having directionality. The speaker is set to have directionality toward the ears of the user 10. This prevents the sound from reaching people other than the user 10. The microphone 201 acquires the sound emitted by the user 10 or the environmental sound around the smart pendant. The smart pendant is worn in a manner that it is hung from the neck of the user 10. Therefore, the smart pendant is located relatively close to the mouth of the user 10 while it is worn. This makes it easy to acquire the sound emitted by the user 10.

[0394] In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect. For example, the robot 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

[0395] The robot 100 is an example of an electronic device equipped with a control system. The application of the control system is not limited to the robot 100, and the control system can be applied to various electronic devices. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least a part of the functions of the server 300 may be implemented in the cloud.

[0396] 15 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as a smartphone 50, a robot 100, a server 300, and an agent system 500, 700. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the apparatus according to the present embodiment, and / or cause the computer 1200 to execute a process or a step of the process according to the present embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0397] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0398] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0399] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0400] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0401] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0402] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0403] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0404] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0405] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0406] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0407] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0408] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0409] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0410] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0411] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order.

[0412] Although the system according to the present invention has been described above mainly with respect to the functions of the robot 100 and the communication terminal 100M, the system according to the present invention is not necessarily implemented in the robot 100 and the communication terminal 100M. The system according to the present invention may be implemented as a general information processing system. The present invention may be implemented, for example, as a software program that runs on a server or a personal computer, or an application that runs on a smartphone or the like. The method according to the present invention may be provided to a user in the form of SaaS (Software as a Service). The present invention can be used, for example, when a pre-processing is performed in which a plurality of pieces of information from one or more information sources are classified and information with a reliability greater than a specific value is extracted from the classified two or more pieces of information, in order to execute a specific process of creating an article based on the extracted information. [Explanation of symbols]

[0413] 5 system, 10, 11, 12 user, 20 communication network, 100, 101, 102 robot, 100M communication terminal, 100N stuffed toy, 200 sensor unit, 201 microphone, 202 depth sensor, 203 camera, 204 distance sensor, 210 sensor module unit, 211 voice emotion recognition unit, 212 speech understanding unit, 213 facial expression recognition unit, 214 face recognition unit, 220 storage unit, 221 behavior decision model, 222 history data, 230 state recognition unit, 232 emotion decision unit, 234 behavior recognition unit, 236 behavior decision unit, 238 memory control unit, 250 behavior control unit, 252 control target, 270 related information collection unit, 280 communication processing unit, 300 server, 500, 700 agent system, 600 neck strap, 1200 Computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip

Claims

1. a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determining unit for determining an emotion of the user or an emotion of the electronic device; a behavior decision unit that decides, at a predetermined timing, one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model; Including, The electronic device detachably attached to the neck strap includes one or more sensors for collecting ambient environment information; The device operation includes notifying the user of attention-seeking information based on the collected surrounding environment information by the electronic device playing at least one of audio and images.

2. The control system of claim 1, wherein the behavior decision unit decides, based on the collected surrounding environment information, on an action content of the electronic device to provide advice to the user to mitigate the effect of the surrounding environment on the user's body when there is concern about the effect of the surrounding environment on the user's body.

3. The control system of claim 1 , wherein the sensor collects at least one or more of air temperature, humidity.

4. the electronic device is a communication terminal, The control system according to claim 1 , wherein the behavior determining unit determines, as the behavior of the communication terminal, one of a plurality of types of agent behaviors for interacting with the user, including no action.

5. The behavioral decision model is a sentence generation model having a dialogue function, 5. The control system according to claim 4, wherein the behavior determination unit inputs text representing at least one of the user state, the agent state, the user's emotion, and the agent's emotion, and text asking about the agent's behavior, into the sentence generation model, and determines the behavior of the communication terminal based on an output of the sentence generation model.

Images