Electronic device

Abstract

To enable provision of care according to the state of mind of a user.SOLUTION: An electronic device according to an embodiment comprises a control unit configured to control an avatar so as to recognize the state of mind of a user and determine the emotions of the user, conduct a conversation with the user according to the determined emotions, and provide care according to the recognized state of mind.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The disclosed embodiments relate to electronic devices. [Background technology]

[0002] Conventionally, a technology for determining an appropriate behavior of a robot for a user's state has been disclosed (for example, see Patent Document 1). Patent Document 1 discloses that a robot recognizes a user's reaction when the robot executes a specific action, and if the robot's action for the recognized user's reaction cannot be determined, the robot updates its action by receiving information about an action appropriate for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, it may not be possible to provide care that is appropriate for the user's mental state.

[0005] The present invention has been made in view of the above, and has an object to provide an electronic device that can provide care according to the mental state of a user. [Means for solving the problem]

[0006] An electronic device according to one aspect of an embodiment includes a control unit that recognizes a user's mental state and determines the user's emotions, and based on the determined emotions, engages in a conversation with the user and controls an avatar to provide care according to the recognized mental state. Effect of the Invention

[0007] According to one aspect of the embodiment, care can be provided according to the user's mental state. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a control system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a schematic functional configuration of a control device that controls an avatar. [Diagram 3] FIG. 3 is a diagram illustrating an example of an operational flow relating to an operation of determining an action of an avatar in the control device. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a computer that functions as a control device and a server. [Diagram 5] FIG. 5 is a diagram showing an emotion map onto which a plurality of emotions are mapped. [Figure 6] FIG. 6 is a diagram showing another example of the emotion map. [Figure 7] FIG. 7 is a diagram illustrating an example of an emotion table. [Figure 8] FIG. 8 is a diagram illustrating an example of an emotion table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described below through embodiments, but the following embodiments do not limit the scope of the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0010] FIG. 1 is a diagram illustrating an example of a control system 1 according to the present embodiment. As illustrated in FIG. 1, the control system 1 includes a control device 100 that controls multiple avatars 120, a linked device 400, and a server 300. The multiple control devices 100a to 100d are managed by users. The appearance of the multiple avatars 120 may be a human figure like the avatars 120a to 120c, or a stuffed toy figure like the avatar 120d. In the description of the present embodiment, when the avatars 120a to 120d are not distinguished from each other, they may be simply referred to as avatars 120. The control device 100 that controls the avatar 120 is connected wirelessly or wired to a display device such as a PC monitor or a smartphone display, or to a wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality. The control device 100 is an example of an electronic device.

[0011] As shown in FIG. 1, the avatar 120 controlled by the control device 100 communicates with the user based on various forms. For example, as shown in screens 110a to 110c in FIG. 1, the avatars 120a to 120c are expressed as people or the like displayed on a display device or the like, and communicate with the user via an input / output device provided in the control devices 100a to 100d. Also, as shown in a screen 110d in FIG. 1, the avatar 120d communicates in a virtual space with the users 10a and 10b displayed as people or characters on a display device or the like. Also, as shown in the screen 110d in FIG. 1, the avatar 120d can be displayed not as a person, but as a character such as a stuffed toy, and communicates with the users 10a and 10b as the character. Note that, although the display device or the like described above is described with a display monitor as an example in FIG. 1, the present invention is not limited to this. For example, in this embodiment, the display device may include a monitor, a display unit of a notebook PC or a mobile terminal, a wearable terminal device related to augmented reality, virtual reality, mixed reality, etc., such as VR / AR goggles, VR / AR glasses, and other wearable devices that provide information perceived by the user's five senses.

[0012] The appearance of the avatar may be a human figure like avatars 120a to 120c, or a stuffed animal character like avatar 120d. For example, avatars 120a to 120c are avatars disguised as the children or grandchildren of elderly users, and nod and listen to what the elderly users have to say, providing mental care through conversation, support for daily life, and advice on health management.

[0013] The avatar 120 is controlled by the control device 100, and converses with the user and provides the user with video via an input / output device provided in the control device 100. At this time, the control device 100 cooperates with a server 300 or the like that can communicate with the control device 100 via the communication network 20 to conduct a conversation between the avatar 120 and the user and provide the user with video. For example, the control device 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn so as to be able to have a more appropriate conversation with the user. In addition, the control device 100 records the captured video data of the user in the server 300, and requests the video data from the server 300 as necessary to provide it to the user.

[0014] The control device 100 also stores emotion values ​​that represent the types of emotions of the avatar 120 itself. For example, the control device 100 stores emotion values ​​that represent the strength of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral" as emotion values ​​of the avatar 120. For example, when the user and the avatar 120 converse in a state where the emotion value of excitement is high, the control device 100 controls the avatar 120 so that the avatar 120 speaks at a fast speed. In this way, the avatar 120 can express its own emotions through its actions under the control of the control device 100.

[0015] Furthermore, the control device 100 may be configured to determine the behavior of the avatar 120 corresponding to the user's emotion by matching a sentence generation model (so-called an AI (Artificial Intelligence) chat engine) with an emotion engine. Specifically, the control device 100 may be configured to recognize the user's behavior, determine the user's emotion regarding the user's behavior, and determine the behavior of the avatar 120 corresponding to the determined emotion.

[0016] More specifically, when the control device 100 recognizes a user's action, it automatically generates an action content that the avatar 120 should take in response to the user's action, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is described, for example, in JP2018-081444A and chatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), this is well known, and detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM: Large Language Model). As described above, in this embodiment, by combining a large language model with an emotion engine, it is possible to reflect the emotions of the user and the avatar 120 and various linguistic information in the behavior of the avatar 120. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.

[0017] The control device 100 also has a function of recognizing the user's behavior. The control device 100 recognizes the user's behavior by analyzing the user's face image acquired by the camera function and the user's voice acquired by the microphone function. The control device 100 determines the behavior to be performed by the avatar 120 based on the recognized user's behavior, etc.

[0018] The control device 100 stores rules that define actions to be taken by the avatar 120 based on the user's emotions, the emotions of the avatar 120, and the user's actions, and performs various actions in accordance with the rules.

[0019] Specifically, the control device 100 stores reaction rules for determining the behavior of the avatar 120 based on the user's emotion, the emotion of the avatar 120, and the user's behavior. For example, the reaction rules define the behavior of the avatar 120 as "laughing" when the user's behavior is "laughing". The reaction rules also define the behavior of the avatar 120 as "apologizing" when the user's behavior is "angry". The reaction rules also define the behavior of the avatar 120 as "answering" when the user's behavior is "asking a question". The reaction rules also define the behavior of the avatar 120 as "calling out" when the user's behavior is "sad".

[0020] When the control device 100 recognizes the user's behavior as "angry" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the avatar 120. For example, when the avatar 120 selects the behavior of "apologizing", it performs the motion of "apologizing" and outputs a voice expressing the words "apologize".

[0021] In addition, when the emotion of avatar 120 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the user's state is "alone and looks lonely" is met, it is defined that the emotion of avatar 120 will change to "worried" and the action of "calling out" can be executed.

[0022] When the control device 100 recognizes based on the reaction rules that the current emotion of the avatar 120 is "normal" and that the user is alone and appears to be lonely, the control device 100 increases the emotion value of "sadness" of the avatar 120. Furthermore, the control device 100 selects the action of "calling out" defined in the reaction rules as the action that the avatar 120 will perform toward the user. For example, when the control device 100 selects the action of "calling out", the control device 100 converts the words "What's wrong?", which express concern, into a worried voice of the avatar 120 and outputs it.

[0023] Furthermore, the control device 100 transmits user reaction information indicating that this action has elicited a positive reaction from the user to the server 300. The user reaction information includes, for example, the user action of "getting angry," the avatar 120 action of "apologizing," the fact that the user reaction was positive, and the user attributes.

[0024] The server 300 stores the user reaction information received from each control device 100. Then, the server 300 analyzes the user reaction information from each control device 100 and updates the reaction rules.

[0025] The control device 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The control device 100 incorporates the updated reaction rules into the reaction rules stored in the control device 100. This allows the control device 100 to incorporate reaction rules acquired by other control devices 100 into its own reaction rules. When the reaction rules are updated, they may be automatically transmitted from the server 300 to the control device 100.

[0026] The control device 100 can also execute actions in cooperation with the linked devices 400. The linked devices 400 are, for example, a thermometer, a blood pressure monitor, a smart watch (including one capable of measuring heart rate, etc.), a body composition monitor, a terminal device (such as a personal computer (PC), a smartphone, or a tablet), a television, a display, or training equipment (such as a road runner or a fitness bike). These linked devices 400 are communicatively connected to the control device 100 via the communication network 20, and transmit and receive information to and from the control device 100. With this configuration, the linked devices 400 control themselves, converse with the user, provide mental care for the user through the conversation, support daily life, and give advice on health management, in accordance with instructions from the control device 100.

[0027] In the present disclosure, examples will be described in which various actions are performed on an elderly user, such as mental care through conversation, support for daily life, and health management advice.

[0028] Specifically, the control device 100 is placed in a care facility, a hospital, or the like, and performs actions such as providing care according to the user's mental state (mental condition), supporting daily life, and giving advice on health management through conversation with the user and vital signs, etc., using the avatar 120. The control device 100 may be linked to linked devices 400 such as a thermometer, a blood pressure monitor, a smart watch, a body composition monitor, and training equipment.

[0029] For example, the avatar 120 appears as a person imitating the child or grandchild of the user (elderly person) and executes an action to care for the user's mental state by uttering, for example, "Let's move our bodies" or "Let's watch videos that will relax us" according to the user's mental state. At this time, the control device 100 performs care according to the content of the conversation by the avatar 120 and the emotion (user / self). For example, when the emotion of the user is a positive emotion such as "joy," "pleasure," or "excitement," the control device 100 executes an action to encourage exercise in cooperation with a training device, together with the utterance of "Let's move our bodies" by the avatar 120. On the other hand, when the emotion of the user is a negative emotion such as "sorrow," "discomfort," "anxiety," "sorrow," "worry," or "emptiness," the control device 100 causes, for example, a person imitating the child or grandchild of the user (elderly person) to appear as the avatar 120, and executes an action to encourage the user to watch a relaxation video in cooperation with a terminal device, a television, a display, or the like, together with the utterance of "Let's watch videos that will relax us."

[0030] The control device 100 also executes an action of encouraging the user to measure the user's body temperature, blood pressure, weight, etc. by making an utterance such as "Let's measure your blood pressure this morning" at a predetermined time every morning. The control device 100 also executes an action of encouraging the user to exercise by making an utterance such as "Let's start exercising" or "Let's go for a walk" at a specific time each morning and afternoon. The control device 100 also executes an action of encouraging the user to exercise by making an utterance such as "Let's start exercising" or "Let's go for a walk" at a specific time each morning and afternoon. The control device 100 also executes an action of supporting the user's life rhythm by making an utterance such as "It's morning, let's get up" at a predetermined time each morning. Furthermore, the control device 100 performs an action to support the daily rhythm of the user by making a person, for example, a child or grandchild of the user (elderly person) appear as the avatar 120 and making the person say, for example, "Make sure to go to the toilet before going to sleep" before going to sleep. In other words, the control device 100 performs an action to support the daily life of the user via the avatar 120, etc.

[0031] In addition, the control device 100 performs an action of providing the user with health management advice by having a person modeled, for example, on the user's (elderly person's) child or grandchild appear as the avatar 120 and speaking utterances such as "Increase your exercise" or "Avoid staying up late at night" in accordance with the changes in the user's body temperature, blood pressure, weight, etc. over a specific period such as one week or one month.

[0032] In this manner, in the present disclosure, the control device 100 can perform actions such as providing mental care to the user through conversations with the avatar 120, supporting daily life, and giving health care advice, etc. In other words, the control device 100 according to the present disclosure can provide care according to the user's mental state.

[0033] 2 is a diagram illustrating a schematic functional configuration of the control device 100 that controls the avatar 120. The control device 100 is configured by a control unit having a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0034] The control target 252 includes a display device, a speaker, and the like. The display device displays the avatar 120 itself and images related to the avatar 120. The speaker outputs sounds related to the conversation of the avatar 120 and the actions of the avatar 120. The display device may also display the contents of the conversation with the user as text. The gestures and facial expressions of the avatar 120 are examples of the attitude of the avatar 120.

[0035] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, an acceleration sensor 205, a thermosensor 206, and a touch sensor 207. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on a dummy head, a simulator, or the like, and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image with visible light and generates video information of the visible light. The contour of the object may be detected from the video information generated by the 2D camera 203. The distance sensor 204 detects the distance to the object by irradiating, for example, a laser or ultrasonic waves. The acceleration sensor 205 is, for example, a gyro sensor, and detects the acceleration of the control device 100. The thermosensor 206 detects the temperature around the control device 100. The touch sensor 207 is a sensor that detects a touch operation by a user, and is arranged, for example, on a display device connected to the control device 100. The sensor unit 200 may also include a clock, a sensor for motor feedback, and the like.

[0036] The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user's past emotion values ​​and behavioral history. The emotion values ​​and behavioral history are recorded for each user, for example, by being associated with the user's identification information. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the user's face image, user attribute information, and the like. Note that the functions of the components of the control device 100 shown in FIG. 2, except for the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operation of a CPU by an operating system (OS) and a program operating on the OS.

[0037] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the user state recognition unit 230.

[0038] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the user's voice detected by the microphone 201 to recognize the user's emotion. For example, the voice emotion recognition unit 211 extracts features such as frequency components of the voice and recognizes the user's emotion based on the extracted features. The speech understanding unit 212 analyzes the user's voice detected by the microphone 201 and outputs text information representing the contents of the user's utterance. For example, the speech understanding unit 212 outputs text information representing the contents of a conversation between the avatar 120 and the user.

[0039] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user from the image of the user captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user based on the shapes, positional relationship, etc. of the eyes and mouth. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user during a conversation with the user.

[0040] The face recognition unit 214 recognizes the face of the user. The face recognition unit 214 recognizes the user by matching a face image stored in a person DB (not shown) with a face image of the user captured by the 2D camera 203.

[0041] The user state recognition unit 230 recognizes the state of the user based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, the user state recognition unit 230 generates perception information such as "The user's mental state is 70% likely to be mentally fatigued" or "The user's mental state is 30% likely to be homesick" and performs processing to understand the meaning of the generated perception information. For example, the user state recognition unit 230 generates semantic information such as "The user's mental state is weak, and it seems that mental care that provides a sense of security is required."

[0042] The emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input to a pre-trained neural network to obtain an emotion value indicating the user's emotion.

[0043] Here, the emotion value indicating the user's emotion is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "pleasure", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value indicates 0.

[0044] Furthermore, the emotion determining unit 232 determines an emotion value indicating the emotion of the avatar 120 based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognizing unit 230 .

[0045] The emotion value of avatar 120 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "happiness."

[0046] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the avatar 120 in accordance with a rule for updating the emotion value of the avatar 120, which rule is determined in correspondence with the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230.

[0047] For example, when the user state recognition unit 230 recognizes that the user looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the avatar 120. Also, when the user state recognition unit 230 recognizes that the user is smiling, the emotion determination unit 232 increases the emotion value of "happy" of the avatar 120.

[0048] The emotion determination unit 232 may determine the emotion value indicating the emotion of the avatar 120 by further considering the state of the avatar 120. For example, the emotion determination unit 232 may increase the emotion value of "sadness" of the avatar 120 when the user does not say anything to the avatar 120 or responds in a disorganized manner. Furthermore, the emotion determination unit 232 may increase the emotion value of "anger" when the user continues to respond in a disorganized manner.

[0049] The behavior recognition unit 234 recognizes the user's behavior based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (e.g., "laughing," "getting angry," "asking a question," "getting sad") is obtained, and the behavioral category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 234 recognizes user behavior such as "not participating in group activities," "restless," "looking down," etc. in a nursing home, hospital, etc.

[0050] As described above, in this embodiment, the control device 100 identifies the user and acquires the user's speech content. When acquiring and using the speech content, the control device 100 obtains the necessary consent in accordance with laws and regulations from the user, and the behavior control system for the avatar 120 in this embodiment takes into consideration the protection of the user's personal information and privacy.

[0051] The behavior determining unit 236 determines the behavior of the avatar corresponding to the user's behavior based on the emotion value of the user and the emotion value of the avatar. Specifically, the behavior determining unit 236 determines the behavior corresponding to the user's behavior recognized by the behavior recognizing unit 234 based on the current emotion value of the user determined by the emotion determining unit 232, the history data 222 of past emotion values ​​determined by the emotion determining unit 232 before the current emotion value of the user was determined, and the emotion value of the avatar 120. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the user's past emotion value, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use the most recent emotion values ​​as the user's past emotion value, or may use an emotion value from a unit period ago, such as one day ago. The behavior determining unit 236 may determine the behavior corresponding to the user's behavior by further considering not only the current emotion value of the avatar 120 but also the history of the past emotion values ​​of the avatar 120. The behavior determined by the behavior determining unit 236 includes a gesture performed by the avatar 120 or the contents of the speech of the avatar 120 .

[0052] The behavior determining unit 236 may determine a behavior corresponding to the user's behavior based on the emotion of the avatar 120. For example, when the avatar 120 is verbally abused by the user or when the user is arrogant (i.e., when the user's reaction is poor), when the surrounding noise is loud and the user's voice cannot be detected, and the emotion value of "anger" or "sorrow" of the avatar 120 increases, the behavior determining unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotion value of "anger" or "sorrow". Also, when the user's reaction is good, and the emotion value of "joy" or "pleasure" of the avatar 120 increases, the behavior determining unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotion value of "joy" or "pleasure". Furthermore, the behavior determining unit 236 may determine a behavior for a user who has increased the emotion values ​​of "joy" or "pleasure" of the avatar 120 to be different from the behavior for a user who has increased the emotion values ​​of "anger" or "sorrow" of the avatar 120. In this way, the behavior determining unit 236 may determine a different behavior depending on the emotion of the avatar 120 itself or how the user has changed the emotion of the avatar 120 through the user's behavior.

[0053] The behavior determining unit 236 according to this embodiment determines, as a behavior corresponding to a user's behavior, a behavior of the avatar 120 corresponding to the reaction rule 221 when a combination including at least one of the user's past emotion value and current emotion value, the emotion value of the avatar 120, and the user's behavior satisfy the conditions of a preset reaction rule. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, the behavior determining unit 236 determines, as a behavior corresponding to the user's behavior, a behavior for changing the user's emotion value to positive.

[0054] Reaction rules 221 prescribe a combination of the user's past emotion value and current emotion value, the emotion value of avatar 120, and the behavior of avatar 120 according to the user's behavior. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, and the user's behavior is sad, a combination of gestures and speech content when asking a question to encourage the user with gestures is predefined as the behavior of avatar 120.

[0055] For example, the reaction rule 221 defines the behavior of the avatar 120 for all combinations of patterns of the emotion values ​​of the avatar 120 (1296 patterns, which are the fourth power of six values ​​of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); combination patterns of the user's past emotion values ​​and current emotion values; and behavior patterns of the user. That is, for each pattern of the emotion values ​​of the avatar 120, behavior of the avatar 120 according to the behavior pattern of the user is defined for each of a plurality of combinations of the user's past emotion values ​​and current emotion values, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that the behavior determining unit 236 may determine the behavior of the avatar 120 using the history data 222 when the user makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0056] Note that reaction rules 221 may define at least one of a gesture and a statement as the behavior of avatar 120 for each of the emotion value patterns (1296 patterns) of avatar 120. Alternatively, reaction rules 221 may define at least one of a gesture and a statement as the behavior of avatar 120 for each group of emotion value patterns of avatar 120.

[0057] The strength of each gesture included in the behavior of avatar 120 defined in reaction rule 221 is determined in advance. The strength of each speech included in the behavior of avatar 120 defined in reaction rule 221 is determined in advance.

[0058] For example, reaction rules 221 define behaviors for avatar 120 corresponding to behavioral patterns such as not participating in group activities, being restless, and looking down.

[0059] The memory control unit 238 determines whether or not to store data including the user's behavior in the history data 222, based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the avatar 120 determined by the emotion determination unit 232.

[0060] Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the avatar 120, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is decided to store data including the user's behavior in the history data 222.

[0061] When the memory control unit 238 decides to store data including the user's behavior in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the user's state recognized by the user state recognition unit 230 (e.g., the user's facial expression, emotions, etc.).

[0062] The behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 236. Specifically, the behavior control unit 250 controls the movement of the avatar 120 and the movement of the control target related to the movement of the avatar 120 based on the behavior of the avatar 120 corresponding to the reaction rule 221 determined by the behavior determination unit 236. For example, in a case where the avatar 120 and the user are having a conversation, when the behavior determination unit 236 determines an action including speaking, the behavior control unit 250 controls the avatar 120 to take a predetermined movement and causes the speaker included in the control target 252 to output the voice of the avatar 120. At this time, the behavior control unit 250 may determine the speech speed of the voice based on the emotion value of the avatar 120. For example, the behavior control unit 250 determines a faster speech speed as the emotion value of the avatar 120 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232. For example, when the user is not participating in a group activity, the behavior control unit 250 controls the avatar 120 to utter such things as "Do you want to join in some recreation?" or "Are you tired? Would you like to watch some relaxing videos over there?", and guides the user to take care of the user's mental state. In addition, when the user is restless, the behavior control unit 250 controls the avatar 120 to utter such things as "Is there something bothering you?", etc. In addition, when the user is looking down, the behavior control unit 250 controls the avatar 120 to utter such things as "Are you OK?", etc., and reports any abnormality in the vital signs to a care manager, nurse, etc. In addition, if the user's emotion is a negative emotion such as "sorrow," "discomfort," "anxiety," "grief," "worry," or "emptiness," the avatar 120 is controlled to make speech that eases the user's negative emotion, such as "The nurse will be here shortly, so don't worry."

[0063] The behavior control unit 250 may recognize a change in the user's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the user's voice or facial expression. Alternatively, the change in the user's emotion may be recognized based on the detection of an impact by the touch sensor included in the sensor unit 200. When an impact is detected by the touch sensor included in the sensor unit 200, the user's emotion may be recognized as having become worse, and when the detection result of the touch sensor included in the sensor unit 200 indicates that the user's reaction is laughing, happy, or the like, the user's emotion may be recognized as having become better. Information indicating the user's reaction is output to the communication processing unit 280.

[0064] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the avatar 120, the emotion determination unit 232 further changes the emotion value of the avatar 120 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the avatar 120 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Furthermore, the emotion determination unit 232 increases the emotion value of "sadness" of the avatar 120 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.

[0065] Furthermore, the behavior control unit 250 expresses the emotion of the avatar 120 based on the determined emotion value of the avatar 120. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the avatar 120, the behavior control unit 250 controls the movement of the avatar 120 to make the avatar 120 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the avatar 120, the behavior control unit 250 controls the movement of the avatar 120 to make the avatar 120 assume a drooping posture.

[0066] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221. The communication processing unit 280 can transmit and receive information to and from the linked device 400.

[0067] The server 300 communicates between each control device 100 and the server 300, receives user reaction information transmitted from the control device 100 controlling the avatar 120, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0068] Fig. 3 is a diagram showing an example of an operation flow relating to an operation of determining the behavior of the avatar 120 in the control device 100. The operation flow shown in Fig. 3 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates a step to be executed.

[0069] First, in step S101, the user state recognition unit 230 recognizes the state of the user based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as "The user's mental state is 70% likely to be mentally fatigued" or "The user's mental state is 30% likely to be homesick" and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as "The user's mental state is weak, and it appears that mental care that provides a sense of security is required."

[0070] In step S102, the emotion determining unit 232 determines an emotion value indicating the emotion of the user based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognizing unit 230.

[0071] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the avatar 120, based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user to the history data 222.

[0072] In step S104, the behavior recognition unit 234 recognizes a behavior classification of the user based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230. For example, the behavior recognition unit 234 recognizes user behavior such as "not participating in group activities," "restless," "looking down," etc. in a nursing home, hospital, etc.

[0073] In step S105, the behavior determination unit 236 determines the behavior of the avatar 120 corresponding to the reaction rule 221 when a combination including at least one of the user's current emotion value determined in step S102 and the past emotion values ​​included in the history data 222, the emotion value of the avatar 120, and the user's behavior recognized by the behavior recognition unit 234 satisfy the conditions of a preset reaction rule.

[0074] In step S106, the behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 236. For example, the behavior control unit 250 controls the avatar 120 to perform a behavior related to guidance when the user is not participating in a group activity, or a behavior of support including mental care for the user when the user is restless or has his / her head down.

[0075] In step S107, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determining unit 236 and the emotion value of the avatar 120 determined by the emotion determining unit 232.

[0076] In step S108, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the user's behavior in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S109.

[0077] In step S109, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the user's state recognized by the user state recognition unit 230 are stored in the history data 222.

[0078] As described above, the control device 100 includes a control unit that recognizes the user's mental state, determines the user's emotion, and controls the avatar 120 to converse with the user based on the determined emotion and provide care according to the recognized mental state. This allows the control device 100 to use the avatar 120 to provide care according to the user's mental state.

[0079] Moreover, the control device 100 further controls the avatar 120 to support the user's daily life. In this way, the control device 100 can use the avatar 120 to prepare an environment in which the user can live with peace of mind.

[0080] Moreover, the control device 100 further controls the avatar 120 to provide the user with health management advice. In this way, the control device 100 can use the avatar 120 to create an environment in which the user can live a healthy life.

[0081] In addition, the user is an elderly person. In this way, the control device 100 can use the avatar 120 to provide care to the elderly person according to their mental state, and can also provide an environment in which the elderly person can live a safe and healthy life.

[0082] In the above embodiment, the control device 100 recognizes a user by using a face image of the user, but the disclosed technology is not limited to this aspect. For example, the control device 100 may recognize a user by using a voice uttered by the user, a user's email address, a user's SNS ID, or an ID card with a built-in wireless IC tag that the user possesses.

[0083] The control device 100 is an example of a control unit that controls the avatar 120, and is an example of an electronic device. The behavior control by the control device 100 is not limited to the avatar 120, and can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.

[0084] 4 is a diagram illustrating an example of a hardware configuration of a computer 1200 functioning as the control device 100 and the server 300. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0085] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphic controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0086] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 retrieves image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0087] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0088] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0089] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0090] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0091] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0092] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0093] The above-mentioned programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0094] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0095] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0096] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or either source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0097] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0098] (Other embodiments) The above-described control device 100 may be mounted on a stuffed toy or a robot, or may be applied to a control device connected wirelessly or via a wire to a control target device (speaker or camera) mounted on a stuffed toy or a robot.

[0099] The emotion determining unit 232 may determine the emotion of the user according to a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user according to an emotion map (see FIG. 5), which is a specific mapping.

[0100] FIG. 5 is a diagram showing an emotion map 700 on which multiple emotions are mapped. In the emotion map 700, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 700, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0101] (1) For example, if the emotion engine, which is the emotion determination unit 232 that controls the emotion of the avatar 120, detects emotions in about 100 msec, the frequency of determining the reaction action of the avatar 120 (for example, a backchannel) may be set at least at the same timing as the emotion engine's detection frequency (100 msec), or may be set at a faster timing than this. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0102] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. The avatar 120 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 700. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

[0103] (2) The directionality of emotions and the strength of their intensity may be preset in reference to emotion map 700, and the movement of the interjections and the strength of the interjections may be set. For example, if avatar 120 feels a sense of stability or security, avatar 120 may nod and continue listening. If avatar 120 feels anxious, confused, or suspicious, avatar 120 may tilt its head or stop shaking its head.

[0104] These emotions are distributed in the three o'clock direction of emotion map 700 and usually fluctuate between relief and anxiety. In the right half of emotion map 700, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0105] (3) If the avatar 120 feels good after being praised, the filler "ah" may precede the line, and if the avatar 120 feels hurt after receiving harsh words, the filler "ugh!" may precede the line. Also, a physical reaction such as the avatar 120 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 700.

[0106] (4) In the left half of the emotion map 700, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

[0107] When avatar 120 feels a favorable feeling in its situational awareness while having an internal feeling (reaction) of a sense of satisfaction, avatar 120 may nod deeply while looking at the other person, or may say "uh-huh." In this way, avatar 120 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on emotion map 700.

[0108] Conversely, if avatar 120 is aware of an internal sensation (reaction) of discomfort, avatar 120 may shake its head when it feels disgust, or glare at the other person when it feels hatred. These types of emotions are distributed around 6 o'clock on emotion map 700.

[0109] (5) The inside of emotion map 700 represents what is going on in one's mind, while the outside of emotion map 700 represents behavior. Therefore, the further out on the outside of emotion map 700 you go, the more visible the emotions become (the more they are expressed in behavior).

[0110] (6) When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock area of ​​emotion map 700, avatar 120 may lightly nod his head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, he may nod vigorously, nodding his head deeply.

[0111] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the user. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user, and emotion values ​​indicating each emotion shown in the emotion map 700. In addition, this neural network is trained so that emotions that are arranged close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 is a diagram showing another example of an emotion map. FIG. 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "encouraging," have similar emotion values.

[0112] Furthermore, the emotion determination unit 232 may determine the emotion of the avatar 120 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user recognized by the user state recognition unit 230, and the state of the avatar 120 to a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the avatar 120. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user, and the state of the avatar 120, and emotion values ​​indicating each emotion shown in the emotion map 700. For example, the neural network is trained based on learning data that indicates that when it is recognized from the output of the touch sensor 207 that the avatar 120 is being stroked by the user, the emotion value becomes "3" for "happiness," and learning data that indicates that when it is recognized from the output of the acceleration sensor 205 that the avatar 120 is being hit by the user, the emotion value becomes "3" for "anger." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0113] Furthermore, the emotion determination unit 232 may determine the emotion of the avatar 120 based on the action content of the avatar 120 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the action content of the avatar 120 generated by the sentence generation model into a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 700, and integrates the acquired emotion values ​​indicating each emotion with the emotion values ​​indicating each emotion of the current avatar 120 to update the emotion of the avatar 120. For example, the emotion values ​​indicating each acquired emotion and the emotion values ​​indicating each emotion of the current avatar 120 are averaged and integrated. This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the action content of the avatar 120 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 700. For example, if the speech content of avatar 120, "That's great. You're lucky," is obtained as the behavior content of avatar 120 generated by the sentence generation model, then when the text representing this speech content is input into the neural network, a high emotion value for the emotion "happy" is obtained, and the emotion of avatar 120 is updated so that the emotion value of the emotion "happy" becomes higher.

[0114] The behavior determination unit 236 generates the behavior content of the avatar by adding a fixed sentence for asking the avatar's behavior content corresponding to the user's behavior to the text representing the user's behavior, the user's emotion, and the avatar's emotion, and inputting the result into the dialogue function.

[0115] For example, the behavior determining unit 236 obtains text representing the state of the avatar 120 from the emotion of the avatar 120 determined by the emotion determining unit 232, using an emotion table as shown in Fig. 7. Fig. 7 is a diagram showing an example of an emotion table. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the avatar 120 is stored for each index number.

[0116] When the emotion of avatar 120 determined by emotion determination unit 232 corresponds to index number "2", the text "very happy state" is obtained. When the emotions of avatar 120 correspond to multiple index numbers, multiple texts representing the state of avatar 120 are obtained.

[0117] Also, for the user's emotions, an emotion table as shown in FIG. 8 is prepared. FIG. 8 is a diagram showing an example of an emotion table. Here, if the user's behavior is speaking "This is AAA," the emotion of the avatar 120 is index number "2," and the user's emotion is index number "3," then "The avatar is in a very happy state. The user is in a normal happy state. The user spoke to him / her, "This is AAA. How would you respond as the avatar?" is input to the sentence generation model, and the content of the avatar's behavior is obtained. The behavior decision unit 236 decides the avatar's behavior from this content of the behavior.

[0118] In this way, avatar 120 can change its behavior according to the index number corresponding to the avatar's emotion, so that the user gets the impression that avatar 120 has a heart, and is encouraged to take actions such as talking to the avatar.

[0119] Furthermore, the behavior determining unit 236 may generate the behavioral content of the avatar by adding not only texts representing the user's behavior, the user's emotions, and the avatar's emotions, but also texts representing the contents of the history data 222, adding a fixed sentence for asking about the avatar's behavioral content corresponding to the user's behavior, and inputting the result into the dialogue function. This allows the avatar 120 to change the behavior of the avatar according to the history data representing the user's emotions and behavior, so that the user has the impression that the avatar has individuality, and is encouraged to take actions such as talking to the avatar. Furthermore, the history data may further include the emotions and behavior of the avatar.

[0120] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0121] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0122] 1. Control System 20 Communication Network 100 Control device 120 Avatar 200 Sensor section 201 Mike 202 3D Depth Sensor 203 2D Camera 204 Distance Sensor 210 Sensor module section 211 Voice Emotion Recognition Unit 212 Speech Understanding Department 213 Facial expression recognition unit 214 Face Recognition Unit 220 Storage Unit 221 Reaction Rules 222 Historical Data 230 User state recognition unit 232 Emotion Determination Department 234 Behavior Recognition Department 236 Action Decision-Making Department 238 Memory control unit 250 Behavior Control Unit 252 Control Object 280 Communication Processing Unit 300 Servers 400 Related Devices

Claims

1. a control unit that recognizes a mental state of a user, determines an emotion of the user, and controls an avatar to converse with the user based on the determined emotion and to provide care according to the recognized mental state; An electronic device comprising:

2. The control unit further controls the avatar to support the user's daily life.

2. The electronic device according to claim 1.

3. The control unit further controls the avatar to provide health management advice to the user.

2. The electronic device according to claim 1.

4. The user is an elderly person.

4. The electronic device according to claim 1.

5. The electronic device includes: The device is connected wirelessly or wired to a wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality.

4. The electronic device according to claim 1.

6. The electronic device includes: It is mounted on the robot, or connected wirelessly or by wire to the controlled device mounted on the robot.

4. The electronic device according to claim 1.

Images