Control device

Abstract

To support a user.SOLUTION: A control device for controlling an avatar includes an estimation part, an acquisition part, and an avatar control part. The estimation part estimates the emotion of a user who performs communication with the avatar. The acquisition part acquires information regarding health and medication of at least one of the user and the user's family. The avatar control part controls the avatar so as to perform support according to the emotion of the user estimated by the estimation part based on the information regarding health and medication of at least one of the user and the user's family acquired by the acquisition part.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a control device. [Background technology]

[0002] Patent Document 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot executes a specific action, and if the robot is unable to determine an action for the recognized user's reaction, it updates the robot's behavior by receiving information about an action suitable for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the prior art leaves room for improvement in terms of supporting users. [Means for solving the problem]

[0005] According to a first aspect of the present invention, there is provided a control device for controlling an avatar. The control device includes an estimation unit, an acquisition unit, and an avatar control unit. The estimation unit estimates the emotion of a user who communicates with the avatar. The acquisition unit acquires health and medical information of at least one of the user and the user's family. The avatar control unit controls the avatar to provide support according to the emotion of the user estimated by the estimation unit based on the health and medical information of at least one of the user and the user's family acquired by the acquisition unit. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system 5 according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a schematic functional configuration of the control device 100 that controls the avatar 30. As shown in FIG. [Diagram 3] FIG. 3 is a diagram illustrating an example of an operational flow for determining the behavior of the avatar 30. As shown in FIG. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of the computer 1200. As shown in FIG. [Diagram 5] FIG. 5 is a diagram showing an emotion map onto which a plurality of emotions are mapped. [Figure 6] FIG. 6 is a diagram showing another example of the emotion map. [Figure 7] FIG. 7 is a diagram illustrating an example of an emotion table. [Figure 8] FIG. 8 is a diagram illustrating an example of an emotion table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0008] 1 is a schematic diagram showing an example of a system 5 according to the present embodiment. The system 5 includes control devices 100-102 that control avatars such as avatar 30, avatar 31, and avatar 32, and a server 300. The control devices 100-102 are connected wirelessly or by wire to a display device such as a PC monitor or a smartphone display, or a predetermined wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality. The control devices 100-102 include various input / output devices such as a speaker that outputs sound such as voice, a microphone that detects sound such as voice, and a camera.

[0009] User 10a, user 10b, user 10c, and user 10d are users who communicate with avatar 30. User 11a, user 11b, and user 11c are users who communicate with avatar 31. User 12a and user 12b are users who communicate with avatar 32. In the description of this embodiment, user 10a, user 10b, user 10c, and user 10d may be collectively referred to as user 10. Furthermore, user 11a, user 11b, and user 11c may be collectively referred to as user 11. Furthermore, user 12a and user 12b may be collectively referred to as user 12.

[0010] As shown in FIG. 1, avatars 30-32 controlled by the control devices 100-102 communicate with users 10-12 based on various forms. For example, as shown in FIG. 1(1), the avatar 30 is expressed as a person or the like displayed on a display device or the like, and communicates with the user 10 by interacting with the user 10 via an input / output device provided in the control device 100. Also, as shown in FIG. 1(2), the avatar 31 communicates with the user 11 displayed as a person or character on the display device or the like in a virtual space. Also, as shown in FIG. 1(3), the avatar 32 can be displayed as a character rather than a person, and communicates with the user 12 as the character. Note that, as for the above-mentioned display device or the like, a display monitor has been described in FIG. 1 as an example, but is not limited thereto. For example, in this embodiment, the display device may include a monitor, a display unit of a notebook PC or a mobile terminal, and a wearable terminal device related to augmented reality, virtual reality, mixed reality, etc., such as VR / AR goggles, VR / AR glasses, and other wearable devices that provide information perceived by the five senses of the user.

[0011] The appearance of the avatar may be a human figure like the avatar 30 and the avatar 31, or a stuffed toy like the avatar 32. The avatar 32 is considered to be particularly familiar to children because of its stuffed toy appearance. For example, the avatars 30 and 31 are avatars dressed as counselors or doctors, and can provide counseling by nodding and listening to the user's story like a real counselor or doctor, making it easier for the user to consult the avatar about their worries. Also, for example, the avatar 32 is considered to be particularly familiar to children because of its character appearance.

[0012] The avatar 31 and the control device 101, and the avatar 32 and the control device 102 have substantially the same functions as the avatar 30 and the control device 100. Therefore, in the following, the system 5 will be described mainly with reference to the functions of the avatar 30 and the control device 100.

[0013] The avatar 30 is controlled by the control device 100, and converses with the user 10 and provides images to the user 10 via an input / output device provided in the control device 100. At this time, the control device 100 cooperates with a server 300 or the like that can communicate with the control device 100 via the communication network 20 to conduct a conversation between the avatar 30 and the user 10 and provide images or the like to the user 10. For example, the control device 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn so as to be able to have a more appropriate conversation with the user 10. In addition, the control device 100 records the captured image data of the user 10 in the server 300, and requests the image data or the like from the server 300 as necessary to provide it to the user 10.

[0014] The control device 100 also stores emotion values ​​that represent the types of emotions of the avatar 30 itself. For example, the control device 100 stores emotion values ​​that represent the strength of each of the emotions of "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "concern," "relief," "fulfillment," "emptiness," and "neutral" as emotion values ​​of the avatar 30. For example, when the user 10 and the avatar 30 converse in a state where the emotion value of excitement is high, the control device 100 controls the avatar 30 to emit voices at a fast speed. In this way, the avatar 30 can express its own emotions through its actions under the control of the control device 100.

[0015] Furthermore, the control device 100 may be configured to match a sentence generation model (so-called an AI (Artificial Intelligence) chat engine) with an emotion engine, thereby determining the behavior of the avatar 30 corresponding to the emotion of the user 10. Specifically, the control device 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 with respect to the behavior of the user 10, and determine the behavior of the avatar 30 corresponding to the determined emotion.

[0016] More specifically, when the control device 100 recognizes the action of the user 10, the control device 100 automatically generates an action content that the avatar 30 should take in response to the action of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is described, for example, in JP 2018-081444 A and chatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), this is well known, and detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM: Large Language Model). As described above, in this embodiment, by combining a large language model with an emotion engine, it is possible to reflect the emotions of the user 10 and the avatar 30, and various linguistic information, in the behavior of the avatar 30. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model with an emotion engine.

[0017] The control device 100 also has a function of recognizing the behavior of the user 10. The control device 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The control device 100 determines the behavior to be performed by the avatar 30 based on the recognized behavior of the user 10, etc.

[0018] The control device 100 stores rules that define actions to be taken by the avatar 30 based on the emotions of the user 10, the emotions of the avatar 30, and the actions of the user 10, and determines the actions to be taken by the avatar 30 in accordance with the rules.

[0019] Specifically, the control device 100 stores reaction rules for determining the behavior of the avatar 30 based on the emotion of the user 10, the emotion of the avatar 30, and the behavior of the user 10. For example, the reaction rules define the behavior of the avatar 30 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the avatar 30 as "apologizing" when the behavior of the user 10 is "angry". The reaction rules also define the behavior of the avatar 30 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the avatar 30 as "calling out" when the behavior of the user 10 is "sad".

[0020] When the control device 100 recognizes the behavior of the user 10 as "angry" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the avatar 30. For example, when the avatar 30 selects the behavior of "apologizing", it performs the motion of "apologizing" and outputs a voice expressing the words "apologize".

[0021] In addition, when the emotion of the avatar 30 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is met, it is determined that the emotion of the avatar 30 will change to "worried" and the action of "calling out" can be executed.

[0022] When the control device 100 recognizes based on the reaction rules that the current emotion of the avatar 30 is "normal" and that the user 10 is alone and appears lonely, the control device 100 increases the emotion value of "sadness" of the avatar 30. Furthermore, the control device 100 selects the action of "calling out" defined in the reaction rules as the action to be performed on the user 10. For example, when the avatar 30 selects the action of "calling out", it converts the words "What's wrong?", which express concern, into a worried voice and outputs it.

[0023] Furthermore, the control device 100 transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the avatar 30 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.

[0024] The control device 100 stores user reaction information regarding the user's reaction to the avatar 30. The server 300 may store the user reaction information received from the avatar 30. The server 300 receives and stores user reaction information not only from the control device 100 that controls the avatar 30, but also from the control devices 101 and 102 that control the avatar 31 and 32. The server 300 then analyzes the user reaction information from the control devices 100, 101, and 102 to update the reaction rules.

[0025] The control device 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The control device 100 incorporates the updated reaction rules into the reaction rules stored in the control device 100. This allows the control device 100 to incorporate the reaction rules acquired by the control devices 101 and 102 of the avatars 31 and 32, etc., into its own reaction rules. When the reaction rules are updated, the updated reaction rules may be automatically transmitted from the server 300 to the control device 100.

[0026] Furthermore, the control device 100 executes support according to the health condition of the user 10 or the family of the user 10 through the avatar 30. The control device 100 acquires health and medical information of the user 10 or the family of the user 10. The health and medical information of the user 10 or the family of the user 10 may be acquired by detection by the control device 100. Furthermore, the health and medical information of the user 10 or the family of the user 10 may be acquired by detection by another device. For example, the health and medical information of the user 10 or the family of the user 10 may be acquired from an IoT (Internet of Things) device owned by the user 10 or the family of the user 10 or a system of a medical institution. For example, the server 300 acquires the health and medical information of the user 10 or the family of the user 10 from an IoT device owned by the user 10 or the family of the user 10 or a system of a medical institution. The server 300 stores the acquired health and medical information in association with the identification information of each user 10. For example, the server 300 stores, as health and medical information, whether the user 10 or a family member of the user 10 is pregnant or when the user 10 gave birth, if the user 10 or a family member of the user 10 is ill, the name of the illness and the condition of the illness, and if the user 10 or a family member of the user 10 is hospitalized, the planned length of hospitalization. Also, for example, the server 300 stores, as health and medical information, if the user 10 or a family member of the user 10 has given birth, that the user 10 or a family member of the user 10 is raising an infant for a predetermined period of time (e.g., one year) from the date of birth. The control device 100 queries the server 300 for health and medical information regarding the user 10 or a family member of the user 10, thereby receiving (obtaining) the health and medical information regarding the user 10 or a family member of the user 10 from the server 300.

[0027] 2 shows a schematic functional configuration of the control device 100 that controls the avatar 30. The control device 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, an acquisition unit 240, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0028] The control target 252 includes a display device 2521 and a speaker 2522. The display device 2521 displays the avatar 30 itself and images related to the avatar 30. The speaker 2522 outputs sounds related to the conversation of the avatar 30, actions of the avatar 30, and the like.

[0029] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on a dummy head, a simulator, or the like, and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include an acceleration sensor, a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.

[0030] The storage unit 220 includes a reaction rule 221, history data 222, and assistance rule 223. The history data 222 includes the history of the user 10's past emotional state and behavior. For example, the history data 222 stores data on the user 10's past emotional value and behavior history. The emotional value and behavior history are recorded for each user 10 by, for example, being associated with the user 10's identification information. The assistance rule 223 is a rule that defines the behavior that the avatar 30 will assist in depending on the health condition and medical situation of the user 10 or the user 10's family. The assistance rule 223 stores information on the content of the response and assistance that the avatar 30 will provide to the user 10's behavior for each health condition and medical situation of the user 10 and the user 10's family. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. The storage unit 220 may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. 2, the functions of the components other than the control target 252, the sensor unit 200, and the storage unit 220 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operation of a CPU by an operating system (OS) and a program that operates on the OS.

[0031] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the user state recognition unit 230.

[0032] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0033] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0034] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0035] The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0036] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0037] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user 10. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the emotion value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user 10 is an emotion that makes the user 10 feel bad, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the emotion value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user 10 is not any of the above ("neutral"), the value of 0 is indicated.

[0038] Furthermore, the emotion determination unit 232 determines an emotion value indicating the emotion of the avatar 30 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .

[0039] The emotion value of the avatar 30 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness", "anger", "sorrow", and "happiness".

[0040] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the avatar 30 in accordance with a rule for updating the emotion value of the avatar 30, which rule is defined in correspondence with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0041] For example, when the user state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the avatar 30. Also, when the user state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the avatar 30.

[0042] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing", "getting angry", "asking a question", "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0043] As described above, in this embodiment, the control device 100 identifies the user 10 and then acquires the content of the user 10's speech. When acquiring and using the content of the speech, the control device 100 not only obtains the necessary consent in accordance with laws and regulations from the user 10, but also takes into consideration the protection of the personal information and privacy of the user 10.

[0044] The acquisition unit 240 acquires health and medical information of at least one of the user 10 and the user 10's family. For example, the acquisition unit 240 acquires health and medical information of the user 10 and the user 10's family from the server 300. For example, the acquisition unit 240 queries the server 300 for identification information of the user 10 and the user 10's family. When the server 300 stores health and medical information corresponding to the identification information, it transmits the health and medical information corresponding to the identification information. The acquisition unit 240 acquires the health and medical information of the user 10 and the user 10's family by receiving it from the server 300. For example, the acquisition unit 240 acquires from the server 300 as information on the family's health and medical information whether the user 10 or the user 10's family is pregnant, when the user 10 or the user 10's family gave birth, if the user 10 or the user 10's family is sick, the name of the illness and the condition of the illness, and if the user 10 or the user 10's family is hospitalized, the planned length of hospitalization.

[0045] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 234 based on the current emotion value of the user 10 determined by the emotion determining unit 232, the history data 222 of past emotion values ​​determined by the emotion determining unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the avatar 30. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the avatar 30 but also the history of the past emotion values ​​of the avatar 30. The behavior determined by the behavior determining unit 236 includes a gesture performed by the avatar 30 or the content of the speech of the avatar 30 .

[0046] The behavior determining section 236 according to this embodiment determines the behavior of the avatar 30 as a behavior corresponding to the behavior of the user 10, based on a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the avatar 30, the behavior of the user 10, and the reaction rules 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior determining section 236 determines the behavior corresponding to the behavior of the user 10 as a behavior for changing the emotion value of the user 10 to positive.

[0047] The reaction rules 221 prescribe a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the avatar 30, and the behavior of the avatar 30 according to the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of a gesture and a speech content when asking a question to encourage the user 10 with a gesture is predefined as the behavior of the avatar 30.

[0048] For example, the reaction rule 221 defines behaviors of the avatar 30 for all combinations of emotion value patterns of the avatar 30 (1296 patterns, which are the fourth power of six values, "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); combination patterns of past emotion values ​​and current emotion values ​​of the user 10; and behavior patterns of the user 10. That is, for each emotion value pattern of the avatar 30, behaviors of the avatar 30 are defined according to the behavior patterns of the user 10 for each of a plurality of combinations of past emotion values ​​and current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that the behavior determination unit 236 may transition to an operation mode in which the behavior of the avatar 30 is determined using the history data 222, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0049] Note that the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the avatar 30 for each of the emotion value patterns (1296 patterns) of the avatar 30. Alternatively, the reaction rules 221 may prescribe at least one of a gesture and a statement as the behavior of the avatar 30 for each group of emotion value patterns of the avatar 30.

[0050] The strength of each gesture included in the behavior of the avatar 30 defined in the reaction rules 221 is determined in advance. The strength of each utterance included in the behavior of the avatar 30 defined in the reaction rules 221 is determined in advance.

[0051] Furthermore, when the acquisition unit 240 acquires information regarding the health and medical care of at least one of the user 10 and the user's 10 family, the behavior decision unit 236 provides support according to the emotions of the user 10 based on the acquired information regarding the health and medical care of at least one of the user 10 and the user's 10 family. For example, the behavior decision unit 236 reads out information regarding a response corresponding to the health and medical care information of the user 10 or the user's 10 family and the content of support to be provided from the assistance rules 223. The behavior decision unit 236 decides the behavior of the avatar 30 based on the read information.

[0052] The support rule 223 stores information on the contents of responses and support to be given to the individual and the family for each health condition and medical situation and for each issue. For example, the support rule 223 stores answer contents and support contents for each issue that occurs during pregnancy and after childbirth when the individual and the family are pregnant or after childbirth. For example, the support rule 223 stores answer contents to emotional issues that occur during pregnancy and methods for dealing with worries during pregnancy. In addition, the support rule 223 stores answer contents to emotional issues that occur after childbirth, methods for dealing with stress after childbirth, and information on childcare. For example, the support rule 223 stores answer contents to emotional issues, methods for dealing with stress, and information on childcare for each period from childbirth. In addition, the support rule 223 stores information on support for care and end-of-life care for the family when the family is a terminally ill patient. For example, the support rule 223 stores care contents and methods for end-of-life care for each issue that occurs to the terminally ill patient and the family. In addition, the support rule 223 stores information on emotional care and rehabilitation for patients who need to be hospitalized. For example, the support rule 223 stores information on care to be taken and rehabilitation for each problem that arises for patients who need to be hospitalized. In addition, the support rule 223 stores information on emotional care, vital signs, and medication for patients with chronic diseases. For example, the support rule 223 stores care to be taken, vital signs to be aware of, and precautions regarding medication for each problem that arises for patients with chronic diseases. In addition, in the support rule 223, when the patient and his / her family are raising an infant, answer contents and support contents are stored for each problem that arises during the childcare of the infant. For example, the support rule 223 stores actions related to the childcare of an infant and information for navigating the childcare. For example, the support rule 223 stores information on changing diapers, how to make powdered milk, how to soothe the infant, how to set up a child seat, and how to deal with night crying.

[0053] When the acquisition unit 240 acquires information on the health and medical care of at least one of the user 10 and the user's family, the behavior decision unit 236 provides support according to the user's feelings based on the information on the health and medical care of at least one of the user 10 and the user's family. The behavior decision unit 236 according to this embodiment identifies a problem from the content of the user's utterance. The behavior decision unit 236 analyzes character information representing the content of the user's utterance to identify a problem included in the content of the utterance. The behavior decision unit 236 reads out information on a response corresponding to the identified problem and support content from the support rule 223 in the acquired health and medical situation of the user 10 or the user's family. The behavior decision unit 236 determines the behavior of the avatar 30 based on the read information.

[0054] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the avatar 30 determined by the emotion determination unit 232.

[0055] Specifically, when a total intensity value, which is the sum of the emotional values ​​for each of the multiple emotional classifications of the avatar 30, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is decided to store data including the behavior of the user 10 in the history data 222.

[0056] The behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the avatar 30. For example, when the avatar 30 is insulted by the user 10 or is arrogantly treated by the user 10 (i.e., when the user 10 has a bad reaction), when the surrounding noise is loud and the voice of the user 10 cannot be detected, when the battery level of the avatar 30 is low, etc., and the emotion value of "anger" or "sorrow" of the avatar 30 increases, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "anger" or "sorrow". Also, when the reaction of the user 10 is good or the battery level of the avatar 30 is high, etc., and the emotion value of "joy" or "pleasure" of the avatar 30 increases, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "joy" or "pleasure". Furthermore, the behavior determining unit 236 may determine a behavior for a user 10 who has increased the emotional values ​​of "joy" or "pleasure" of the avatar 30 to be different from the behavior for the user 10 who has increased the emotional values ​​of "anger" or "sadness" of the avatar 30. In this way, the behavior determining unit 236 may determine different behaviors depending not only on the emotion of the avatar 30 itself or the behavior of the user 10 but also on how the user 10 has changed his or her emotion.

[0057] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

[0058] The behavior control unit 250 controls the avatar 30 and the control target 252 based on the behavior of the avatar 30 determined by the behavior determination unit 236. Specifically, the behavior control unit 250 controls the movement of the avatar 30 and the operation of the control target related to the movement of the avatar 30 based on the behavior of the avatar 30 corresponding to the reaction rule 221 determined by the behavior determination unit 246. For example, in a case where the avatar 30 and the user 10 are having a conversation, when the behavior determination unit 236 determines an action including speaking, the behavior control unit 250 controls the avatar 30 to take a predetermined movement and outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speech speed of the sound based on the emotion value of the avatar 30. For example, the behavior control unit 250 determines a faster speech speed as the emotion value of the avatar 30 is larger. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0059] The behavior control unit 250 may control the control target 252 to express an emotion corresponding to the estimated emotion of the user 10 in synchronization with the recognized state of the user 10. For example, in the avatar 30 according to the embodiment, the sensor module unit 210 detects the user 10 by voice, image, or the like using the sensor unit 200, and the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. Then, the emotion determination unit 232 determines the emotional state of the user 10 from the recognized state of the user 10, thereby estimating the emotion of the user 10. The behavior control unit 250 controls the control target 252 to express an emotion corresponding to the emotion of the user 10 estimated by the emotion determination unit 232 in synchronization with the state of the user 10 recognized by the user state recognition unit 230. When the emotion of the user 10 estimated by the emotion determination unit 232 is a positive state, the behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10 in synchronization with the state of the user 10 recognized by the user state recognition unit 230. For example, the behavior control unit 250 determines the state as positive when the emotion values ​​of positive emotions such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", "joy", "pleasure", and "pleasure" are large, and the emotion values ​​of negative emotions such as "anger", "sorrow", "discomfort", "anxiety", "sadness", "concern", and "emptiness" are small. On the other hand, the behavior control unit 250 determines the state as negative when the emotion values ​​of positive emotions such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", "joy", "pleasure", and "pleasure" are small, and the emotion values ​​of negative emotions such as "anger", "sorrow", "discomfort", "anxiety", "sadness", "concern", and "emptiness" are large. When the emotion of the user 10 is in a positive state, the behavior control unit 250 controls the control target 252 to express an emotion according to the estimated emotion of the user 10 in the same state as the state of the user 10. For example, the user state recognition unit 230 determines the speech speed of the voice based on the emotion value of the user 10. Furthermore, for example, the user state recognition unit 230 detects the user 10 by voice, image, or the like using the sensor unit 200, and recognizes any one of the characteristics of the user 10, such as gestures, speaking style, or choice of words.The behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10, with any one of gestures, speech, and word usage as the same characteristics as the user 10. In addition, for example, the user state recognition unit 230 recognizes any one of the facial expression, tone of voice, and nuances of words of the user 10. The behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10, with any one of the facial expression, tone of voice, and nuances of words as the same state as the state of the user 10. By the avatar 30 expressing an emotion corresponding to the emotion of the user 10 to the user 10 in this way, the psychological connection between the user 10 and the avatar 30 is improved, and the relationship is improved.

[0060] Furthermore, when the control device 100 acquires information on the health and medical care of at least one of the user 10 and the user's family who communicates with the avatar 30, the control device 100 controls the avatar 30 to provide support according to the emotions of the user 10 based on the information on the health and medical care of at least one of the user 10 and the user's family. This allows the control device 100 to provide support according to the recognized emotions of the user 10. For example, the control device 100 can help pregnant and postpartum parents navigate challenges that arise during pregnancy and after childbirth through the avatar 30. For example, the control device 100 can suggest ways to deal with worries during pregnancy and postpartum stress through the avatar 30, thereby improving parental confidence. Furthermore, the control device 100 can provide answers to emotional challenges, ways to deal with stress, and information on childcare for each period from birth through the avatar 30, and can also provide support for adapting to the life of a new family. Furthermore, the control device 100 can provide mental care through dialogue to terminally ill patients and their families through the avatar 30. The control device 100 can provide an appropriate care method and support for end-of-life care via the avatar 30, and can help patients to have a peaceful final moments. The control device 100 can provide mental care to patients who need to be hospitalized through dialogue via the avatar 30. The control device 100 can support patients who need to be hospitalized in preparing for rehabilitation and life after discharge via the avatar 30, and can provide an environment in which the patient can concentrate on recuperation with peace of mind. If the user 10 or the user's family is raising an infant, the control device 100 can help with raising the infant via the avatar 30 by having the avatar 30 show actual movements of changing diapers, making powdered milk, soothing the infant, setting up a child seat, and dealing with night crying.

[0061] The behavior control unit 250 may determine the speech rate of the voice based on the emotion value of the avatar 30. For example, when the emotion of the user 10 is in a positive state, the behavior control unit 250 determines the speech rate of the voice based on the emotion value of the avatar 30. For example, the behavior control unit 250 determines a faster speech rate as the emotion value of the avatar 30 is larger. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0062] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. When an impact is detected by the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0063] In addition, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the avatar 30, the emotion determination unit 232 further changes the emotion value of the avatar 30 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the avatar 30 when the reaction of the user 10 to the behavior determined by the behavior determination unit 236 being performed on the user 10 in the execution mode determined by the behavior control unit 250 is not bad. In addition, the emotion determination unit 232 increases the emotion value of "sadness" of the avatar 30 when the reaction of the user 10 to the behavior determined by the behavior determination unit 236 being performed on the user 10 in the execution mode determined by the behavior control unit 250 is bad.

[0064] Furthermore, the behavior control unit 250 expresses the emotion of the avatar 30 based on the determined emotion value of the avatar 30. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the avatar 30, the behavior control unit 250 controls the control object 252 to make the avatar 30 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the avatar 30, the behavior control unit 250 controls the control object 252 to make the avatar 30 assume a droopy posture.

[0065] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221.

[0066] The server 300 communicates between the server 300 and the control devices 100-102 that control the avatars 30-32, receives user reaction information transmitted from the control devices 100-102, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0067] Fig. 3 shows an example of an operational flow for determining the action of the avatar 30. The operational flow shown in Fig. 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operational flow indicates the step to be executed.

[0068] First, in step S100, the state of the user 10 communicating with the avatar 30 is recognized. For example, the user state recognition unit 230 recognizes the state of the user 10 communicating with the avatar 30 based on information analyzed by the sensor module unit 210.

[0069] In step S102, the emotion of the user 10 is estimated based on the recognized state of the user 10. For example, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0070] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the avatar 30, based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

[0071] In step S104 , the behavior recognition unit 234 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .

[0072] In step S105, the acquisition unit 240 acquires health and medical information of the user 10 and / or the user's 10 family from the server 300. For example, the acquisition unit 240 queries the server 300 for identification information of the user 10 and / or the user's 10 family. If the server 300 stores health and medical information corresponding to the identification information, the server 300 transmits the health and medical information corresponding to the identification information. The acquisition unit 240 acquires the health and medical information of the user 10 and / or the user's 10 family by receiving it from the server 300.

[0073] In step S106, the behavior determining unit 236 determines the behavior of the avatar 30. For example, the behavior determining unit 236 determines the behavior of the avatar 30 based on a combination of the current emotion value of the user 10 and the past emotion value of the user 10 determined in step S102, the emotion value of the avatar 30, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rule 221. In addition, when the behavior determining unit 236 acquires the health and medical information of at least one of the user 10 and the user 10's family in step S105, the behavior determining unit 236 provides support according to the emotion of the user 10 based on the acquired health and medical information of at least one of the user 10 and the user 10's family. For example, the behavior determining unit 236 reads out a response corresponding to the health and medical information of the user 10 or the user 10's family from the support rule 223, and information on the content of support to be provided. The behavior determining unit 236 determines the behavior of the avatar 30 based on the read information.

[0074] In step S108, the behavior control unit 250 controls the avatar 30 and the control target 252 based on the behavior determined by the behavior determination unit 236. At this time, when the emotion of the user 10 estimated by the emotion determination unit 232 is in a positive state, the behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10 in synchronization with the state of the user 10 recognized by the user state recognition unit 230. For example, when the emotion of the user 10 is in a positive state, the behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10 in the same state as the state of the user 10. Furthermore, when the emotion of the user 10 is in a positive state, the behavior control unit 250 determines the speech rate of the voice based on the emotion value of the avatar 30.

[0075] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determining unit 236 and the emotion value of the avatar 30 determined by the emotion determining unit 232.

[0076] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S114.

[0077] In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.

[0078] As described above, the control device 100 includes an estimation unit (e.g., the sensor unit 200, the sensor module unit 210, the user state recognition unit 230, and the emotion determination unit 232), an acquisition unit 240, and an avatar control unit (e.g., the behavior determination unit 236 and the behavior control unit 250). The estimation unit estimates the emotion of the user 10 who communicates with the avatar 30. The acquisition unit 240 acquires information on the health and medical care of at least one of the user 10 and the family of the user 10. The avatar control unit controls the avatar 30 so as to provide support according to the emotion of the user 10 estimated by the estimation unit based on the information on the health and medical care of at least one of the user 10 and the family of the user 10 acquired by the acquisition unit 240. In this way, the control device 100 can assist the user 10 according to the health and medical conditions of the user 10 and the family of the user 10 via the avatar 30, and can support the user 10.

[0079] Furthermore, when the user 10 or a family member of the user 10 is pregnant or has just given birth, the avatar control unit controls the avatar 30 to provide information for navigating the emotional challenge of the user 10 estimated by the estimation unit. This allows the control device 100 to assist and support the user 10 via the avatar 30 when the user 10 or a family member of the user 10 is pregnant or has just given birth.

[0080] Furthermore, when a family member of the user 10 is a terminally ill patient, the avatar control unit controls the avatar 30 to provide information related to support for the care and nursing of the family member of the user 10 in accordance with the emotion of the user 10 estimated by the estimation unit. This allows the control device 100 to assist and support the user 10 via the avatar 30 when a family member of the user 10 is a terminally ill patient.

[0081] Furthermore, the acquisition unit 240 acquires health and medical information about the user 10. When the user 10 is a patient who needs to be hospitalized, the avatar control unit controls the avatar 30 to provide information about at least one of emotional care and rehabilitation. This allows the control device 100 to assist and support the user 10 via the avatar 30 when the user 10 is a patient who needs to be hospitalized.

[0082] Furthermore, the acquisition unit 240 acquires health and medical information about the user 10. When the user 10 is a chronic disease patient, the avatar control unit controls the avatar 30 to provide information about at least one of emotional care, vital signs, and medication. In this way, when the user 10 is a chronic disease patient, the control device 100 can assist and support the user 10 via the avatar 30.

[0083] Furthermore, when the user 10 or a family member of the user 10 is raising an infant, the avatar control unit controls the avatar 30 so that the avatar shows actual movements of changing diapers, making powdered milk, soothing the infant, setting up a child seat, and dealing with night crying. In this way, when the user 10 or a family member of the user 10 is raising an infant, the control device 100 can support the parenting of the infant via the avatar 30, and can support the user 10.

[0084] In the above embodiment, the control device 100 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect. For example, the control device 100 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

[0085] The control device 100 is an example of a control device that controls an avatar. The target of behavior control by the control device 100 is not limited to the avatar 30, and the control device 100 can be applied to various electronic devices, etc. The functions of the control device 100 and the server 300 may be implemented by one or more computers. At least some of the functions of the control device 100 and the server 300 may be implemented by a virtual machine. At least some of the functions of the control device 100 and the server 300 may be implemented in the cloud.

[0086] 4 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the control device 100 and the server 300. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0087] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0088] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0089] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0090] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0091] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0092] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0093] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0094] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0095] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0096] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0097] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0098] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0099] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0100] (Other embodiments) The above-described control device 100 may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on a stuffed toy.

[0101] The emotion determining unit 232 may determine the emotion of the user 10 according to a specific mapping. Specifically, the emotion determining unit 232 may determine the emotion of the user 10 according to an emotion map (see FIG. 5), which is a specific mapping.

[0102] FIG. 5 is a diagram showing an emotion map 700 on which multiple emotions are mapped. In the emotion map 700, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 700, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0103] (1) For example, if the emotion engine, which is the emotion determination unit 232 that determines the emotion of the avatar 30, detects emotions in about 100 msec, the frequency of determining the reaction action of the avatar 30 (e.g., a backchannel) may be set to at least the same timing as the emotion engine's detection frequency (100 msec), or may be set to an earlier timing. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0104] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. The avatar 30 performs a reaction (such as a backchannel) according to the directionality and the degree (strength) of the mandala in the emotion map 700. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user 10, etc.

[0105] (2) The directionality of emotions and the strength of their intensity may be preset in reference to emotion map 700, and the movement of the interjections and the strength of the interjections may be set. For example, if avatar 30 feels a sense of stability and security, avatar 30 may nod and continue listening. If avatar 30 feels anxious, confused, or suspicious, avatar 30 may tilt its head or stop shaking its head.

[0106] These emotions are distributed in the three o'clock direction of emotion map 700 and usually fluctuate between relief and anxiety. In the right half of emotion map 700, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0107] (3) If the avatar 30 feels good after being praised, the filler "ah" may precede the line, and if the avatar 30 feels pain after receiving harsh words, the filler "ugh!" may precede the line. Also, a physical reaction such as the avatar 30 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 700.

[0108] (4) In the left half of the emotion map 700, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

[0109] When the avatar 30 feels a favorable feeling in its situational awareness while having an internal feeling (reaction) of a sense of satisfaction, the avatar 30 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the avatar 30 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 700.

[0110] Conversely, if the avatar 30 is aware of an internal sensation (reaction) of discomfort, the avatar 30 may shake its head when it feels disgust, or glare at the other person when it feels hatred. These types of emotions are distributed around the 6 o'clock position on the emotion map 700.

[0111] (5) The inside of emotion map 700 represents what is going on in one's mind, while the outside of emotion map 700 represents behavior. Therefore, the further out on the outside of emotion map 700 you go, the more visible the emotions become (the more they are expressed in behavior).

[0112] (6) When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock area of ​​emotion map 700, avatar 30 may lightly nod his head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, he may nod vigorously, nodding his head deeply.

[0113] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the user 10. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 700. In addition, this neural network is trained so that emotions that are arranged close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 is a diagram showing another example of an emotion map. FIG. 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "encouraging," have similar emotion values.

[0114] Furthermore, the emotion determination unit 232 may determine the emotion of the avatar 30 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the avatar 30 into a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 700, and determines the emotion of the avatar 30. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the avatar 30, and emotion values ​​indicating each emotion shown in the emotion map 700. For example, the neural network is trained based on learning data that indicates that when it is recognized from the output of the touch sensor that the avatar 30 is being stroked by the user 10, the emotion value becomes "3" which indicates "happiness," and learning data that indicates that when it is recognized from the output of the acceleration sensor that the avatar 30 is being hit by the user 10, the emotion value becomes "3" which indicates "anger." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0115] The emotion determination unit 232 may determine the emotion of the avatar 30 based on the action content of the avatar 30 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the action content of the avatar 30 generated by the sentence generation model into a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 700, and integrates the acquired emotion values ​​indicating each emotion with the emotion values ​​indicating each emotion of the current avatar 30 to update the emotion of the avatar 30. For example, the emotion values ​​indicating each acquired emotion and the emotion values ​​indicating each emotion of the current avatar 30 are averaged and integrated. This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the action content of the avatar 30 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 700. For example, if the speech content of avatar 30 generated by the sentence generation model is "That's great. You're lucky.", then when the text representing this speech content is input into the neural network, a high emotion value for the emotion "happy" is obtained, and the emotion of avatar 30 is updated so that the emotion value of the emotion "happy" becomes higher.

[0116] The behavior determination unit 236 generates the behavior content of the avatar 30 by adding fixed sentences for asking about the behavior content of the avatar 30 corresponding to the behavior of the user 10 to text representing the behavior of the user 10, the emotions of the user 10, and the emotions of the avatar 30, and inputting the added text into a sentence generation model with an interactive function.

[0117] For example, the behavior determining unit 236 obtains text representing the state of the avatar 30 from the emotion of the avatar 30 determined by the emotion determining unit 232, using an emotion table as shown in Fig. 7. Fig. 7 is a diagram showing an example of an emotion table. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the avatar 30 is stored for each index number.

[0118] When the emotion of the avatar 30 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the avatar 30 correspond to multiple index numbers, multiple texts representing the state of the avatar 30 are obtained.

[0119] Also, for the emotions of the user 10, an emotion table as shown in FIG. 8 is prepared. FIG. 8 is a diagram showing an example of an emotion table. Here, if the behavior of the user 10 is speaking "I'm AAA," the emotion of the avatar 30 is index number "2," and the emotion of the user 10 is index number "3," then "The avatar is in a very happy state. The user is in a normal happy state. The user spoke to me saying, "I'm AAA. How would you respond as the avatar?" is input to the sentence generation model, and the behavior content of the avatar 30 is obtained. The behavior decision unit 236 decides the behavior of the avatar 30 from this behavior content. Note that "AAA" is the name (nickname) given to the avatar 30 by the user 10.

[0120] In this way, the control device 100 can change the behavior of the avatar 30 according to the index number corresponding to the emotion of the avatar 30, so that the user 10 gets the impression that the avatar 30 has a heart, and is encouraged to take actions such as talking to the avatar 30.

[0121] The behavior determining unit 236 may generate the behavior content of the avatar 30 by adding not only texts representing the behavior of the user 10, the emotions of the user 10, and the emotions of the avatar 30, but also texts representing the contents of the history data 222, adding a fixed sentence for asking about the behavior content of the avatar 30 corresponding to the behavior of the user 10, and inputting the result into a sentence generation model having a dialogue function. This allows the control device 100 to change the behavior of the avatar 30 according to the history data representing the emotions and behavior of the user 10, so that the user 10 has the impression that the avatar 30 has individuality, and is encouraged to take a behavior such as talking to the avatar 30. The history data may further include the emotions and behavior of the avatar 30.

[0122] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0123] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0124] 5 system, 10, 11, 12 user, 20 communication network, 30, 31, 32 avatar, 100, 101, 102 control device, 200 sensor unit, 201 microphone, 202 3D depth sensor, 203 2D camera, 204 distance sensor, 210 sensor module unit, 211 voice emotion recognition unit, 212 speech understanding unit, 213 facial expression recognition unit, 214 face recognition unit, 220 storage unit, 221 reaction rule, 222 history data, 230 user state recognition unit, 232 emotion determination unit, 234 behavior recognition unit, 236 behavior determination unit, 238 memory control unit, 240 acquisition unit, 250 behavior control unit, 252 control target, 280 communication processing unit, 300 server, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip

Claims

1. A control device for controlling an avatar, An estimation unit that estimates an emotion of a user who communicates with the avatar; An acquisition unit that acquires health and medical information of at least one of the user and the user's family; an avatar control unit that controls the avatar so as to provide support according to the emotion of the user estimated by the estimation unit based on health and medical information of at least one of the user and the user's family acquired by the acquisition unit; A control device comprising:

2. The avatar control unit controls the avatar to provide information for navigating the emotional challenge of the user estimated by the estimation unit when the user or the user's family member is pregnant or has just given birth. The control device according to claim 1 .

3. The avatar control unit controls the avatar to provide information regarding support for care and nursing of the user's family member in accordance with the emotion of the user estimated by the estimation unit when the family member of the user is a terminally ill patient. The control device according to claim 1 .

4. The acquisition unit acquires health and medical information of the user, The avatar control unit controls the avatar to provide information regarding at least one of emotional care and rehabilitation if the user is a patient who needs hospitalization. The control device according to claim 1 .

5. The acquisition unit acquires health and medical information of the user, The avatar control unit controls the avatar to provide information regarding at least one of emotional care, vital signs, and medication if the user is a chronic disease patient. The control device according to claim 1 .

6. The avatar control unit controls the avatar so that the avatar shows actual movements of changing diapers, making powdered milk, soothing the baby, setting up a child seat, and dealing with night crying when the user or the user's family is raising a baby. The control device according to claim 1 .

Images