Behavior control system, control device, electronic apparatus, and avatar display device

Abstract

A behavior control system according to an embodiment of the present invention recognizes the behavior of an artist and determines the behavior of an avatar corresponding to the recognized behavior of the artist. Then, the behavior control system controls the avatar on the basis of the determined behavior of the avatar.

Description

Behavior control system, control device, electronic device, and avatar display device

[0001] The present invention relates to a behavior control system, a control device, an electronic device, and an avatar display device.

[0002] Patent Literature 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Literature 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

[0003] Patent No. 6053847

[0004] However, in the conventional technology, it is sometimes impossible to output appropriate information according to the user's emotions.

[0005] According to a first aspect of the present invention, the system includes a behavior determination unit that recognizes the behavior of an artist and determines the behavior of an avatar corresponding to the recognized behavior of the artist, and a behavior control unit that controls the avatar based on the behavior of the avatar determined by the behavior determination unit.

[0006] According to one aspect of the embodiment, it is possible to output appropriate information according to the user's emotion.

[0007] FIG. 1 is a diagram schematically illustrating an example of a system according to a first embodiment. FIG. 2 is a diagram schematically illustrating the functional configuration of a control device that controls an avatar. FIG. 3 is a diagram schematically illustrating the data structure of character data. FIG. 4 is a diagram schematically illustrating an example of an operation flow related to character setting. FIG. 5 is a diagram schematically illustrating an example of an operation flow related to an operation for determining the behavior of an avatar. FIG. 6 is a diagram schematically illustrating an example of the hardware configuration of a computer that functions as a control device and a server. FIG. 7 is a diagram illustrating an emotion map to which multiple emotions are mapped. FIG. 8 is a diagram illustrating another example of an emotion map. FIG. 9 is a diagram illustrating an example of an emotion table. FIG. 10 is a diagram illustrating an example of an emotion table. FIG. 11 is a diagram schematically illustrating the functional configuration of a control device. FIG. 12 is a diagram schematically illustrating an example of an operation flow related to an operation for determining the behavior of an avatar. FIG. 13 is a diagram schematically illustrating the functional configuration of a support unit. FIG. 14 is a diagram schematically illustrating an example of an operation flow performed by the support unit. FIG. 15 is a diagram schematically illustrating the data structure of character data. FIG. 16 is a diagram schematically illustrating the functional configuration of an output unit. FIG. 17 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 18 is a diagram schematically illustrating the functional configuration of the output unit. FIG. 19 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 20 is a diagram schematically illustrating the functional configuration of a control device that controls an avatar. FIG. 21 is a diagram schematically illustrating an example of an operation flow related to an operation that determines the behavior of an avatar. FIG. 22 is a diagram schematically illustrating the functional configuration of the output unit. FIG. 23 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 24 is a diagram schematically illustrating the functional configuration of the output unit. FIG. 25 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 26 is a diagram schematically illustrating the functional configuration of the output unit. FIG. 27 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 28 is a diagram schematically illustrating the functional configuration of the output unit. FIG. 29 is a diagram schematically illustrating an example of an operation flow by the output unit. FIG. 30 is a diagram schematically illustrating an example of an operation flow by the support unit. Fig. 31 is a diagram schematically illustrating an example of an operation flow by the output unit. Fig. 32 is a diagram schematically illustrating an example of an operation flow by the output unit.FIG. 33 is a diagram schematically showing the functional configuration of a control device that controls an avatar. FIG. 34 is a diagram schematically showing the functional configuration of an event detection unit. FIG. 35 is a diagram schematically showing an example of an operational flow by the event detection unit. FIG. 36 is a diagram schematically showing an example of a system according to the fifteenth embodiment. FIG. 37 is a diagram schematically showing the functional configuration of a control device that controls an avatar. FIG. 38 is a diagram schematically showing an example of an operational flow related to an operation for determining the behavior of an avatar. FIG. 39 is a diagram schematically showing an example of a control system according to the sixteenth embodiment. FIG. 40 is a diagram schematically showing the functional configuration of a control device that controls an avatar. FIG. 41 is a diagram schematically showing an example of an operational flow related to an operation for determining the behavior of an avatar in the control device. FIG. 42 is a diagram schematically showing an example of a control system according to the nineteenth embodiment. FIG. 43 is a diagram schematically showing the functional configuration of a control device. FIG. 44 is a diagram schematically showing an example of a control system according to the twentieth embodiment. FIG. 45 is a diagram schematically showing the functional configuration of a control device that controls an avatar. FIG. 46 is a diagram schematically illustrating an example of an operational flow relating to an operation for determining the behavior of an avatar.

[0008] (First Embodiment) The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] 1 is a diagram schematically illustrating an example of a system 5 according to this embodiment. The system 5 includes control devices 200-202 that control avatars such as avatar 100, avatar 101, and avatar 102, and a server 300. The control device 200 that controls the avatar 100 is connected wirelessly or by wire to a display device such as a PC monitor or a smartphone display, or to a predetermined wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality. The control device 200 includes various input / output devices such as a speaker that outputs sounds such as voice, a microphone that detects sounds such as voice, and a camera.

[0010] User 10a, user 10b, user 10c, and user 10d are users who communicate with avatar 100. User 11a, user 11b, and user 11c are users who communicate with avatar 101. User 12a and user 12b are users who communicate with avatar 102. In the description of this embodiment, user 10a, user 10b, user 10c, and user 10d may be collectively referred to as user 10. Furthermore, user 11a, user 11b, and user 11c may be collectively referred to as user 11. Furthermore, user 12a and user 12b may be collectively referred to as user 12. The avatar 101 and control device 201, and the avatar 102 and control device 202 have substantially the same functions as the avatar 100 and control device 200. Therefore, system 5 will be described mainly focusing on the functions of avatar 100.

[0011] As shown in FIG. 1, avatars 100-102 controlled by control devices 200-202 communicate with users 10-12 in various forms. For example, as shown in FIG. 1 (1), avatar 100 is represented as a person or the like displayed on a display device or the like, and communicates with user 10 via an input / output device provided in control device 200. As shown in FIG. 1 (2), avatar 101 communicates in a virtual space with user 11, who is displayed as a person or character on a display device or the like. As shown in FIG. 1 (3), avatar 102 can also be displayed as a character rather than a person, and communicates with user 12 as that character. Note that, although a display monitor has been described as an example of the above-mentioned display device or the like in FIG. 1, the present invention is not limited to this. For example, in this embodiment, the display device may include a monitor, a display unit of a notebook PC or a mobile terminal, a wearable terminal device related to augmented reality, virtual reality, mixed reality, etc., such as VR / AR goggles, VR / AR glasses, and other wearable devices that provide information perceived by the user's five senses.

[0012] The appearance of the avatar may be a human figure, as in avatar 100 and avatar 101, or a character, as in avatar 102. For example, avatars 100 and 101 may be disguised as counselors, nodding and listening like a real counselor, to provide counseling to businesspeople with labor issues, etc., by providing emotional support, appropriate ways to deal with worries and stress in the workplace, and self-care methods. For example, avatar 102, with its character-like appearance, is thought to be particularly familiar to children.

[0013] The avatar 100 is controlled by the control device 200, and converses with the user 10 and provides images to the user 10 via input / output devices provided in the control device 200. At this time, the control device 200 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to conduct a conversation between the avatar 100 and the user 10 and provide images, etc. to the user 10. For example, the control device 200 not only learns appropriate conversations by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The control device 200 also records captured image data, etc. of the user 10 in the server 300, and requests the image data, etc. from the server 300 as needed and provides it to the user 10.

[0014] The control device 200 also stores emotion values ​​that represent the types of emotions of the avatar 100 itself. For example, the control device 200 stores emotion values ​​representing the intensity of each of the following emotions: "joy," "anger," "sadness," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral" as the emotion values ​​of the avatar 100. For example, when the user 10 and the avatar 100 are conversing and the excitement emotion value is high, the control device 200 controls the avatar 100 so that the avatar 100 speaks at a fast speed. In this way, the avatar 100 can express its own emotions through its actions under the control of the control device 200.

[0015] Furthermore, the control device 200 may be configured to match a sentence generation model (a so-called AI (Artificial Intelligence) chat engine) with an emotion engine, thereby determining the behavior of the avatar 100 corresponding to the emotion of the user 10. Specifically, the control device 200 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 in response to the user's behavior, and determine the behavior of the avatar 100 corresponding to the determined emotion.

[0016] More specifically, when the control device 200 recognizes the behavior of the user 10, it automatically generates behavioral content that the avatar 100 should take in response to the user 10's behavior using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic text-based dialogue processing. The sentence generation model is publicly known, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444, and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large-scale language model (LLM). As described above, this embodiment combines a large-scale language model with an emotion engine to reflect the emotions of the user 10 or the avatar 100 and various linguistic information in the behavior of the avatar 100. In other words, according to this embodiment, a synergistic effect can be achieved by combining the sentence generation model with the emotion engine.

[0017] The control device 200 recognizes the artist's behavior and determines the behavior of the avatar 100 that corresponds to the recognized artist's behavior. The control device 200 controls the avatar based on the determined behavior of the avatar.

[0018] An artist is a person who is involved in creative activities. Artists include people who create and produce works of art. For example, artists include sculptors, painters, directors, and writers. Artists also include performers and musicians.

[0019] The control device 200 determines behavior related to the artist's performance. The control device 200 also determines behavior related to the artist's creativity. The control device 200 also determines behavior related to the artist's expressiveness. Note that in the following sections, the user 10 will be described as an artist, but this is not limiting, and for example, the avatar 100 may also be an artist.

[0020] The control device 200 also has a function of recognizing the behavior (state) of the user 10. The control device 200 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The control device 200 determines the behavior to be performed by the avatar 100 based on the recognized behavior of the user 10, etc.

[0021] The control device 200 stores rules that define actions to be taken by the avatar 100 based on the emotions of the user 10, the emotions of the avatar 100 itself, and the actions of the user 10, and performs various actions in accordance with the rules.

[0022] Specifically, the control device 200 stores reaction rules for determining the behavior of the avatar 100 based on the emotions of the user 10, the emotions of the avatar 100, and the behavior of the user 10. For example, the reaction rules define the behavior of "laughing" as the behavior of the avatar 100 when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of "apologizing" as the behavior of the avatar 100 when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of "answering" as the behavior of the avatar 100 when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of "calling out" as the behavior of the avatar 100 when the behavior of the user 10 is "sad."

[0023] When the control device 200 recognizes that the behavior of the user 10 is "angry" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the avatar 100. For example, when the behavior of "apologizing" is selected, the avatar 100 performs the motion of "apologizing" and outputs a sound representing the words "apologize."

[0024] Furthermore, when the condition that the emotion of the avatar 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the state of the user 10 is "alone and looks lonely" is met, it is determined that the emotion of the avatar 100 will change to "worried" and the action of "calling out" can be performed.

[0025] When the control device 200 recognizes, based on the reaction rules, that the current emotion of the avatar 100 is "normal" and that the user 10 appears lonely, the control device 200 increases the emotion value of "sadness" of the avatar 100. Furthermore, the control device 200 selects the action of "calling out" defined in the reaction rules as the action that the avatar 100 will perform toward the user 10. For example, when the control device 200 selects the action of "calling out," the control device 200 converts the phrase "What's wrong?", which expresses concern, into a worried voice of the avatar 100 and outputs it. In this way, the action of the avatar 100 is determined according to, for example, the emotion or action of the user 10, the emotion of the avatar 100, and its emotion value.

[0026] The control device 200 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the avatar 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and the attributes of the user 10. The user reaction information may also include a negative reaction.

[0027] The control device 200 stores user reaction information regarding user reactions to the avatar 100. The server 300 receives and stores user reaction information not only from the control device 200 that controls the avatar 100, but also from the control devices 201 and 202 that control the avatars 101 and 102. The server 300 then analyzes the user reaction information from the control devices 200, 201, and 202 and updates the reaction rules. In other words, the server 300 analyzes user reaction information from other control devices and updates the reaction rules.

[0028] The control device 200 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The control device 200 incorporates the updated reaction rules into the reaction rules stored in the control device 200. This allows the control device 200 to incorporate the reaction rules acquired by the control device 201 and the control device 202 of the avatar 101, avatar 102, etc. into its own reaction rules. When the reaction rules are updated, the updated reaction rules may be automatically transmitted from the server 300 to the control device 200.

[0029] 2 is a diagram illustrating a schematic functional configuration of the control device 200. The control device 200 includes a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291.

[0030] The control target 252 includes a display device 2521 and a speaker 2522. The display device 2521 displays the avatar 100 itself and images related to the avatar 100. The speaker 2522 outputs sounds related to the conversation of the avatar 100, the actions of the avatar 100, and the like.

[0031] The sensor unit 210 includes a microphone 211, a 3D depth sensor 212, a 2D camera 213, and a distance sensor 214. The microphone 211 continuously detects sound and outputs audio data. The microphone 211 may be provided on a dummy head, a simulator, or the like and have a binaural recording function. The 3D depth sensor 212 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 213 is an example of an image sensor. The 2D camera 213 captures images using visible light and generates visible light video information. The distance sensor 214 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 210 may also include a thermograph, a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, or the like.

[0032] The storage unit 230 includes reaction rules 231, history data 232, and character data 233. The history data 232 includes the user 10's past emotional values ​​and behavioral history. This emotional value and behavioral history is recorded for each user 10, for example, by being associated with the user 10's identification information. At least a portion of the storage unit 230 is implemented as a storage medium such as a memory. It may also include a person DB that stores the user 10's facial image, attribute information, etc. Among the components of the control device 200 shown in FIG. 2 , the functions of the components excluding the control target 252, the sensor unit 210, and the storage unit 230 can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using basic software (OS) and programs running on the OS.

[0033] The character data 233 is data that associates a character with an age. For example, a character may be a person who appears in existing content such as animation, video games, manga, or movies. Furthermore, a character may be an animal or plant with a personality, or may be an inanimate object (such as a robot).

[0034] For example, the age (user age) associated with a character in the character data 233 is determined based on the age group of viewers expected to be the target audience for the content in which the character appears. Figure 3 is a diagram showing a schematic data structure of the character data 233.

[0035] For example, assume that character "A" appears in an animation aimed at kindergarten children. In this case, as shown in Figure 3, character "A" is associated with a user age of "3 to 7 years old."

[0036] For example, if a movie featuring a character "C" contains violent scenes and is not suitable for viewing by young children, the user age of "12 years or older" is associated with the character "C" as shown in FIG.

[0037] The age in the character data 233 may be determined based on an age rating from a rating organization such as the Pan European Game Information (PEGI), the Motion Picture Ethics Organization, or the Computer Entertainment Rating Organization (CERO). The age of use may be determined as a range such as "3 to 5 years old" or "12 years old or older," or as a single value such as "10 years old" or "15 years old."

[0038] The sensor module unit 220 includes a voice emotion recognition unit 221, a speech understanding unit 222, a facial expression recognition unit 223, and a face recognition unit 224. Information detected by the sensor unit 210 is input to the sensor module unit 220. The sensor module unit 220 analyzes the information detected by the sensor unit 210 and outputs the analysis result to the state recognition unit 240.

[0039] The voice emotion recognition unit 221 of the sensor module unit 220 analyzes the voice of the user 10 detected by the microphone 211 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 221 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 222 analyzes the voice of the user 10 detected by the microphone 211 and outputs text information representing the content of the utterance of the user 10.

[0040] The facial expression recognition unit 223 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 213. For example, the facial expression recognition unit 223 recognizes the facial expression and emotion of the user 10 based on the shape, positional relationship, etc. of the eyes and mouth.

[0041] The face recognition unit 224 recognizes the face of the user 10. The face recognition unit 224 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 213.

[0042] The state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 mainly performs processing related to perception using the analysis results of the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is only one user," or "There is a 90% chance that the user is not smiling." A process is performed to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "There is only one user, he looks lonely." Furthermore, for example, the state recognition unit 240 generates perceptual information such as "Dad is alone," or "There is a 90% chance that Dad is not smiling." For example, the state recognition unit 240 generates semantic information such as "Dad is alone, he looks lonely."

[0043] Furthermore, the state recognition unit 240 mainly performs processing related to perception, for example, using the analysis results of the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is a 90% chance that the user is not smiling." Furthermore, the state recognition unit 240 performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "One user is worried."

[0044] When the user 10 is recognized, the state recognition unit 240 reads out information about the user 10. The information about the user 10 is linked to a facial image of the user 10 and stored in the storage unit 230. The information about the user 10 includes the name of the user 10, the occupation of the user 10, the age of the user 10, and the like. If the user 10 is an artist, the information about the user 10 may include information about the user 10's past performances, for example, information about works created by the user 10 in the past and information about videos in which the user 10 has appeared in the past. For example, the state recognition unit 240 may estimate the occupation of the recognized user 10 from the content of past utterances by the user 10, etc.

[0045] The emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0046] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0047] Furthermore, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0048] The emotion value of the avatar 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."

[0049] Specifically, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 in accordance with rules for updating the emotion value of the avatar 100 that are determined in correspondence with the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240.

[0050] For example, if the state recognition unit 240 recognizes that the user 10 looks lonely, the emotion determination unit 242 increases the emotion value of "sadness" of the avatar 100. Also, if the state recognition unit 240 recognizes that the user 10 is smiling, the emotion determination unit 242 increases the emotion value of "joy" of the avatar 100.

[0051] The emotion determination unit 242 may determine the emotion value indicating the emotion of the avatar 100 by further considering the state of the avatar 100. For example, the emotion determination unit 242 may increase the emotion value of "sadness" of the avatar 100 when the user 10 does not say anything to the avatar 100 or responds in a disorganized manner. Furthermore, the emotion determination unit 242 may increase the emotion value of "anger" when the user 10 continues to respond in a disorganized manner.

[0052] The behavior recognition unit 244 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavior categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavior category with the highest probability is recognized as the behavior of the user 10.

[0053] As described above, in this embodiment, the control device 200 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the control device 200 obtains the necessary consent from the user 10 in accordance with laws and regulations, and the behavior control system for the avatar 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0054] The behavior determination unit 246 determines an avatar behavior corresponding to the behavior of the user 10 based on the emotion value of the user 10 and the emotion value of the avatar. Specifically, the behavior determination unit 246 determines an behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 244 based on the current emotion value of the user 10 determined by the emotion determination unit 242, history data 232 of past emotion values ​​determined by the emotion determination unit 242 before the current emotion value of the user 10 was determined, and the emotion value of the avatar 100. In this embodiment, a case is described in which the behavior determination unit 246 uses one most recent emotion value included in the history data 232 as the user 10's past emotion value, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 246 may use a plurality of most recent emotion values ​​as the user 10's past emotion value, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determining unit 246 may determine a behavior corresponding to a behavior of the user 10 by taking into consideration not only the current emotional value of the avatar 100 but also the history of past emotional values ​​of the avatar 100. The behavior determined by the behavior determining unit 246 includes gestures made by the avatar 100 or speech content of the avatar 100.

[0055] The behavior determination unit 246 according to the present embodiment determines, as a behavior corresponding to the behavior of the user 10, a behavior of the avatar 100 that corresponds to the reaction rule 231 when a combination including at least one of the past emotion value and the current emotion value of the user 10, the emotion value of the avatar 100, and the behavior of the user 10 satisfy the conditions of the preset reaction rule. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior determination unit 246 determines, as a behavior corresponding to the behavior of the user 10, a behavior that changes the emotion value of the user 10 to a positive value.

[0056] The behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the avatar 100. For example, if the avatar 100 is verbally abused by the user 10 or is arrogantly treated by the user 10 (i.e., if the user's reaction is poor), if the user's voice cannot be detected due to loud ambient noise, or if the emotion value of "anger" or "sadness" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 that corresponds to the behavior of the user 10, in accordance with the increase in the emotion value of "anger" or "sadness." Furthermore, if the user's reaction is good or the emotion value of "joy" or "happiness" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 that corresponds to the behavior of the user 10, in accordance with the increase in the emotion value of "joy" or "happiness." Furthermore, the behavior determining unit 246 may determine a behavior for the user 10 who has increased the emotional value of "joy" or "happiness" of the avatar 100 that is different from the behavior for the user 10 who has increased the emotional value of "anger" or "sadness" of the avatar 100. In this way, the behavior determining unit 246 may determine a different behavior depending on the emotion of the avatar 100 itself or how the user 10 has changed the emotion of the avatar 100 through the action of the user 10.

[0057] The reaction rule 231 defines a combination of the past and present emotional values ​​of the user 10, the emotional value of the avatar 100, and the behavior of the avatar 100 according to the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the present emotional value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the avatar 100.

[0058] For example, the reaction rule 231 defines behaviors of the avatar 100 for all combinations of patterns of the avatar 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values ​​and current emotional values, and behavioral patterns of the user 10. That is, for each pattern of the avatar 100's emotional values, behaviors of the avatar 100 corresponding to the user 10's behavioral patterns are defined for each of a plurality of combinations of the user 10's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 246 may determine the behavior of the avatar 100 using the history data 232 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0059] Note that the reaction rules 231 may define at least one gesture or one utterance as the behavior of the avatar 100 for each of the patterns (1296 patterns) of the emotional value of the avatar 100. Alternatively, the reaction rules 231 may define at least one gesture or one utterance as the behavior of the avatar 100 for each group of patterns of the emotional value of the avatar 100.

[0060] The strength of each gesture included in the behavior of avatar 100 defined in reaction rule 231 is predetermined. The strength of each utterance included in the behavior of avatar 100 defined in reaction rule 231 is predetermined.

[0061] For example, the reaction rule 231 may define behavior of the avatar 100 according to the user 10 who is an artist. For example, the reaction rule 231 may define behavior related to the performance of the user 10.

[0062] For example, the behavior determining unit 246 determines a behavior related to the creativity of the user 10 who is an artist. For example, the behavior determining unit 246 determines a behavior that will bring out inspirational creativity in the user 10. The behavior determining unit 246 may also determine a behavior that will improve management of a project in which the user 10 is involved.

[0063] The behavior determining unit 246 may also determine behavior related to the expressiveness of the user 10 who is an artist. For example, the behavior determining unit 246 may determine behavior that will improve the self-expression of the user 10. The behavior determining unit 246 may also determine behavior related to emotion management of the user 10. The behavior determining unit 246 may also determine behavior related to mental training of the user 10. Specifically, the behavior determining unit 246 determines behavior that suggests ways to relieve tension for the user 10 or suggests communication techniques with the audience.

[0064] The memory control unit 248 determines whether or not to store data including the behavior of the user 10 in the history data 232 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0065] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the avatar 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 246, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 246, is equal to or greater than a threshold value, it is decided to store data including the behavior of the user 10 in the history data 232.

[0066] When the memory control unit 248 decides to store data including the behavior of the user 10 in the history data 232, it stores in the history data 232 the behavior determined by the behavior determination unit 246, information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 240 (for example, the facial expression, emotions, etc. of the user 10).

[0067] The behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior of the avatar 100 determined by the behavior determination unit 246. Specifically, the behavior control unit 250 controls the movement of the avatar 100 and the operation of the control target 252 related to the movement of the avatar 100 based on the behavior of the avatar 100 associated with the reaction rule 231 determined by the behavior determination unit 246. For example, in a case where the avatar 100 and the user 10 are having a conversation, if the behavior determination unit 246 determines an action that includes speaking, the behavior control unit 250 controls the avatar 100 to take a predetermined movement and causes the speaker 2522 included in the control target 252 to output the voice of the avatar 100. At this time, the behavior control unit 250 may determine the speech rate of the voice based on the emotion value of the avatar 100. For example, the behavior control unit 250 determines a faster speech rate as the emotion value of the avatar 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determining unit 246 based on the emotion value determined by the emotion determining unit 242 .

[0068] The behavior control unit 250 may recognize a change in the user 10's emotions in response to the execution of the behavior determined by the behavior determination unit 246. For example, the change in emotions may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotions of the user 10 may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 210. If an impact is detected by the touch sensor included in the sensor unit 210, the behavior control unit 250 may recognize that the user 10's emotions have worsened. Alternatively, if the detection result of the touch sensor included in the sensor unit 210 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotions have improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0069] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 246 in the execution mode determined according to the emotion of the avatar 100, the emotion determination unit 242 further changes the emotion value of the avatar 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 242 increases the emotion value of "joy" of the avatar 100 when the user's reaction to the behavior determined by the behavior determination unit 246 being executed in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 242 increases the emotion value of "sad" of the avatar 100 when the user's reaction to the behavior determined by the behavior determination unit 246 being executed in the execution mode determined by the behavior control unit 250 is negative.

[0070] Furthermore, the behavior control unit 250 expresses the emotion of the avatar 100 based on the determined emotion value of the avatar 100. For example, if the behavior control unit 250 increases the emotion value of "happiness" of the avatar 100, it controls the movement of the avatar 100 to make the avatar 100 perform a happy gesture. Also, if the behavior control unit 250 increases the emotion value of "sadness" of the avatar 100, it controls the movement of the avatar 100 to make the avatar 100 assume a drooping posture.

[0071] For example, if the user 10 says, "I was nervous," and the user 10 is laughing, the emotional value of "joy" of the avatar 100 is increased, and the behavior control unit 250 makes an utterance such as, "You're performing well," etc. Also, if the user 10 says, "I was nervous," and the user 10 is sad, the emotional value of "sad" of the avatar 100 is increased, and the behavior control unit 250 makes an utterance such as, "I'm sure you can do it, A," or "Next time, let's try BBB before the performance."

[0072] In this way, for example, the behavior control unit 250 may cause the user 10 to execute a behavior related to emotion management or a behavior suggesting a method for alleviating tension. The behavior control unit 250 may also cause the user 10 to execute a behavior suggesting communication techniques with the audience. The behavior control unit 250 may also cause the user 10 to execute a behavior to improve self-expression. The behavior control unit 250 may also cause the user 10 to execute a behavior related to mental training. In other words, the behavior control unit 250 may cause the user 10 to execute a behavior related to expressiveness.

[0073] The behavior control unit 250 may also cause the user 10 to perform an action related to the creativity of the user 10. The behavior control unit 250 may also cause the user 10 to perform an action that draws out inspirational creativity in the user 10. The behavior control unit 250 may also cause the user 10 to perform an action that improves management of a project in which the user 10 is involved.

[0074] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 231.

[0075] The output unit 291 realizes the above-mentioned output function. Details of the output unit 291 will be described later.

[0076] The server 300 communicates between the server 300 and the control devices 200-202 that control the avatars 100-102, receives user reaction information transmitted from the control devices 200-202, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0077] (Determining Behavior Based on Character) So far, the description has been given of the case where the behavior determination unit 246 determines the behavior of the avatar 100 based on the state recognized by the state recognition unit 240. However, the behavior determination unit 246 may determine the behavior of the avatar 100 based not only on the state of the user but also on the set character. In this case, the behavior determination unit 246 may obtain an age (user age) associated with the character from the character data 233, and determine the behavior of the avatar 100 based on the obtained user age.

[0078] That is, the behavior determining unit 246 determines the behavior of the avatar 100 based on the state recognized by the state recognizing unit 240 and the set character or the age associated with the character. This makes it possible to cause the avatar 100 to perform an appropriate behavior according to the age of the user. In particular, it becomes possible to restrict the avatar 100 from performing actions that are inappropriate for young users (for example, outputting violent content).

[0079] Characters are set in advance in the system 5. The character settings are input as prompts (command statements). The prompts may be input via an input device provided in the control device 200, or via an external device such as a server communicatively connected to the control device 200. The prompt may specify the name of the character, or may specify an ID defined for each character.

[0080] Furthermore, for example, the behavior determining unit 246 determines an behavior that causes information to be output to the display device 2521 or the speaker 2522 (examples of an output device) provided in the control device 200 in a manner according to the age of the user. For example, the behavior determining unit 246 changes the voice of the avatar 100 emitted from the speaker 2522 to the tone of voice of the character.

[0081] Furthermore, for example, the behavior determining unit 246 determines the behavior of outputting a voice or a message using text composed of words appropriate for the user's age. Here, it is assumed that usable words for each age are set in advance. The behavior determining unit 246 acquires the user's age from the character data 233.

[0082] For example, it is assumed that the words "What's wrong?" and "How are you doing?" are stored in advance in the storage unit 230 as words to be output when the avatar 100 performs the action of "calling out" to the user 10. It is also assumed that "What's wrong?" is associated with an age of "under 12 years old" and "How are you doing?" is associated with an age of "12 years old or older." For example, the behavior determining unit 246 determines to output the words "How are you doing?" if the user age is "18 years old or older." It is also assumed that the behavior determining unit 246 determines to output the words "What's wrong?" if the user age is "3 to 7 years old."

[0083] In this way, the control device 200 can change the tone of voice and the words output depending on the age of the user, thereby restricting behavior that is particularly inappropriate for younger users, while increasing familiarity with the device for younger users.

[0084] Furthermore, the action determining unit 246 determines an action to output content corresponding to the character to an output device (such as the display device 2521). For example, the action determining unit 246 determines an action to display video content (such as a movie or animation) in which the character appears on the display device 2521.

[0085] The behavior determining unit 246 may also determine a behavior to output educational content according to the age of the user. Here, the educational content is text, video, audio, etc. related to subjects such as English, arithmetic, Japanese, science, and social studies. The educational content may also be interactive content in which the user inputs answers to questions. For example, the behavior determining unit 246 determines a behavior to display on the display device 2521 text of calculation problems corresponding to the grade according to the age of the user. For example, if the age of the user is "under 8 years old," the behavior determining unit 246 determines to display addition problems, and if the age of the user is "8 years old or older," the behavior determining unit 246 determines to display multiplication problems.

[0086] The behavior determining unit 246 may also determine the behavior of causing the output device to output content that is appropriate for the age of the user, rather than a character. In this case, the content may be content featuring a character, or content that does not depend on a character, such as a commonly known folk tale or fairy tale.

[0087] In addition, content corresponding to the character and grade and educational content corresponding to the age of the user may be stored in advance in the storage unit 230, or may be obtained from an external device such as a server that is communicatively connected to the control device 200.

[0088] 4 is a diagram showing an example of an operation flow for character setting. Note that "S" in the operation flow indicates a step to be executed.

[0089] In step S50, the control device 200 accepts the setting of the character. Then, in step S51, the control device 200 outputs a screen corresponding to the character of the avatar 100 (for example, a screen displaying the appearance of the character).

[0090] In step S52, the behavior determining unit 246 acquires the usage age corresponding to the set character from the character data 233.

[0091] 5 is a diagram illustrating an example of an operation flow relating to an operation for determining the behavior of the avatar 100. The operation flow illustrated in FIG. 5 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 220 has been input. Note that "S" in the operation flow indicates the step that is executed.

[0092] First, in step S100, the state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220. When the state recognition unit 240 recognizes the user 10, it reads out information about the user 10.

[0093] In step S101 , the emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0094] In step S102, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user 10 to the history data 232.

[0095] In step S103 , the behavior recognition unit 244 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0096] In step S104, the behavior determination unit 246 determines the behavior of the avatar 100 based on a combination of the current emotional value of the user 10 determined in step S101 and the past emotional values ​​included in the history data 232, the emotional value of the avatar 100, the behavior of the user 10 recognized by the behavior recognition unit 244, and the reaction rules 231.

[0097] For example, if the user 10 says something like "I'm nervous," and the user 10 is laughing, the "joy" emotion value of the avatar 100 is increased, and the behavior decision unit 246 decides the behavior of the avatar 100 so that the avatar 100 will say something like "You're performing well."

[0098] In step S105 , the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 246 .

[0099] In step S106, the memory control unit 248 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0100] In step S107, the storage control unit 248 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S108.

[0101] In step S108, the memory control unit 248 stores the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 240 in the history data 232.

[0102] The control device 200 recognizes the artist's behavior and determines the avatar's behavior corresponding to the recognized artist's behavior. The control device 200 then controls the avatar based on the determined avatar behavior. In this way, the control device 200 can cause the avatar 100 to perform an appropriate behavior in relation to the artist. Therefore, for example, the control device 200 can improve the performance of the user 10.

[0103] The control device 200 determines the creativity-related behavior of the user 10. In this way, the control device 200 can bring out inspirational creativity in the user 10. For example, the control device 200 can support the creation of a work by the user 10. Furthermore, for example, the control device 200 can provide advice to improve the management of a project in which the user 10 is involved.

[0104] The control device 200 also determines behavior related to the expressiveness of the user 10. As a result, the control device 200 can improve the self-expression of the user 10. The control device 200 can also improve the emotion management of the user 10. For example, the control device 200 can relieve tension in the user 10. The control device 200 can also improve communication techniques with audiences.

[0105] As described above, the control device 200 determines an emotion value indicating the emotion of the avatar 100 based on the user state, and determines whether to store data including the behavior of the user 10 in the history data 232 based on the emotion value of the avatar 100. This reduces the capacity of the history data 232 that stores data including the behavior of the user 10. For example, when the control device 200 determines that the user state 10 years from now will be the same as that 10 years ago, the control device 200 reads the history data 232 from 10 years ago, and can present to the user 10 the state of the user 10 from 10 years ago (e.g., the user's facial expression, emotions, etc.), as well as all peripheral information such as data on the sound, image, smell, etc. of the location, via the avatar 100, the display device 2521, the speaker 2522, etc.

[0106] Furthermore, the control device 200 can cause the avatar 100 to perform an appropriate action in response to the user 10's action. Conventionally, the user's actions are classified and actions, including the avatar's facial expressions and appearance, are determined. In contrast, the control device 200 determines the user 10's current emotional value and performs an action on the user 10 based on the past emotional value and the current emotional value. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the control device 200 can cause the avatar 100 to say, "You were cheerful yesterday, but what's wrong with you today?" The control device 200 can also cause the avatar 100 to speak with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the control device 200 can cause the avatar 100 to say, "You were depressed yesterday, but you seem to be in good spirits today, don't you?" Furthermore, for example, if user 10 was in good spirits yesterday and is in better spirits today than yesterday, control device 200 can cause avatar 100 to say something like, "You're in better spirits today than yesterday. Has anything better happened to you than yesterday?" Furthermore, for example, for user 10 whose emotion value is equal to or greater than 0 and whose emotion value continues to fluctuate within a certain range, control device 200 can cause avatar 100 to say something like, "You've been feeling stable lately, which is good."

[0107] Furthermore, for example, when the avatar 100 asks the user 10, "Did you finish the homework I told you about yesterday?" and the user 10 replies, "Yes, I did it," the control device 200 can cause the avatar 100 to utter a positive utterance such as "Great!" and make a positive gesture such as clapping or a thumbs-up. Furthermore, for example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the control device 200 can cause the avatar 100 to utter a positive utterance such as "Good job!" and make the above-mentioned positive gesture. In this way, the avatar 100 controlled by the control device 200 performs an action based on the state history of the user 10, and it is expected that the user 10 will feel a sense of affinity with the avatar 100.

[0108] Furthermore, for example, the control system 1 can support creative people by controlling the behavior of the avatar 100 based on the user's emotions to bring out the inspiration and creativity of the user 10 and assist them in creating works. The control system 1 can also provide advice on project management for the user 10 and on improving the user's 10 self-expression.

[0109] Furthermore, for example, if the avatar 100 is an artist, the control system 1 controls the behavior of the avatar 100 to draw out the inspiration and creativity of the user 10 and support the creation of a work. Specifically, the control system 1 reproduces, by the avatar 100, a painter who paints in a style similar to that of the user 10. For example, the control system 1 can reproduce, as the avatar 100, a famous painter, an ordinary person, an art teacher, or the like, and support the user 10 in drawing out the inspiration of the user 10 so that the user 10 can paint a painting in each of the respective styles.

[0110] In the above embodiment, the control device 200 recognizes the user 10 using a facial image of the user 10. However, the disclosed technology is not limited to this. For example, the control device 200 may recognize the user 10 using a voice uttered by the user 10, the email address of the user 10, the SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

[0111] The control device 200 is an example of an information processing device that controls the avatar 100. The target of behavior control by the control device 200 is not limited to the avatar 100, but can be applied to various electronic devices, etc. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.

[0112] 6 schematically illustrates an example of the hardware configuration of a computer 1200 that functions as the control device 200 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to perform operations associated with the apparatus according to the present embodiment or one or more "parts," and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0113] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0114] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0115] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0116] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0117] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0118] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0119] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0120] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0121] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0122] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0123] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0124] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0125] Computer-readable instructions may be provided locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0126] (Other embodiments) The above-mentioned control device 200 may be mounted on a stuffed toy, a robot, etc., or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on a stuffed toy, a robot, etc.

[0127] The emotion determining unit 242 may determine the emotion of the user according to a specific mapping. Specifically, the emotion determining unit 242 may determine the emotion of the user according to an emotion map (see FIG. 9 ), which is a specific mapping.

[0128] FIG. 7 is a diagram showing an emotion map 2000 on which multiple emotions are mapped. In the emotion map 2000, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are arranged in more primitive states. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental state. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 2000, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0129] (1) For example, if the emotion engine, which is the emotion determination unit 242 that controls the emotions of the avatar 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the avatar 100 (e.g., a backchannel response) may be set at a timing at least as frequent as the emotion engine's detection frequency (100 msec), or may be set at a timing earlier than this. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0130] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., a back-channel response), unnatural back-channel responses are eliminated, enabling a natural, well-read dialogue. Avatar 100 performs a reaction (e.g., a back-channel response) according to the directionality and degree (strength) of the mandala in emotion map 2000. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., playing sports), the user's age, etc.

[0131] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 2000, and the movement of the interjections and the strength of the interjections may be set. For example, if the avatar 100 feels a sense of stability, security, etc., the avatar 100 may nod and continue listening. If the avatar 100 feels anxious, confused, or suspicious, the avatar 100 may tilt its head or stop shaking its head.

[0132] These emotions are distributed in the three o'clock direction on emotion map 2000, and usually fluctuate between relief and anxiety. In the right half of emotion map 2000, situational awareness takes precedence over internal sensations, resulting in a calm impression.

[0133] (3) If the avatar 100 feels good after receiving a compliment, the filler "ah" may be inserted before the line, and if the avatar 100 feels pain after receiving harsh words, the filler "ugh!" may be inserted before the line. A physical reaction, such as the avatar 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around 9 o'clock on the emotion map 2000.

[0134] (4) In the left half of the emotion map 2000, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0135] If the avatar 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the avatar 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the avatar 100 may generate a balanced positive feeling toward the other person, i.e., behavior such as tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 2000.

[0136] Conversely, even when avatar 100 is aware of an internal sensation (reaction) of discomfort, avatar 100 may shake its head when feeling disgust, or glare at the other person when feeling hatred. These emotions are distributed around 6 o'clock on emotion map 2000.

[0137] (5) The inside of emotion map 2000 represents what is going on in the mind, and the outside of emotion map 2000 represents behavior, so the further outside emotion map 2000 you go, the more visible the emotion becomes (is expressed in behavior).

[0138] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on emotion map 2000, avatar 100 may lightly nod its head and say "hmm," but when listening to someone while feeling love, which is distributed around 12 o'clock, avatar 100 may nod its head strongly.

[0139] The emotion determination unit 242 inputs the information analyzed by the sensor module unit 220 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 2000, and determines the emotion of the user 10. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 220, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 2000. FIG. 8 is a diagram showing another example of an emotion map. This neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 2100 shown in FIG. 8. FIG. 8 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0140] Furthermore, the emotion determination unit 242 may determine the emotion of the avatar 100 according to a specific mapping. Specifically, the emotion determination unit 242 inputs the information analyzed by the sensor module unit 220, the state of the user 10 recognized by the state recognition unit 240, and the state of the avatar 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 2000, and determines the emotion of the avatar 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 220, the recognized state of the user 10, the state of the avatar 100, and emotion values ​​indicating each emotion shown in the emotion map 2000. For example, the neural network is trained based on training data indicating that when it is recognized from the output of the touch sensor that the avatar 100 is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when it is recognized from the output of the acceleration sensor that the avatar 100 is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 2100 shown in FIG.

[0141] The emotion determination unit 242 may also determine the emotion of the avatar 100 based on the behavior of the avatar 100 generated by the sentence generation model. Specifically, the emotion determination unit 242 inputs the behavior of the avatar 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 2000, and integrates the obtained emotion values ​​with emotion values ​​indicating the current emotion of the avatar 100 to update the emotion of the avatar 100. For example, the emotion determination unit 242 averages and integrates the emotion values ​​indicating each obtained emotion with emotion values ​​indicating the current emotion of the avatar 100. This neural network is pre-trained based on multiple pieces of training data that are combinations of text indicating the behavior of the avatar 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 2000.

[0142] For example, if the utterance of avatar 100, "That's great. You're lucky," is obtained as the behavior of avatar 100 generated by the sentence generation model, when the text representing this utterance is input into the neural network, a high emotional value is obtained for the emotion "happy," and the emotion of avatar 100 is updated so that the emotional value of the emotion "happy" becomes higher.

[0143] The behavior determination unit 246 generates the behavior content of the avatar by adding fixed sentences to ask about the avatar's behavior content corresponding to the user's behavior to text representing the user's behavior, the user's emotions, and the avatar's emotions, and inputting the text into a sentence generation model with an interactive function.

[0144] For example, the behavior determining unit 246 obtains text representing the state of the avatar 100 from the emotion of the avatar 100 determined by the emotion determining unit 242, using an emotion table such as that shown in Fig. 9. Fig. 9 is a diagram showing an example of an emotion table. In the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the avatar 100 is stored for each index number.

[0145] If the emotion of the avatar 100 determined by the emotion determination unit 242 corresponds to index number "2," the text "very happy" is obtained. Note that if the emotions of the avatar 100 correspond to multiple index numbers, multiple pieces of text representing the state of the avatar 100 are obtained.

[0146] An emotion table such as that shown in FIG. 10 is also prepared for the emotions of the user 10. FIG. 10 is a diagram showing an example of an emotion table. Here, if the user's behavior is speaking "This is AAA," the emotion of the avatar 100 is index number "2," and the emotion of the user 10 is index number "3," then the following is input into the sentence generation model, and the content of the avatar's behavior is obtained: "The avatar is in a very happy state. The user is in a normal happy state. The user spoke to you saying, 'This is AAA.' How would you respond as the avatar?" The behavior determination unit 246 determines the avatar's behavior from this behavior content.

[0147] In this way, avatar 100 can change its behavior according to the index number corresponding to the avatar's emotion, so the user gets the impression that avatar 100 has a heart, and is encouraged to take actions such as talking to the avatar.

[0148] Furthermore, the behavior determination unit 246 may generate the avatar's behavior by adding not only text representing the user's behavior, the user's emotions, and the avatar's emotions, but also text representing the contents of the history data 232, adding a fixed sentence for asking about the avatar's behavior corresponding to the user's behavior, and inputting the resulting text into the dialogue function. This allows the avatar 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the avatar has individuality and encouraging the user to take actions such as talking to the avatar. The history data may also include the avatar's emotions and behavior.

[0149] Second Embodiment Next, a second embodiment will be described. Here, differences from the first embodiment will be mainly described, and a description of the same configuration and processing as the first embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0150] Each control device has a learning support function that detects the learning status of the user 10 in various learning fields via each avatar and outputs information according to the detected learning status. For example, each control device can detect the user's 10 level of understanding of a specific learning field and output learning guidance, a learning plan, and learning resources (reference books, etc.) according to the level of understanding via the avatar, thereby outputting appropriate information to the user 10 according to the learning status of the user 10.

[0151] 11 is a diagram illustrating a schematic functional configuration of a control device 200 that controls the avatar 100. The control device 200 includes a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and a support unit 290.

[0152] The support unit 290 realizes the above-mentioned learning support function. Details of the support unit 290 will be described later.

[0153] Fig. 12 shows an example of an operational flow for determining the behavior of the avatar 100. The operational flow shown in Fig. 12 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 220 has been input.

[0154] First, in step S200 , the state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220 .

[0155] In step S201 , the emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0156] In step S202, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user 10 to the history data 232.

[0157] In step S203 , the behavior recognition unit 244 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0158] In step S204, the behavior determination unit 246 determines the behavior of the avatar 100 based on the usage age acquired in step S52 of FIG. 4, the combination of the current emotional value of the user 10 determined in step S201 of FIG. 12 and the past emotional value included in the history data 232, the emotional value of the avatar 100, the behavior of the user 10 recognized by the behavior recognition unit 244, and the reaction rules 231.

[0159] In step S205 , the behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior determined by the behavior determination unit 246 .

[0160] In step S206, the memory control unit 248 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0161] In step S207, the storage control unit 248 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S208.

[0162] In step S208, the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 240 are stored in the history data 232.

[0163] (Support Unit 290) The support unit 290 will be described in detail. Fig. 13 shows a schematic functional configuration of the support unit 290. Here, the support unit 290 is provided in the control device 200 and causes information related to the avatar 100 to be output.

[0164] As shown in FIG. 13 , the support unit 290 includes a detection unit 2901 and an output control unit 2902 .

[0165] Each component of the support unit 290 is realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by operating an operating system (OS) and a program running on the OS.

[0166] The detection unit 2901 detects the learning status of the user 10 in various learning fields. The output control unit 2903 controls the control device 200 equipped with the sentence generation model to output information corresponding to the learning status detected by the detection unit 2901 to the user 10 via the avatar 100.

[0167] For example, the detection unit 2901 identifies a learning field according to the age of the user 10 and detects the learning situation in the identified learning field. For example, if the user 10 is in the third grade of elementary school, the detection unit 2901 detects the user's proficiency in the learning field for the third grade of elementary school or before the third grade of elementary school. For example, the detection unit 2901 detects daily homework and test scores through image analysis. The detection unit 2901 also detects information such as the user's facial expression and tone of voice when the output control unit 2902 outputs information, and estimates the user's level of understanding of the output information based on the detected information. The detection unit 2901 also estimates the user's problem-solving ability in arithmetic (mathematics), logical thinking ability, and reading comprehension and writing ability in Japanese based on the detected learning situation.

[0168] The output control unit 2902 then controls the avatar 100 to output information corresponding to the learning situation detected by the detection unit 2901. For example, the output control unit 2902 changes the explanation and instruction method in the learning field depending on the detected proficiency level. The output control unit 2902 also controls the output of information corresponding to the level of understanding estimated by the detection unit 2901 and changes in the user 10's emotions. This allows follow-up on areas where understanding is lacking while taking into consideration not to damage the user 10's self-esteem (i.e., to prevent the user 10 from developing negative emotions due to a high level of understanding). Furthermore, if the user's problem-solving ability or logical thinking ability in arithmetic (mathematics) is low (e.g., if the score is below a threshold), the output control unit 2902 controls the output of information to improve the problem-solving ability or logical thinking ability. Furthermore, if the user's reading comprehension or writing ability in Japanese is low, the output control unit 2902 controls the output of information to improve the reading comprehension or writing ability. For example, the output control unit 2902 controls the output of information that improves the expressiveness and communication skills of the user 10 according to the user's understanding of sentence structure and vocabulary. The output control unit 2902 also controls the output of information that matches the learning style and understanding of each user 10. This allows the user 10 to study arithmetic and Japanese in a fun way while maintaining their motivation.

[0169] The detection unit 2901 also detects the user 10's interest in an unknown learning field, for example, through conversation with the user 10. The output control unit 2902 then controls the output of information related to the detected learning field of interest via the avatar 100. The detection unit 2901 also detects the user 10's level of understanding of the information output by the output control unit 2902 and changes in the user 10's emotions. The output control unit 2902 controls the output of information corresponding to the detected level of understanding and changes in emotions. This makes it possible to manage the motivation of the user 10 studying a learning field that requires a great deal of time and effort, such as data science, and to provide appropriate learning resources. For example, learning resources include methods for deepening understanding of the basic principles of data analysis and machine learning, and information on learning through practical projects.

[0170] The detection unit 2901 also detects the learning status of learning fields related to sensibility, such as art and music, which are targeted at a wide range of ages (generally targeting all ages from children to adults). The output control unit 2902 then controls the output of information on learning fields related to sensibility via the avatar 100 in accordance with the learning status detected by the detection unit 2901. The output control unit 2902 also outputs information according to the interests and goals of each user 10 in that learning field. This can enhance a user's passion for art and music. The output control unit 2902 can also contribute to maintaining the user's motivation by outputting information using an emotion engine.

[0171] The detection unit 2901 also detects the learning status of the user 10 taking the test in the test area. The output control unit 2902 then controls the avatar 100 to output information corresponding to the detected learning status. The user 10 taking the test is, for example, a student taking an entrance exam, or a user 10 taking a school test or mock exam. For example, the output control unit 2902 outputs information about the university entrance exam (school information, entrance exam information, and past exam question collections) and information to assist in the development of a study plan leading up to the exam to a student taking a university entrance exam. This can contribute to improving the efficiency of exam preparation and the pass rate. The output control unit 2902 can also contribute to maintaining the motivation of students by outputting information using an emotion engine.

[0172] The output control unit 2902 may also change the design of the avatar 100 according to the proficiency level detected by the detection unit 2901. For example, the output control unit 2902 may increase the strength of the character that is the avatar 100 as the proficiency level increases. For example, the output control unit 2902 may change the monster character from a weak monster (a monster that is likely to be encountered in the game field, such as a slime) to a strong monster character (a boss monster) as the proficiency level increases. Furthermore, in the case of an avatar 100 representing the user 10 (e.g., a main character character), the output control unit 2902 may change the facial expression of the avatar 100 according to the proficiency level. For example, the output control unit 2902 may change the facial expression from a negative expression (a stern expression or a listless expression) to a positive expression (a bold expression or a lively expression) as the proficiency level increases.

[0173] 14 shows an example of an operation flow performed by the support unit 290. First, the support unit 290 detects the user's learning situation in the learning field (step S300). Then, the support unit 290 controls the control device 200 equipped with a sentence generation model to output information corresponding to the detected learning situation to the user 10 via the avatar 100 (step S301), and the process ends.

[0174] Third Embodiment Next, a third embodiment will be described. Here, differences from the above-described embodiment will be mainly described, and a description of the same configuration and processing as the above-described embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0175] Each control device has an output function for outputting information according to the user's social situation or the occurrence of an event related to the user via each avatar. For example, each control device can output appropriate information according to the user's situation or the occurrence of an event related to the user by outputting information about emotional support to a businessperson having a labor problem via the avatar controlled by the control device.

[0176] As shown in FIG. 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control object 252, a communication processing unit 280, and an output unit 291.

[0177] The character data 233 is data that associates a character with an age and a setting condition. For example, a character may be a person who appears in existing content such as animation, video games, manga, or movies. A character may also be an animal or plant with a personality, or an inanimate object (such as a robot). A character may also be a person with characteristics specific to a profession, such as a counselor, doctor, nurse, teacher, lawyer, fortune teller, or insurance salesperson. A character may also be modeled after a real (or former) celebrity.

[0178] 15 is a diagram showing a schematic data structure of the character data 233. The setting conditions associated with the characters in the character data 233 are conditions for automatically setting the character. When the conditions are met, the corresponding character may be automatically set as the avatar. Details of the automatic setting of characters will be described later. Note that the character may also be determined by the input prompt, regardless of the setting conditions.

[0179] The output unit 291 realizes the above-mentioned output function. Details of the output unit 291 will be described later.

[0180] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0181] (Output of Information) The output unit 291 will be described in detail. Fig. 16 is a diagram schematically illustrating the functional configuration of the output unit 291. Here, the output unit 291 is provided in the control device 200 and outputs information related to the avatar 100.

[0182] 16, the output unit 291 includes a detection unit 2911, a setting unit 2912, and an output control unit 2913. The output unit 291 also stores user information 2921 and handling information 2922.

[0183] Each component of the output unit 291 is realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs running on the OS. The user information 2921 is implemented using a storage medium such as a memory.

[0184] The detection unit 2911 detects that the user 10 is in a specific social situation or that an event related to the user has occurred, based on user information about the user 10 stored in the user information 2921. The setting unit 2912 automatically sets the avatar character according to the detection result. The output control unit 2913 then controls the control device 200 equipped with the sentence generation model to output information corresponding to the situation or event detected by the detection unit 2911 to the user 10 via the avatar 100.

[0185] For example, the detection unit 2911 detects a specific situation in the work environment of the user 10. Then, the output control unit 2913 controls the output of information about the work environment to the user 10 via the avatar 100 according to the situation. As a specific example, the output control unit 2913 controls the output of information about work issues and emotional support to a business person (user 10) who is experiencing work-related issues via the avatar 100. As an even more specific example, the output control unit 2913 controls the output of information about appropriate ways to deal with worries and stress in the workplace and about self-care methods via the avatar 100.

[0186] Furthermore, for example, the output control unit 2913 controls the output of information about systems available in the working environment of the user 10 to the user 10 via the avatar 100, depending on the situation in the working environment of the user 10. To give a specific example, the output control unit 2913 controls the output of information about labor laws, information about appropriate procedures among systems available to the user 10, and the like via the avatar 100.

[0187] Furthermore, for example, the detection unit 2911 detects that the user 10 is isolated in society. Then, the output control unit 2913 controls the output of information according to the degree of isolation of the user 10 to the user 10 via the avatar 100. To give a specific example, the output control unit 2913 controls the output of information on methods for providing emotional support and encouraging self-understanding to people who are isolated from society, such as the unemployed, via the avatar 100. Furthermore, the output control unit 2913 controls the output of information on skill development and communication methods for reintegrating into society and becoming independent via the avatar 100, thereby supporting the user 10 in taking a step towards a new life.

[0188] Furthermore, for example, the detection unit 2911 detects that the user 10 belongs to a minority group. Then, the output control unit 2913 controls so that information corresponding to the minority group to which the user 10 belongs is output to the user 10 via the avatar 100. To give a specific example, the output control unit 2913 controls so that information regarding emotional support, information regarding methods for promoting self-understanding, etc. is output to the user 10 who belongs to a minority group via the avatar 100. Furthermore, the output control unit 2913 controls so that information regarding methods for dealing with difficult situations, information regarding suggested methods for communicating with others, etc. is output via the avatar 100, thereby supporting the user 10 in strengthening their position in society.

[0189] (Support for Alleviating Negative Emotions) Here, the detection unit 2911 detects the occurrence of an event related to the user's circumstances that causes the user to feel negative emotions. The output control unit 2913 outputs information corresponding to the detected event. For example, events related to the user's circumstances that cause the user to feel negative emotions include mental illness (posttraumatic stress disorder (PTSD), depression, adjustment disorder, etc.) or addiction, the death of a close relative, etc. Negative emotions are emotions that cause the user's emotional value to become negative, as described below, and include "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness."

[0190] The avatar 100 feels sympathy for the user's negative emotions caused by the occurrence of an event. Specifically, when an event is detected, the emotion determination unit 242 changes the value of an emotion related to sympathy from among multiple types of emotions prepared in advance as emotions for the avatar 100, each of which can have a value set. The output control unit 2913 then performs output according to the emotion value. For example, the emotion related to sympathy is "unbearable" in the emotion map described below.

[0191] For example, the output control unit 2913 outputs text explaining how to deal with the event by speech, thereby enabling the output unit 291 to encourage the user's emotions to change in a positive direction.

[0192] The detection unit 2911 can detect an event from the user's utterance. For example, if the user utters, "I have PTSD," the detection unit 2911 detects that the user suffers from a mental illness (PTSD). For example, if the user utters, "My grandfather passed away the other day," the detection unit 2911 detects that a close relative of the user has died. For example, if the user utters, "I get shaky if I don't drink," the detection unit 2911 detects that the user suffers from an addiction (alcoholism). The detection unit 2911 may also detect an event based on user information 2921. For example, the user information 2921 includes information on the mental illness the user suffers from.

[0193] Furthermore, the output control unit 2913 outputs information relating to countermeasures for the event that has occurred, which is stored in the countermeasure information 2922. This allows the output unit 291 to provide psychological support to the user in response to the event. For example, the countermeasure information 2922 stores information relating to countermeasures in the form of text, associated with the event.

[0194] When the user takes an action (such as speaking) that indicates a negative emotion in response to the occurrence of an event, the emotion determination unit 242 increases the emotion value of an emotion related to sympathy (for example, "sorrow," "unbearable," etc.) (strengthens the emotion) based on that action. The output control unit 2913 controls the output of the avatar 100 according to the emotion value. For example, when the emotion value related to sympathy increases, the output control unit 2913 causes the avatar 100 to lower the tone of voice and speak at a slower speed.

[0195] When the detection unit 2911 detects the occurrence of an event, the setting unit 2912 refers to the character data 233 and sets the character of the avatar 100. The appearance of the avatar 100 displayed on a display device or the like changes according to the setting of the setting unit 2912.

[0196] For example, when the detection unit 2911 detects the occurrence of an event related to the user's circumstances that causes the user to feel negative emotions, the setting unit 2912 sets a character that is expected to alleviate the user's negative emotions. Specifically, when the detection unit 2911 detects the occurrence of an event, the setting unit 2912 can set the avatar 100 to a person wearing a white coat. Furthermore, when the detection unit 2911 detects the occurrence of an event, the setting unit 2912 may set the character of the avatar 100 to a character that resembles a counselor, a doctor, or a nurse.

[0197] Furthermore, the behavior decision unit 246 and the behavior control unit 250 may change the processing content in response to the occurrence of an event being detected by the detection unit 2911. In other words, the behavior control unit 250 causes the avatar 100 to perform an action in response to the event.

[0198] For example, if the detection unit 2911 detects the occurrence of an event related to the user's circumstances that causes the user to have negative emotions, the behavior control unit 250 causes the avatar 100 to take an action that alleviates the user's negative emotions when information is output via the avatar 100. Specifically, the behavior control unit 250 can cause the avatar 100 to take any one of the following actions when information is output via the avatar 100: taking a deep breath, tapping the user on the shoulder, or stroking the user.

[0199] Furthermore, for example, the setting unit 2912 refers to the character data 233 and sets the character "E" if the user has a "mental illness" and is "under 15 years old."

[0200] Below, examples of the operation of the output unit 291 will be described along with examples of events.

[0201] (Event: Mental Illness) For example, if the detection unit 2911 detects, as an event, that the user is suffering from a mental illness, the output control unit 2913 outputs, via the avatar 100, information regarding how to deal with the mental illness.

[0202] At this time, the output control unit 2913 causes the avatar 100 to make speech that explains the user's individual coping strategies, grounding techniques, stress relief methods, and relaxation techniques, thereby helping the user to recover emotionally.

[0203] (Event: Loss) For example, if the detection unit 2911 detects the death of a close relative of the user as an event, the output control unit 2913 outputs, via the avatar 100, information on how to deal with the sense of loss caused by the death of the close relative.

[0204] At this time, the output control unit 2913 increases the emotional value of sympathy, causing the avatar 100 to empathize with the loss or grief, understand the emotions, make speech that explains how to deal with the loss, and encourage the user to consult a specialist at an appropriate time, thereby helping the user recover emotionally.

[0205] (Event: Addiction) For example, if the detection unit 2911 detects, as an event, that the user has an addiction, the output control unit 2913 outputs, via the avatar 100, information on how to deal with the addiction.

[0206] At this time, the output control unit 2913 causes the avatar 100 to make speech that supports the mental aspects of the user recovering from addiction, explains stress relief methods, causes of addiction, challenges in the recovery process, and successful experiences of others who have recovered from addiction, thereby helping the user recover emotionally.

[0207] (Example of Output Based on Reaction Rules) The output control unit 2913 can control the output unit 291 based on the reaction rules 231. The reaction rules 231 store a message to be output when an occurrence of an event related to the user's circumstances that causes the user to feel negative emotions is detected, or when it is detected that the user actually feels negative emotions due to such an event. Note that the message may be stored in the handling information 2922 instead of the reaction rules 231.

[0208] For example, the reaction rule 231 stores messages about how to deal with the situation when an event that causes negative emotions in the user is detected, such as "Get enough sleep," "Get moderate exercise," "It will get better with time," and "Consult a specialist."

[0209] 17 is a diagram illustrating an example of an operation flow by the output unit 291. In step S400, the output unit 291 determines whether or not it is a predetermined timing for performing detection (step S400). If it is not the predetermined timing (step S400; No), the output unit 291 waits until the predetermined timing arrives.

[0210] On the other hand, if it is the predetermined timing (step S400; Yes), the output unit 291 detects that a specific event has occurred regarding the user 10 (step S401). Subsequently, the output unit 291 sets the character of the avatar 100 according to the detection result (step S402).

[0211] The behavior control unit 250 controls the behavior of the avatar 100 according to the detection result (step S403). The output control unit 2913 controls the output of information according to the detection result to the user 10 via the avatar 100 equipped with the sentence generation model (step S404), and the process ends.

[0212] Fourth Embodiment Next, a fourth embodiment will be described. Here, differences from the above-described embodiment will be mainly described, and a description of the same configuration and processing as the above-described embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0213] The appearance of the avatar may be a human figure, such as avatar 100 and avatar 101, or a character, such as avatar 102. For example, avatars 100 and 101 may be avatars dressed as a master of ceremonies or announcer who narrate a memorial video about the bride and groom (users) at a wedding.

[0214] Here, each control device has an output function for outputting information according to the user's social situation via each avatar. For example, each control device can output information about emotional support to a businessperson experiencing a labor problem via the avatar controlled by the control device, thereby outputting appropriate information according to the user's situation.

[0215] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0216] The behavior determining unit 246 determines the behavior of the avatar 100 associated with the reaction rule 231 when the situation information of the user 10 over time, the emotional value of the user 10 corresponding to this situation information, and the emotional value of the avatar 100 satisfy the conditions of the preset reaction rule 231. The situation information of the user 10 over time will be described later.

[0217] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0218] (Information Output) The output unit 291 will be described in detail. Fig. 18 is a diagram schematically showing the functional configuration of the output unit 291. As shown in Fig. 18, the output unit 291 includes a detection unit 2911, a collection unit 2914, and an output control unit 2913. The output unit 291 also stores user information 2921.

[0219] The detection unit 2911 detects the occurrence of a predetermined event. In this embodiment, such events include events that require support for recording and / or managing memorial items such as photographs (images), videos, letters, and diaries that are irreplaceable family memories, and events that require support for managing the family's schedule. The detection unit 2911 detects the user 10 at the user's home or when the user 10 is away from home (for example, outdoors, such as a park, amusement park, beach, or mountain, or indoors, such as a lodging facility like an inn, concert hall, or aquarium). The user 10 includes family members. The family members include the husband, wife, children, and the husband's and / or wife's parents (great-grandfathers). The family members may also include pets. The output control unit 2913 controls the avatar 100, controlled by the control device 200 having a sentence generation model, to output information corresponding to the event detected by the detection unit 2911 to the outside.

[0220] The collection unit 2914 collects situation information indicating the situation of the user 10, including family members. The output control unit 2913 controls the avatar 100 to output information corresponding to the situation information. The collection unit 2914 acquires chronological situation information of the user 10. The chronological situation information of the user 10 is situation information from a certain time in the past to a certain time a predetermined period of time has passed (e.g., from a certain time in the past to the present). The chronological situation information of the user 10 includes at least one of an image and a video of the user 10. For example, the chronological situation information of the user 10 is information regarding the growth and development of a child. The avatar 100 is capable of recognizing the emotions of the user 10 from the image, video, audio, etc. of the user 10.

[0221] The output control unit 2913 controls the avatar 100 to take an action corresponding to the situation information collected by the collection unit 2914. In this case, the output control unit 2913 outputs information corresponding to the situation information collected by the collection unit 2914 to the outside via the avatar 100. The output control unit 2913 controls the avatar 100 to take an action to be recorded for each piece of selected situation information. Through such control, the avatar 100 records, for example, the growth and development of a child. In other words, the avatar 100 provides appropriate support according to the situation of the user 10. This allows the user 10 to share, for example, records of their child's growth and development with their family.

[0222] The output control unit 2913 also controls the avatar 100 to take photos (images) and videos of the user 10. The output control unit 2913 also controls the avatar 100 to edit the photos (images) and videos taken by the avatar 100. As an example of this case, for example, the avatar 100 on the screen of a smartphone or the like takes photos (images) and videos and edits the taken photos (images) and videos. The output control unit 2913 also controls the avatar 100 to save the photos (images) and videos edited by the avatar 100. In this case, for example, the avatar 100 on the screen of a smartphone or the like edits photos (images) and videos and saves the edited photos (images) and videos. Through this control, the avatar 100 creates a family album and preserves the family's irreplaceable memories.

[0223] The chronological status information of the user 10 may also include at least one of the memorial items, such as photographs (images), videos, letters, and diaries, of the user 10. The output control unit 2913 controls the avatar 100 to perform a predetermined action, including at least one of organizing, storing, and backing up the memorial items collected by the collection unit 2914. In this case, for example, the avatar 100 on the screen of a smartphone or the like organizes, stores, and backs up the memorial items. Through such control, the avatar 100 protects the precious memories of the family. In this way, the avatar 100 can provide support that is sensitive to the emotions of the user 10.

[0224] Furthermore, the time-varying status information of the user 10 may include the schedule of the user 10. The output control unit 2913 controls the avatar 100 to set a reminder based on the schedule of the user 10. The output control unit 2913 may also control the avatar 100 to set an alert based on the schedule of the user 10. In this case, the avatar 100 sets at least one of a reminder and an alert. Through such control, the avatar 100 supports the family's schedule management. This enables smooth schedule management for the family. In this case, the avatar 100 sets the reminder and the alert on the screen of a smartphone or the like, for example.

[0225] The output control unit 2913 also controls the avatar 100 to act in accordance with a photograph (image) or video of the user 10, for example, at a wedding or funeral (memorial service). In this case, for example, the avatar 100 is an avatar dressed as an emcee or announcer, and at a wedding, for example, the avatar 100 narrates memorial items (such as a memorial video) related to the bride and groom. Furthermore, at a funeral, for example, the avatar 100 narrates memorial items (such as a memorial video) related to the deceased. When narrating, the avatar 100 should preferably use the voice of a professional emcee or announcer. This type of control allows appropriate support to be provided according to the situation of the user 10.

[0226] 19 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The detection unit 2911 of the output unit 291 determines whether or not the occurrence of a predetermined event has been detected (step S500). If the occurrence of the predetermined event has not been detected (step S500; No), the detection unit 2911 waits until the occurrence of the predetermined event is detected.

[0227] On the other hand, if the occurrence of a predetermined event is detected (step S500; Yes), the collection unit 2914 of the output unit 291 collects situation information indicating the situation of the user 10 (step S501). Subsequently, the output control unit 2913 of the output unit 291 controls the avatar 100 to output an action corresponding to the situation information to the outside (step S502), and the process ends.

[0228] Fifth Embodiment Next, a fifth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0229] The appearance of the avatar may be a human figure, as in avatar 100 and avatar 101, or a stuffed animal, as in avatar 102. For example, avatars 100 and 101 may be disguised as counselors or doctors, and may provide counseling by nodding and listening to the user's concerns just like a real counselor or doctor, making it easier for the user to consult the avatar about their concerns.

[0230] The control device 200 performs assistance based on the health condition of the user 10 or the user 10's family via the avatar 100. The control device 200 acquires health and medical information about the user 10 or the user 10's family. The health and medical information about the user 10 or the user 10's family may be acquired by detection by the control device 200. Alternatively, the health and medical information about the user 10 or the user 10's family may be acquired by detection by another device. For example, the health and medical information about the user 10 or the user 10's family may be acquired from an IoT (Internet of Things) device owned by the user 10 or the user 10's family or a system of a medical institution. For example, the server 300 acquires the health and medical information about the user 10 or the user 10's family from an IoT device owned by the user 10 or the user 10's family or a system of a medical institution. The server 300 stores the acquired health and medical information in association with the identification information of each user 10. For example, the server 300 stores, as health and medical information, whether the user 10 or a family member of the user 10 is pregnant or when the user 10 gave birth, if the user 10 or a family member of the user 10 is ill, the name and condition of the illness, and if the user is hospitalized, the planned length of hospitalization. Furthermore, as health and medical information, for example, if the user 10 or a family member of the user 10 has given birth, the server 300 stores, as health and medical information, that the user 10 or a family member of the user 10 is raising an infant for a predetermined period of time (e.g., one year) from the date of birth. The control device 200 queries the server 300 for health and medical information about the user 10 or a family member of the user 10, thereby receiving (acquiring) the health and medical information about the user 10 or a family member of the user 10 from the server 300.

[0231] 20 is a diagram schematically illustrating the functional configuration of a control device 200 that controls the avatar 100. The control device 200 includes a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, an acquisition unit 292, a memory control unit 248, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0232] The storage unit 230 includes reaction rules 231, history data 232, and assistance rules 234. The assistance rules 234 are rules that define the actions that the avatar 100 will take to assist the user 10 depending on the health condition and medical situation of the user 10 or the user's 10 family. The assistance rules 234 store information about the content of the response and assistance that the avatar 100 will provide to the user 10's actions for each health condition and medical situation of the user 10 and the user's 10 family.

[0233] The acquisition unit 292 acquires health and medical information about at least one of the user 10 and the user 10's family. For example, the acquisition unit 292 acquires health and medical information about the user 10 and the user 10's family from the server 300. For example, the acquisition unit 292 queries the server 300 for identification information about the user 10 and the user 10's family. If the server 300 stores health and medical information corresponding to the identification information, it transmits the health and medical information corresponding to the identification information. The acquisition unit 292 acquires the health and medical information about the user 10 and the user 10's family by receiving it from the server 300. For example, the acquisition unit 292 acquires from the server 300, as information about the family's health and medical information, information about whether the user 10 or the user 10's family is pregnant, the time of birth, if the user 10 or the user 10's family is ill, the name and condition of the illness, and if the user is hospitalized, the expected length of hospitalization.

[0234] The behavior determination unit 246 in this embodiment determines the behavior of the avatar 100 as a behavior corresponding to the behavior of the user 10 based on a combination of the user 10's past emotional value and current emotional value, the emotional value of the avatar 100, the behavior of the user 10, and the reaction rules 231.

[0235] Furthermore, when the acquisition unit 292 acquires health and medical information about at least one of the user 10 and the user's 10 family members, the behavior determination unit 246 provides support according to the emotions of the user 10 based on the acquired health and medical information about at least one of the user 10 and the user's 10 family members. For example, the behavior determination unit 246 reads information about a response corresponding to the health and medical information about the user 10 or the user's 10 family members and the content of support to be provided from the assistance rules 234. The behavior determination unit 246 determines the behavior of the avatar 100 based on the read information.

[0236] The support rules 234 store information regarding responses and support to be provided to the individual and their family for each health condition, medical situation, and issue. For example, the support rules 234 store responses and support for each issue that arises during pregnancy and after childbirth when the individual and their family are pregnant or have given birth. For example, the support rules 234 store responses to emotional issues that arise during pregnancy and methods for dealing with worries during pregnancy. The support rules 234 also store responses to emotional issues that arise after childbirth, methods for dealing with stress after childbirth, and information regarding childcare. For example, the support rules 234 store responses to emotional issues, methods for dealing with stress, and information regarding childcare for each period from childbirth. The support rules 234 also store information regarding support for caring for and caring for the family when the family member is a terminally ill patient. For example, the support rules 234 store care to be provided and methods for caring for the end-of-life care for each issue that arises for the terminally ill patient and their family. The support rules 234 also store information about emotional care and rehabilitation for patients who require hospitalization. For example, the support rules 234 store information about care to be provided and rehabilitation for each issue that arises in patients who require hospitalization. The support rules 234 also store information about emotional care, vital signs, and medication for patients with chronic diseases. For example, the support rules 234 store information about care to be provided, vital signs to pay attention to, and precautions regarding medication for each issue that arises in patients with chronic diseases. The support rules 234 also store answers and support content for each issue that arises during infant care for patients and their families who are raising infants. For example, the support rules 234 store information about infant care activities and information for navigating infant care. For example, the support rules 234 store information about how to change diapers, how to prepare powdered milk, how to soothe a baby, how to set up a child car seat, and how to deal with nighttime crying.

[0237] When the acquisition unit 292 acquires health and medical information about at least one of the user 10 and the user's 10 family members, the behavior determination unit 246 provides support according to the user's emotions based on the health and medical information about the user 10 and at least one of the user's 10 family members. The behavior determination unit 246 according to this embodiment identifies a problem from the content of the user's utterance. The behavior determination unit 246 analyzes text information representing the content of the user's utterance to identify a problem included in the content of the utterance. The behavior determination unit 246 reads, from the assistance rules 234, information regarding a response corresponding to the identified problem and the content of assistance to be provided based on the acquired health and medical status of the user 10 or the user's 10 family members. The behavior determination unit 246 determines the behavior of the avatar 100 based on the read information.

[0238] The behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the avatar 100. For example, if the avatar 100 is verbally abused by the user 10 or is arrogantly treated by the user 10 (i.e., if the user 10 has a poor reaction), if the user 10's voice cannot be detected due to loud ambient noise, or if the remaining battery charge of the avatar 100 is low, and the emotional value of "anger" or "sadness" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 that corresponds to the behavior of the user 10, in accordance with the increase in the emotional value of "anger" or "sadness." Furthermore, if the user 10 has a good reaction or if the remaining battery charge of the avatar 100 is high, and the emotional value of "joy" or "happiness" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 that corresponds to the behavior of the user 10. Furthermore, the behavior determining unit 246 may determine a behavior for the user 10 who has increased the emotional value of "joy" or "happiness" of the avatar 100 that is different from the behavior for the user 10 who has increased the emotional value of "anger" or "sadness" of the avatar 100. In this way, the behavior determining unit 246 may determine a different behavior depending not only on the emotion of the avatar 100 itself and the behavior of the user 10 but also on how the user 10 has changed their emotion.

[0239] The behavior control unit 250 may control the control target 252 to synchronize with the recognized state of the user 10 and to express an emotion corresponding to the estimated emotion of the user 10. For example, in the avatar 100 according to the embodiment, the sensor module unit 220 detects the user 10 using audio, images, or the like with the sensor unit 210, and the state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220. The emotion determination unit 242 then determines the emotional state of the user 10 from the recognized state of the user 10, thereby estimating the emotion of the user 10. The behavior control unit 250 controls the control target 252 to synchronize with the state of the user 10 recognized by the state recognition unit 240 and to express an emotion corresponding to the emotion of the user 10 estimated by the emotion determination unit 242. When the emotion of the user 10 estimated by the emotion determination unit 242 is a positive state, the behavior control unit 250 controls the control target 252 to synchronize with the state of the user 10 recognized by the state recognition unit 240 and to express an emotion corresponding to the estimated emotion of the user 10. For example, the behavior control unit 250 determines a positive state when the emotion values ​​of positive emotions such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment" are high, and the emotion values ​​of negative emotions such as "anger," "sadness," "discomfort," "anxiety," "sadness," "worry," and "emptiness" are low. On the other hand, the behavior control unit 250 determines a negative state when the emotion values ​​of positive emotions such as "joy," "pleasure," "relief," "excitement," "relief," and "fulfillment" are low, and the emotion values ​​of negative emotions such as "anger," "sadness," "discomfort," "anxiety," "sadness," "worry," and "emptiness" are high. When the emotion of the user 10 is positive, the behavior control unit 250 controls the control target 252 to express an emotion corresponding to the estimated emotion of the user 10 in the same state as the user 10. For example, the state recognition unit 240 determines the speech rate of the voice based on the emotion values ​​of the user 10. Furthermore, for example, the state recognition unit 240 detects the user 10 using sound or images from the sensor unit 210, and recognizes any of the characteristics of the user 10, such as gestures, speaking style, or choice of words.The behavior control unit 250 controls the control object 252 to express an emotion corresponding to the estimated emotion of the user 10, using any one of gestures, speaking style, and wording as the same characteristics as the user 10. Furthermore, for example, the state recognition unit 240 recognizes any one of the facial expression, tone of voice, and verbal nuances of the user 10. The behavior control unit 250 controls the control object 252 to express an emotion corresponding to the estimated emotion of the user 10, using any one of the facial expression, tone of voice, and verbal nuances as the same state as the state of the user 10. When the avatar 100 expresses an emotion corresponding to the emotion of the user 10 to the user 10 in this way, the psychological connection between the user 10 and the avatar 100 is improved, and the relationship between them is enhanced.

[0240] Furthermore, when the control device 200 acquires health and medical information about at least one of the user 10 and the user's family member who communicates with the avatar 100, the control device 200 controls the avatar 100 to provide support according to the user's emotions based on the health and medical information about the user 10 and / or the user's family member. This allows the control device 200 to provide support according to the recognized emotions of the user 10. For example, the control device 200 can help pregnant and postpartum parents navigate challenges that arise during pregnancy and after birth via the avatar 100. For example, the control device 200 can suggest ways to deal with worries during pregnancy and postpartum stress via the avatar 100, thereby improving parental confidence. Furthermore, the control device 200 can provide answers to emotional challenges, ways to deal with stress, and information on childcare for each period after birth via the avatar 100, and can also provide support for adjusting to life as a new family. The control device 200 can also provide psychological care through dialogue with terminally ill patients and their families via the avatar 100. Furthermore, the control device 200 can provide appropriate care methods and end-of-life support via the avatar 100, helping to ensure a peaceful final moment. The control device 200 can also provide psychological care through dialogue to patients who require hospitalization via the avatar 100. The control device 200 can also support patients who require hospitalization in preparing for rehabilitation and life after discharge via the avatar 100, thereby creating an environment where patients can concentrate on their recovery with peace of mind. If the user 10 or the user's family is raising an infant, the control device 200 can provide assistance with raising the infant via the avatar 100 by having the avatar 100 show actual movements such as how to change diapers, how to prepare powdered milk, how to soothe the infant, how to set up a child car seat, and how to respond to night crying.

[0241] The behavior control unit 250 may determine the speech rate of the voice based on the emotion value of the avatar 100. For example, when the emotion of the user 10 is positive, the behavior control unit 250 determines the speech rate of the voice based on the emotion value of the avatar 100. For example, the behavior control unit 250 determines a faster speech rate the greater the emotion value of the avatar 100. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 246 based on the emotion value determined by the emotion determination unit 242.

[0242] Fig. 21 is a diagram illustrating an example of an operational flow relating to an operation for determining the behavior of avatar 100. The operational flow illustrated in Fig. 21 is repeatedly executed. At this time, it is assumed that information analyzed by sensor module unit 220 has been input.

[0243] First, in step S600, the state of the user 10 communicating with the avatar 100 is recognized. For example, the state recognition unit 240 recognizes the state of the user 10 communicating with the avatar 100 based on information analyzed by the sensor module unit 220.

[0244] In step S601, the emotion of the user 10 is estimated based on the recognized state of the user 10. For example, the emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240.

[0245] In step S602, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user 10 to the history data 232.

[0246] In step S603 , the behavior recognition unit 244 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240 .

[0247] In step S604, the acquisition unit 292 acquires health and medical information of at least one of the user 10 and the user's 10 family members from the server 300. For example, the acquisition unit 292 queries the server 300 for identification information of the user 10 and the user's 10 family members. If the server 300 stores health and medical information corresponding to the identification information, the server 300 transmits the health and medical information corresponding to the identification information. The acquisition unit 292 acquires the health and medical information of at least one of the user 10 and the user's 10 family members by receiving it from the server 300.

[0248] In step S605, the behavior determination unit 246 determines the behavior of the avatar 100. For example, the behavior determination unit 246 determines the behavior of the avatar 100 based on the combination of the current emotion value and the past emotion value of the user 10 determined in step S601, the emotion value of the avatar 100, the behavior of the user 10 recognized by the behavior recognition unit 244, and the reaction rule 231. Furthermore, if the behavior determination unit 246 acquires health and medical information about at least one of the user 10 and the user's family member in step S602, the behavior determination unit 246 provides assistance according to the emotion of the user 10 based on the acquired health and medical information about at least one of the user 10 and the user's family member. For example, the behavior determination unit 246 reads, from the assistance rule 234, information about a response corresponding to the health and medical information about the user 10 or the user's family member, and information about the content of assistance to be provided. The behavior determination unit 246 determines the behavior of the avatar 100 based on the read information.

[0249] In step S606, the behavior control unit 250 controls the avatar 100 and the control object 252 based on the behavior determined by the behavior determination unit 246. At this time, if the emotion of the user 10 estimated by the emotion determination unit 242 is in a positive state, the behavior control unit 250 controls the control object 252 to synchronize with the state of the user 10 recognized by the state recognition unit 240 and to express an emotion corresponding to the estimated emotion of the user 10. For example, if the emotion of the user 10 is in a positive state, the behavior control unit 250 controls the control object 252 to express an emotion corresponding to the estimated emotion of the user 10 in the same state as the state of the user 10. Furthermore, if the emotion of the user 10 is in a positive state, the behavior control unit 250 determines the speaking rate of the voice based on the emotion value of the avatar 100.

[0250] In step S607, the memory control unit 248 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0251] In step S608, the storage control unit 248 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S609.

[0252] In step S609, the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 240 are stored in the history data 232.

[0253] As described above, the control device 200 includes an estimation unit (e.g., the sensor unit 210, the sensor module unit 220, the state recognition unit 240, and the emotion determination unit 242), an acquisition unit 292, and an avatar control unit (e.g., the action determination unit 246 and the action control unit 250). The estimation unit estimates the emotion of the user 10 communicating with the avatar 100. The acquisition unit 292 acquires health and medical information of at least one of the user 10 and the user's family. The avatar control unit controls the avatar 100 to provide assistance according to the emotion of the user 10 estimated by the estimation unit, based on the health and medical information of at least one of the user 10 and the user's family acquired by the acquisition unit 292. In this way, the control device 200 can assist the user 10 via the avatar 100 according to the health and medical conditions of the user 10 and the user's family, and can support the user 10.

[0254] Furthermore, when the user 10 or a family member of the user 10 is pregnant or has just given birth, the avatar control unit controls the avatar 100 to provide information for navigating the emotional issues of the user 10 estimated by the estimation unit. This allows the control device 200 to assist and support the user 10 via the avatar 100 when the user 10 or a family member of the user 10 is pregnant or has just given birth.

[0255] Furthermore, when a family member of user 10 is a terminally ill patient, the avatar control unit controls avatar 100 to provide information related to support for caring for and caring for the family member of user 10 in accordance with the emotion of user 10 estimated by the estimation unit. This allows control device 200 to assist and support user 10 via avatar 100 when a family member of user 10 is a terminally ill patient.

[0256] Furthermore, the acquisition unit 292 acquires health and medical information about the user 10. If the user 10 is a patient who requires hospitalization, the avatar control unit controls the avatar 100 to provide information about at least one of emotional care and rehabilitation. This allows the control device 200 to assist and support the user 10 via the avatar 100 if the user 10 is a patient who requires hospitalization.

[0257] Furthermore, the acquisition unit 292 acquires health and medical information about the user 10. If the user 10 is a chronic disease patient, the avatar control unit controls the avatar 100 to provide information about at least one of emotional care, vital signs, and medication. This allows the control device 200 to assist and support the user 10 via the avatar 100 if the user 10 is a chronic disease patient.

[0258] Furthermore, if user 10 or a family member of user 10 is raising an infant, the avatar control unit controls avatar 100 so that the avatar shows actual movements of changing diapers, preparing powdered milk, soothing the infant, setting up a child car seat, or dealing with night crying. In this way, if user 10 or a family member of user 10 is raising an infant, control device 200 can support user 10 in raising the infant via avatar 100.

[0259] Sixth Embodiment Next, a sixth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and descriptions of configurations and processes similar to those of the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0260] Here, each control device has an output function for outputting information according to the user's social situation via each avatar. For example, each control device can output information about emotional support to a businessperson experiencing a labor problem via the avatar controlled by the control device, thereby outputting appropriate information according to the user's situation.

[0261] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0262] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0263] (Information Output) The output unit 291 will be described in detail. Fig. 22 is a diagram schematically illustrating the functional configuration of the output unit 291. As shown in Fig. 22, the output unit 291 has an estimation unit 2915 and an output control unit 2913. In addition, the output unit 291 stores user information 2921.

[0264] The estimation unit 2915 estimates the social situation of the user 10 based on the user information about the user 10 stored in the user information 2921. Then, the output control unit 2913 controls the control device 200 equipped with the sentence generation model to output information according to the situation estimated by the estimation unit 2915 to the user 10 via the avatar 100.

[0265] For example, the estimation unit 2915 estimates the situation in the working environment of the user 10. Then, the output control unit 2913 controls the output of information about the working environment to the user 10 via the avatar 100 according to the situation.

[0266] Furthermore, for example, the estimation unit 2915 estimates the degree of isolation of the user 10 in society. Then, the output control unit 2913 controls so that information according to the degree of isolation of the user 10 is output to the user 10 via the avatar 100. To give a specific example, the output control unit 2913 controls so that information on methods for providing emotional support and encouraging self-understanding is output to people who are isolated from society, such as the unemployed, via the avatar 100. Furthermore, the output control unit 2913 controls so that information on skill development and communication methods for aiming for social reintegration and independence is output via the avatar 100, thereby supporting the user 10 in taking a step towards a new life.

[0267] Furthermore, for example, the estimation unit 2915 estimates the minority group to which the user 10 belongs. Then, the output control unit 2913 controls so that information corresponding to the minority group to which the user 10 belongs is output to the user 10 via the avatar 100. To give a specific example, the output control unit 2913 controls so that information regarding emotional support, information regarding methods for promoting self-understanding, etc. is output to the user 10 who belongs to a minority group via the avatar 100. Furthermore, the output control unit 2913 controls so that information regarding methods for dealing with difficult situations, information regarding suggestions for communication methods with others, etc. is output via the avatar 100, thereby supporting the user 10 in strengthening their position in society.

[0268] 23 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The output unit 291 determines whether or not it is a predetermined timing for estimating the social situation of the user 10 (step S700). If it is not the predetermined timing (step S700; No), the output unit 291 waits until the predetermined timing arrives.

[0269] On the other hand, if it is the predetermined timing (step S700; Yes), the output unit 291 estimates the social situation of the user 10 based on the user information about the user 10 (step S701). Next, the output unit 291 controls so that information according to the estimated situation is output to the user 10 via the avatar 100 equipped with the sentence generation model (step S702), and the process ends.

[0270] Seventh Embodiment Next, a seventh embodiment will be described. Here, differences from the above-described embodiment will be mainly described, and a description of the same configuration and processing as the above-described embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0271] The appearance of the avatar may be a human figure, as in avatar 100 and avatar 101, or a character, as in avatar 102. For example, avatars 100 and 101 may appear as real museum curators, act in response to the user's emotions, and guide the user around the museum.

[0272] Here, each control device 200 has an output function that causes each avatar 100 to output an action corresponding to situation information indicating the user's situation. For example, each control device 200 can recognize the emotions of the user 10 while appreciating an art piece and cause the avatar 100 to output a response (guidance) corresponding to the recognized emotion, thereby allowing the avatar 100 to respond like a human. Also, for example, each control device 200 can recognize the emotions of the user 10 while playing a game and cause the avatar 100 to output a response (game advice) corresponding to the recognized emotion, thereby allowing the avatar 100 to respond like a human. Also, for example, each control device 200 can recognize the emotions of the user 10 who is a traveler and cause the avatar 100 to output a response (travel support) corresponding to the recognized emotion, thereby allowing the avatar 100 to respond like a human.

[0273] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0274] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0275] (Detection of Event) The output unit 291 will be described in detail. Fig. 24 is a diagram schematically illustrating the functional configuration of the output unit 291. As shown in Fig. 24, the output unit 291 includes a detection unit 2911, a collection unit 2914, and an output control unit 2913. The output unit 291 also stores handling information 2922.

[0276] The detection unit 2911 detects the occurrence of a predetermined event. The detection unit 2911 detects a user appreciating artworks in a cultural facility such as an art museum or gallery. The detection unit 2911 also detects a user playing a game using various game devices or various board games. The detection unit 2911 also detects an event, for example, a foreign tourist speaking to the avatar 100. The output control unit 2913 then controls the avatar 100, which is controlled by the control device 200 equipped with a sentence generation model, to take an action corresponding to the event detected by the detection unit 2911.

[0277] The collection unit 2914 also collects situation information indicating the situation of the user 10. The output control unit 2913 then controls the avatar 100 to output information corresponding to the situation information. For example, the collection unit 2914 collects information indicating the artwork that the user 10 is viewing and the situation of the user 10 (such as the voice of the user 10 and an image of the user 10). The collection unit 2914 also collects information indicating the situation of the user 10 (such as the voice of the user 10 and an image of the user 10) when an event occurs in which the user 10 is spoken to by a foreign tourist. The collection unit 2914 collects situation information related to the user's emotions. This allows the control device 200 to recognize the emotions of the user 10 from the voice, image, etc. of the user 10.

[0278] The output control unit 2913 controls the avatar 100 to behave in accordance with the situation information collected by the collection unit 2914. For example, the output control unit 2913 reads the user 10's interest and impression from the emotions of the user 10 while viewing an art piece, and controls the response of the avatar 100 to provide an explanation or suggest related works in accordance with the interest. More specifically, the output control unit 2913 controls the response of the avatar 100 to provide an explanation of the art piece that the user 10 is interested in that is in accordance with the user 10's interests, or to suggest related works that the user 10 is presumed to own. In other words, in this case, the output control unit 2913 controls the response of the avatar 100 to behave as a curator personalized for the user 10.

[0279] Furthermore, the output control unit 2913 controls the response of the avatar 100 so as to give advice to the user 10, for example, depending on the emotions of the user 10 while playing the game or the game situation. For example, the output control unit 2913 controls the response of the avatar 100 so as to provide appropriate gameplay or advice depending on the emotions of the user 10. In other words, in this case, the output control unit 2913 controls the response of the avatar so that the user 10 enjoys the game together with the avatar 100.

[0280] Furthermore, the output control unit 2913 controls the responses of the avatar 100 to provide support such as language translation to a foreign tourist who is sightseeing, while being sensitive to the foreign tourist's feelings. Note that the tourist is not limited to a foreigner. For example, the output control unit 2913 controls the responses of the avatar 100 to alleviate the tourist's anxieties and resolve problems. To cite a more specific example, the output control unit 2913 controls the responses of the avatar 100 to listen to the foreign tourist's anxieties and problems and to have the avatar 100 speak solutions to those anxieties and problems in the foreign tourist's language. As another example, the output control unit 2913 controls the avatar 100 to support the foreign tourist by translating conversations between the foreign tourist and local people.

[0281] 25 is a diagram illustrating an example of an operation flow by the output unit 291. The output unit 291 determines whether or not the occurrence of a predetermined event has been detected (step S800). If the occurrence of the predetermined event has not been detected (step S800; No), the output unit 291 waits until the occurrence of the predetermined event is detected.

[0282] On the other hand, if the occurrence of a predetermined event is detected (step S800; Yes), the output unit 291 collects situation information indicating the situation of the user 10 (step S801). Next, the output unit 291 controls the avatar 100 controlled by the control device 200 equipped with the sentence generation model to respond to the user 10 in accordance with the situation information (step S802), and ends the process.

[0283] Eighth Embodiment Next, an eighth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and descriptions of configurations and processes similar to those of the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0284] The appearance of the avatar may be a human figure, such as avatar 100 and avatar 101, or a character, such as avatar 102. For example, avatars 100 and 101 may appear as restaurant staff (e.g., bartenders) and converse with the user by outputting responses to the user using actual gestures.

[0285] Here, each control device has an output function that recognizes the state of the user and causes each avatar to output a response according to the recognized state. For example, each control device can recognize the emotion of a user visiting a restaurant or the like and cause the avatar to output a response (such as customer service) according to the recognized emotion, allowing the avatar to provide customer service like a human.

[0286] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0287] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0288] (Response Output) The output unit 291 will be described in detail. Fig. 26 is a diagram schematically illustrating the functional configuration of the output unit 291. As shown in Fig. 26, the output unit 291 has a recognition unit 2916 and an output control unit 2913. The output unit 291 also stores detection information 2923. The detection information 2923 is implemented by a storage medium such as a memory.

[0289] For example, the recognition unit 2916 recognizes the state of the user 10 based on detection information detected from the user 10. The output control unit 2913 then causes the avatar 100, controlled by the control device 200 equipped with a sentence generation model, to output a response to the user 10 in accordance with the state of the user 10 recognized by the recognition unit 2916. To give a specific example, the recognition unit 2916 recognizes the state of the user 10 visiting a restaurant or other establishment based on the detection information. The output control unit 2913 then causes the avatar 100 to output concise conversation and explanations to the user 10 who is in a hurry. On the other hand, the output control unit 2913 causes the avatar 100 to output customer service and explanations, interspersed with a little digression, to the user 10 who wants to talk. The output control unit 2913 can also cause the avatar 100 to output expressive gestures. This allows the avatar 100 to provide human-like customer service in accordance with the topic of conversation with the user 10 and the situation.

[0290] Furthermore, for example, the recognition unit 2916 recognizes the emotion of the user 10 based on the detection information. Then, the output control unit 2913 causes the avatar 100 to output a response to the user 10 according to the user's emotion. To give a specific example, if the emotion of the user 10 is "joy," the output control unit 2913 can cause the avatar 100 to output positive gestures and positive utterances corresponding to that emotion, thereby enabling human-like customer service that matches the emotion of the user 10.

[0291] Furthermore, for example, the output control unit 2913 causes the avatar 100 to output a response to the user 10 in accordance with the history of the detection information accumulated in the detection information 2923 and the state of the user 10. As a result, the avatar 100 can provide customer service by listening to the customer's complaints and providing appropriate responses, as traditionally done by a bartender in a bar or coffee shop, and can also respond based on the content of the past complaint in the case of a continuation of the complaint, based on the history of the detection information indicating the past conversation.

[0292] Furthermore, for example, the recognition unit 2916 recognizes whether or not the user 10 is lost on the route to the destination based on the detection information. Then, if the user 10 is lost on the route to the destination, the output control unit 2913 causes the avatar 100 to output a response related to route guidance. Furthermore, the output control unit 2913 causes the avatar 100 to output a response related to route guidance in accordance with the detection information. Furthermore, the output control unit 2913 causes the avatar 100 to output a response related to route guidance in accordance with the emotion of the user 10 based on the detection information.

[0293] In large hospitals such as university hospitals, the user 10 may get lost on the route to an examination or test. Therefore, the avatar 100 can recognize whether the user 10 is lost from, for example, the facial expression or behavior of the user 10 indicated by the detection information, and guide the lost user 10 to the destination. Furthermore, the avatar 100 can adjust the level of care in the guidance and the speed of movement depending on, for example, the age, medical condition, and emotions of the user 10, and can therefore carefully guide the user 10 to the destination.

[0294] Furthermore, for example, the output control unit 2913 causes the avatar 100 to output a response regarding a proposed transaction object according to the state of the user 10. As a result, the avatar 100 can, for example, recognize the user 10's background, reason for moving, financial situation, preferences for real estate, etc. based on the detected information of the user 10 who is considering moving, and thereby introduce appropriate properties to the user 10. Furthermore, the avatar 100 can, for example, recognize the user 10's emotions such as expectations and anxieties associated with moving, and provide customer service that is sensitive to those emotions (for example, if the user is feeling anxious, using positive gestures and positive speech).

[0295] Furthermore, for example, the output control unit 2913 causes the avatar 100 to output a gesture according to the state of the user 10. This allows the avatar 100 to perform a gesture of reaction that matches the emotion of the user 10, for example, and thus allows the avatar 100 to provide human-like customer service.

[0296] The detected information may be the voice, facial expression, or movement of the user 10 detected by various sensors possessed by the control device 200, or may indicate the situation around the user 10.

[0297] In addition, the output control unit 2913 may cause the avatar 100 to output a response whose content corresponds to the state of the user 10, as described above, or may cause the avatar 100 to output a response to the user 10 at a timing that corresponds to the state of the user 10.

[0298] 27 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The output unit 291 determines whether or not the avatar 100 has detected the user 10 to whom the avatar 100 is to output a response (step S900). If the user 10 has not been detected (step S900; No), the output unit 291 waits until the user 10 is detected.

[0299] On the other hand, if the user 10 is detected (step S900; Yes), the output unit 291 recognizes the state of the user 10 based on the detection information detected from the user 10 (step S901). Next, the output unit 291 causes the avatar 100 to output a response to the user 10 in accordance with the state of the user 10 (step S902), and ends the process.

[0300] Ninth Embodiment Next, a ninth embodiment will be described. Here, differences from the above-described embodiment will be mainly described, and a description of the same configuration and processing as the above-described embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0301] The appearance of the avatar may be a human figure, such as avatar 100 and avatar 101, or a character, such as avatar 102. For example, avatars 100 and 101 may act as audience members and output gestures (e.g., changing facial expressions, smiling, nodding) based on the content of the presentation by user 10 and the emotions of avatars 100 and 101 that change in accordance with the content of the presentation.

[0302] Here, each control device has an output function for evaluating the state of the user 10 and outputting, from each avatar to a predetermined output destination, proposal information regarding proposals for the state of the user 10 in accordance with the evaluation. For example, each avatar can make suggestions in accordance with the evaluation of the state of the user 10, thereby improving the state of the user 10.

[0303] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0304] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0305] (Output of Information) The output unit 291 will be described in detail. Fig. 28 is a diagram schematically showing the functional configuration of the output unit 291. As shown in Fig. 28, the output unit 291 has an evaluation unit 2917 and an output control unit 2913. In addition, the output unit 291 stores user information 2921.

[0306] For example, the evaluation unit 2917 evaluates the state of the user 10 based on the user information about the user 10 stored in the user information 2921. Then, the output control unit 2913 causes the avatar 100 controlled by the control device 200 equipped with the sentence generation model to output proposal information regarding a proposal for the state of the user 10 to a predetermined output destination in accordance with the evaluation by the evaluation unit 2917. To give a specific example, the evaluation unit 2917 evaluates the utterance of the user 10 based on user information regarding the utterance of the user 10. Then, the output control unit 2913 causes the avatar 100 to output proposal information regarding a proposal for the utterance of the user 10 to the user 10 in accordance with the evaluation. To give a more specific example, when practicing, the evaluation unit 2917 analyzes the content of the presentation based on user information regarding the content of the presentation, word choice, the facial expression, gestures, tone of voice, etc. of the user 10, and evaluates the presentation (e.g., evaluates the emotions and reactions of the public or audience). The output control unit 2913 then represents the feelings of the public and the audience and outputs information on how to improve the presentation in a demonstration format (for example, outputting appropriate content, words, facial expressions, gestures, and tone of voice). This allows the user 10 to know the reactions of the public and the audience in advance, thereby preventing the risk of a decline in reputation. It also allows the user 10 to know how to improve their presentation to improve their reputation.

[0307] Furthermore, for example, the evaluation unit 2917 evaluates the status of the user 10 in a specific organization based on user information indicating the context of the user 10 who belongs to the organization. Then, the output control unit 2913 causes the avatar 100 to output, to the user 10, proposal information related to proposals for the status of the user 10 in the organization, in accordance with the evaluation. To give a specific example, the evaluation unit 2917 evaluates the motivation, stress, and other states of each user 10 within the organization based on the background, emotions, needs, and other information of each user 10 who belongs to an organization such as a company, volunteer group, or non-profit organization. Then, the output control unit 2913 causes the avatar 100 to output, to each user 10, information on improving communication within the organization, information on strategic approaches toward achieving the organization's goals, educational programs, support for maintaining motivation, information for relieving stress, and the like, in accordance with the evaluation. This allows the avatar 100 to respond to the emotions and needs of members (users 10) with diverse backgrounds within the organization, promote diversity and inclusion within the organization, and support the promotion of understanding and cooperation among members.

[0308] Further, for example, the evaluation unit 2917 evaluates the state of the user 10 in a specific task based on user information indicating the context of the user 10 performing the task. Then, the output control unit 2913 causes the avatar 100 to output, to the user 10, proposal information regarding proposals for the state of the user 10 in the task, in accordance with the evaluation. To give a specific example, the evaluation unit 2917 evaluates the state of the user 10 in the task, such as motivation and stress, based on the background, emotions, needs, stress, etc., of the user 10 who is an employee of a company (e.g., the user 10 working from home). Then, the output control unit 2913 causes the avatar 100 to output, in accordance with the evaluation, information regarding advice on emotion and stress management, suggestions on how to work efficiently, how to take appropriate breaks, and how to balance work and private life. This allows the avatar 100 to support the user 10 in improving their work style during remote work and reduce their mental burden.

[0309] Furthermore, for example, the evaluation unit 2917 evaluates the user 10's status regarding job change based on user information indicating the user 10's context. Then, the output control unit 2913 causes the avatar 100 to output, to the user 10, proposal information regarding proposals for the user 10's status regarding job change, based on the evaluation. To give a specific example, the evaluation unit 2917 evaluates the user 10's state, such as motivation and stress, based on the background, emotions, needs, stress, etc., of the user 10 who is considering a career change. Then, the output control unit 2913 causes the avatar 100 to output, based on the evaluation, information regarding emotional support such as maintaining motivation and relieving stress. Furthermore, the output control unit 2913 causes the avatar 100 to output, to the user 100, information regarding suggestions such as methods to promote self-understanding, methods to improve skills for a smooth transition to a new career, and networking methods. This allows the avatar 100 to support the user 10 in choosing an appropriate career.

[0310] Further, for example, the evaluation unit 2917 evaluates the status of the user 10 regarding career advancement based on user information indicating the context of the user 10. Then, the output control unit 2913 causes the avatar 100 to output, to the user 10, proposal information regarding proposals for the status of the user 10 regarding career advancement, in accordance with the evaluation. To give a specific example, the evaluation unit 2917 evaluates the user 10's state, such as motivation and stress, based on the background, emotions, needs, stress, etc., of the user 10 who is aiming for career advancement. Then, the output control unit 2913 causes the avatar 100 to output, in accordance with the evaluation, information regarding personalized guidance based on the emotional needs and ambitions of the user 10, information regarding setting goals to be achieved, information regarding progress management, information regarding suggestions for methods of acquiring appropriate skills, and the like. This allows the avatar 100 to provide personalized guidance based on the emotional needs and ambitions of individuals aiming for career advancement.

[0311] Furthermore, for example, the evaluation unit 2917 evaluates the transaction object (e.g., a product or a service) based on user information indicating an evaluation from the user 10 who received the transaction object. Then, the output control unit 2913 causes the avatar 100 to output proposal information regarding a proposal for the transaction object to the provider who provides the transaction object, in accordance with the evaluation. To give a specific example, the evaluation unit 2917 evaluates the user 10's feelings and requests regarding the transaction object based on reviews and feedback from the user 10 regarding the transaction object. Then, the output control unit 2913 causes the avatar 100 to output information regarding improvements to the transaction object to the provider, in accordance with the evaluation. In this way, the avatar 100 can contribute to improving customer satisfaction.

[0312] Furthermore, for example, the evaluation unit 2917 evaluates the administrator who manages the user 10 based on the emotions of the user 10, which are based on the user information. The output control unit 2913 then causes the avatar 100 to output, to the administrator, proposal information regarding suggestions for managing the user 10, in accordance with the evaluation. To give a specific example, the evaluation unit 2917 evaluates the user 10's motivation, stress, emotional state, and other conditions based on the user 10's background, emotions, needs, stress, and other factors. The output control unit 2913 then causes the avatar 100 to output, to the administrator, information regarding responses and advice to the user 10 in accordance with the evaluation. This allows the avatar 100 to support the administrator's leadership and decision-making by outputting appropriate responses and advice in accordance with the emotions, etc., of the administrator's subordinates (users 10). The avatar 100 can also output information regarding emotional coaching to the administrator to support self-management.

[0313] The user information may be the voice, facial expression, or movement of the user 10 detected by various sensors included in the control device 200, or may indicate the situation around the user 10. The user information may also be information acquired from an external server.

[0314] 29 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The output unit 291 determines (step S1000) whether or not it is a predetermined timing to evaluate the state of the user 10. If it is not the predetermined timing (step S1000; No), the output unit 291 waits until the predetermined timing arrives.

[0315] On the other hand, if it is the predetermined timing (step S1000; Yes), the output unit 291 evaluates the state of the user 10 based on the user information about the user 10 (step S1001). Subsequently, the output unit 291 causes the avatar 100 to output, in accordance with the evaluation, proposal information about a proposal for the state of the user 10 to a predetermined output destination (step S1002), and ends the process.

[0316] Tenth Embodiment Next, a tenth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0317] Here, each control device has an output function for outputting information according to the user's social situation or the occurrence of an event related to the user via each avatar. For example, each control device can output appropriate information according to the user's situation or the occurrence of an event related to the user by outputting information about emotional support to a businessperson experiencing a labor problem via the avatar controlled by the control device.

[0318] As shown in FIG. 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control object 252, a communication processing unit 280, and an output unit 291.

[0319] The character data 233 included in the storage unit 230 is data that associates a character with an age and a setting condition. The character may be a person with characteristics of a particular occupation, such as a counselor, doctor, nurse, teacher, lawyer, fortune teller, or insurance salesperson. The character may also be modeled after a real (or former) celebrity. The data structure of the character data 233 is shown schematically in FIG. 15.

[0320] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0321] (Output of Information) The output unit 291 will be described in detail. As shown in Fig. 16 , the output unit 291 includes a detection unit 2911, a setting unit 2912, and an output control unit 2913. The output unit 291 also stores user information 2921 and handling information 2922.

[0322] (Support for Adaptation to Circumstances) Here, the detection unit 2911 detects the occurrence of an event related to the user's circumstances. Furthermore, the output control unit 2913 outputs information corresponding to the detected event. For example, events related to the user's circumstances include the user studying abroad, a change in the user's job (getting a job, changing jobs, or re-employment), the birth of the user's child, and the user's child reaching a certain age.

[0323] The avatar 100 experiences an emotion corresponding to the occurrence of an event. Specifically, when an event is detected, the avatar 100 changes the value of the emotion of relief (for example, "relief" in the emotion map described below) from among multiple types of emotions prepared in advance as the robot's emotions, each of which can be assigned a value. The avatar 100 then performs output according to the value of the emotion.

[0324] This is because, when the avatar 100 detects an event such as the above, it is necessary for the avatar 100 to provide advice regarding the user's circumstances. In order to provide persuasive advice, the avatar 100 itself needs to remain calm. Note that, instead of increasing the emotional value of relief, the avatar 100 may reduce the emotional values ​​of multiple types overall, so that it does not show much emotion and provides advice in a calm and collected manner.

[0325] For example, the avatar 100 may output text explaining how to adapt to the situation by speech, thereby encouraging the user's emotions to change in a positive direction (e.g., relief, relief).

[0326] The detection unit 2911 can detect events from the user's utterances. For example, if the user utters, "I came to Japan to study abroad," the detection unit 2911 detects that the user's circumstances have changed as a result of studying abroad. For example, if the user utters, "I'm thinking of changing jobs to work at an IT company," the detection unit 2911 detects that the user is trying to change jobs. For example, if the user utters, "My son, who loves sports, has started elementary school," the detection unit 2911 detects that the user's child has reached a certain age.

[0327] Furthermore, the output control unit 2913 outputs information for adapting to changed circumstances, which is stored in the handling information 2922, via the avatar 100. In this way, the output unit 291 can support the user in adapting to the circumstances in response to the event.

[0328] The emotion determination unit 242 increases the emotion value of "relief" (strengthens the emotion) in response to the occurrence of an event. Alternatively, the emotion determination unit 242 decreases the emotion values ​​overall in response to the occurrence of an event. The output control unit 2913 controls the output of the avatar 100 in response to the emotion value of the avatar 100. For example, the output control unit 2913 causes the avatar 100 to speak in a clear, monotone voice.

[0329] When the detection unit 2911 detects the occurrence of an event, the setting unit 2912 refers to the character data 233 and sets the character of the avatar 100. The appearance of the avatar 100 displayed on a display device or the like changes according to the setting of the setting unit 2912.

[0330] For example, when the detection unit 2911 detects the occurrence of an event related to the user's circumstances, the setting unit 2912 sets a character that is expected to reassure the user. Specifically, when the detection unit 2911 detects the occurrence of an event, the setting unit 2912 can set the avatar 100 to a character that resembles a counselor, a teacher, or the user's parent. Furthermore, when the detection unit 2911 detects the occurrence of an event, the setting unit 2912 may set the character of the avatar 100 to a character that resembles a person of the same race as the user.

[0331] Furthermore, the behavior decision unit 246 and the behavior control unit 250 may change the processing content in response to the occurrence of an event being detected by the detection unit 2911. In other words, the behavior control unit 250 causes the avatar 100 to perform an action in response to the event.

[0332] For example, if the detection unit 2911 detects an event related to the user's circumstances, the behavior control unit 250 causes the avatar 100 to take an action that reassures the user when information is output via the avatar 100. Specifically, when information is output via the avatar 100, the behavior control unit 250 can cause the avatar 100 to take a gesture (such as raising a hand or bowing) that is used to greet someone in the user's home country.

[0333] Furthermore, for example, the setting unit 2912 refers to the character data 233 and sets the character "G" if the user is an "international student," a "Japanese," and "under 15 years old."

[0334] Below, examples of the operation of the output unit 291 will be described along with examples of events.

[0335] (Event: Studying Abroad) For example, if the detection unit 2911 detects that the user has studied abroad as an event, the output control unit 2913 outputs, via the avatar 100, information on how to adapt to the culture and environment of the place of residence that has changed as a result of studying abroad. The output control unit 2913 also outputs, via the avatar 100, information on, for example, an efficient method for learning the language of the country of study as a way to deal with challenges and difficulties that may arise during studying abroad. Note that outputting information via the avatar 100 means, for example, having the avatar 100 speak.

[0336] (Event: Change of job) For example, if the detection unit 2911 detects, as an event, that the user is about to change jobs by changing jobs, getting a job, re-employment, etc., the output control unit 2913 outputs, via the avatar 100, vocational training related to the changed job, methods for improving skills, and a specific curriculum tailored to the user's personal needs and goals.

[0337] Furthermore, the emotion determining unit 242 can change the emotion of the avatar 100 to a positive emotion and have the avatar 100 speak in a supportive manner for the user, thereby improving the motivation of the user.

[0338] (Event: Child) For example, when the detection unit 2911 detects, as an event, that the user's child has been born and that the user's child has grown to a certain age, the output control unit 2913 outputs, via the avatar 100, information indicating the most suitable extracurricular activities for the child.

[0339] The output control unit 2913 causes the avatar 100 to suggest lessons that suit the child's interests and abilities. The output control unit 2913 may also receive a report on the progress of the child's lessons and output advice according to the progress via the avatar 100.

[0340] (Example of Output Based on Reaction Rules) The output control unit 2913 can control the avatar 100 based on the reaction rules 231. The reaction rules 231 store messages to be output when an event related to the user's circumstances is detected. Note that the messages may be stored in the handling information 2922 instead of the reaction rules 231.

[0341] For example, the reaction rule 231 stores a message such as "First, try attending a welcome party for international students" as a message to a user who has studied abroad. Also, for example, the reaction rule 231 stores a message such as "We have online programming courses" as a message to a user who is considering changing jobs to an IT company. Also, for example, the reaction rule 231 stores a message such as "Why don't you try a soccer class" as a message to a user whose sports-loving child has just started elementary school.

[0342] An example of the operation flow by the output unit 291 is shown schematically in FIG.

[0343] Eleventh Embodiment Next, an eleventh embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and descriptions of configurations and processes similar to those of the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0344] Here, each control device has a language support function that detects the language acquisition status of the user 10 via each avatar and outputs information according to the detected acquisition status. For example, each control device can detect the level of understanding of the user's native language or a foreign language (such as English) and output language instruction according to the level of understanding, language skills, and information about vocabulary via the avatar, thereby outputting appropriate information to the user 10 according to the language acquisition status of the user 10.

[0345] 11, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and a support unit 290. The data structure of the character data 233 is shown schematically in FIG.

[0346] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0347] (Support Unit 290) The support unit 290 will be described in detail below. The support unit 290 includes a detection unit 2901 and an output control unit 2902, as shown in FIG.

[0348] The detection unit 2901 detects the learning status of the user 10 in various languages. The output control unit 2902 controls the control device 200 equipped with the sentence generation model to output information corresponding to the learning status detected by the detection unit 2901 to the user 10 via the avatar 100.

[0349] For example, the detection unit 2901 detects the learning status of a language other than the native language of the user 10 (for example, English for a Japanese person). The detection unit 2901 also detects the learning status of each language skill. Such skills include, for example, reading, writing, listening, and speaking.

[0350] The output control unit 2902 then controls the avatar 100 to output information corresponding to the detected learning status. For example, the output control unit 2902 outputs a learning curriculum corresponding to the learning status and the target level of the user 10. The output control unit 2902 may also output a learning curriculum for each skill. This allows for effective language learning with a curriculum tailored to the learning status and target level of each user 10.

[0351] The detection unit 2901 also detects the user 10's acquisition status of their native language. The output control unit 2902 then controls the output of information corresponding to the detected acquisition status of the native language via the avatar 100. For example, the detection unit 2901 detects the language development status of an infant (ages 1 to 3, before entering kindergarten) as the acquisition status. The output control unit 2902 then outputs information related to the mother-child communication method according to the language development status. The communication method may be, for example, a method related to songs, picture books, conversation, etc. This makes it possible to support the normal language development of infants.

[0352] The detection unit 2901 also detects the foreign language acquisition status of infants and kindergarteners. The detection unit 2901 also detects the parents' intentions regarding the foreign language acquisition goals for their infants and kindergarteners (e.g., aiming for native level). The output control unit 2902 then outputs a learning curriculum for each developmental stage (e.g., age) according to the acquisition status and the parents' intentions. This allows the optimal program to be provided according to the child's developmental stage, allowing the child to continue learning while maintaining their motivation.

[0353] The detection unit 2901 also detects the learning status of a user 10 (i.e., a working adult) who has completed learning a foreign language. The detection unit 2901 also detects the intended use of the foreign language (e.g., business use). The output control unit 2902 then outputs, for example, information on communication skills using the foreign language, intercultural understanding, or a learning curriculum for foreign language learning, depending on the learning status and intended use.

[0354] The avatar 100 can be set to, for example, the avatar of a popular cram school instructor, the avatar of a teacher at the school the user 10 attends, or a teacher character appearing in an anime. When outputting information related to a foreign language, the avatar 100 can be set to the avatar of a person from a country whose native language is that foreign language (for example, an American or British person for English). This makes it easier for the user 100 to accept the information output from the avatar 100.

[0355] 30 is a diagram schematically illustrating an example of an operation flow performed by the support unit 290. First, the support unit 290 detects the user's language learning status (step S1100). Then, the support unit 290 controls the control device 200 equipped with a sentence generation model to output information corresponding to the detected learning status to the user 10 via the avatar 100 (step S1101), and the process ends.

[0356] Twelfth Embodiment Next, a twelfth embodiment will be described. Here, differences from the above-described embodiment will be mainly described, and a description of the same configuration and processing as the above-described embodiment will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0357] The appearance of the avatar may be a human figure, as in avatars 100 and 101, or a character, as in avatar 102. For example, avatars 100 and 101 may be wearing clothes coordinated with the same clothes that user 10 owns, and suggest outfits to user 10. For example, avatars 100 and 101 may suggest outfits that include clothes that user 10 does not own.

[0358] Here, each control device 200 has an event detection function that detects the occurrence of a predetermined event and outputs information corresponding to the event. For example, each control device 200 detects an event in which the user 10 needs support. In this embodiment, such events include an event in which the user 10 needs fashion coordination, an event in which the user 10 needs relaxation or stress relief, an event in which the user 10 needs support while traveling, an event in which the user 10 needs cooking support, and the like. For example, each control device 200 can recognize the emotions of the user 10 and cause the avatar 100 to output an outfit corresponding to the recognized emotion, thereby enabling the avatar 100 to respond like a human. Furthermore, for example, each control device 200 can recognize the emotions of the user 10 and cause the avatar 100 to output (sing) music corresponding to the recognized emotion, thereby enabling the avatar 100 to respond like a human. Furthermore, for example, each control device 200 can recognize the emotions of the user 10 and cause the avatar 100 to output tourist information corresponding to the recognized emotion, thereby enabling the avatar 100 to respond like a human. Furthermore, for example, each control device 200 can recognize the emotions of the user 10 and cause the avatar 100 to output recipe suggestions according to the recognized emotions, thereby enabling the avatar 100 to respond like a human.

[0359] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0360] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0361] (Detection of Event) The output unit 291 will be described in detail. As shown in Fig. 24, the output unit 291 includes a detection unit 2911, a collection unit 2914, and an output control unit 2913. The output unit 291 also stores handling information 2922.

[0362] The detection unit 2911 detects the occurrence of a predetermined event. The detection unit 2911 detects an event in which the user 10 needs to coordinate their fashion. For example, the detection unit 2911 detects gestures such as the user 10 worrying about what to wear when going out or gestures such as the user choosing an outfit. The detection unit 2911 also detects an event in which the user 10 seeks relaxation or stress relief. For example, the detection unit 2911 detects these events by detecting speech by the user 10 or a predetermined behavior by the user 10 (e.g., relaxing on a sofa). The detection unit 2911 also detects an event in which the user 10 seeks tourist information at a travel destination. For example, the detection unit 2911 detects such an event based on speech between the user 10 and the avatar 100. The detection unit 2911 also detects an event in which the user 10 needs cooking assistance. For example, the detection unit 2911 detects such an event based on speech between the user 10 and the avatar 100, etc.

[0363] The collection unit 2914 also collects situation information indicating the situation of the user 10. The output control unit 2913 then controls the avatar 100 to output information corresponding to the situation information. For example, the collection unit 2914 collects information indicating the situation of the user 10 when selecting an outfit (such as the user 10's voice and an image of the user 10). At this time, the collection unit 2914 may also ask the avatar 100 about the purpose of the user 10's outfit. The collection unit 2914 also collects information indicating the situation of the user 10 when an event occurs in which the user 10 seeks relaxation or stress reduction (such as the user 10's voice and an image of the user 10). The collection unit 2914 also collects information indicating the situation of the user 10 when an event occurs in which the user 10 seeks tourist information (such as the user 10's voice and an image of the user 10). The collection unit 2914 also collects information indicating the situation of the user 10 when an event occurs in which the user 10 needs cooking assistance (such as the user 10's voice and an image of the user 10). This allows the control device 200 to recognize the emotion of the user 10 from the voice, image, etc. of the user 10 .

[0364] The output control unit 2913 controls the avatar 100 to perform actions according to the event that has occurred and the situation information of the user 10. When an event occurs that requires the user 10 to coordinate their fashion, the output control unit 2913 controls the avatar 100 to suggest fashion items and outfits based on the user 10's body type and face, as well as the user's preferences, mood, season, and situation (e.g., the purpose of going out in fashion). For example, the avatar 100 may be in a state where the suggested outfit is coordinated or wearing the suggested clothes. The avatar 100 may be in a state where the avatar 100 is wearing clothes that the user 10 owns. That is, the output control unit 2913 may wear clothes coordinated with the same items as the clothes the user 10 owns, suggesting, for example, "You should wear this outfit today." The avatar 100 may also be in a state where the user 10 does not own clothes. That is, the output control unit 2913 may wear clothes coordinated with the same items as the clothes the user 10 does not own (e.g., clothes from a brand the user 10 does not own), suggesting, for example, "I recommend you purchase this outfit." Furthermore, the output control unit 2913 controls the avatar 100 based on the emotions of the user 10 to make the user 10 happy, such as by praising the user by saying "It suits you" or by encouraging the user by saying "Good luck in the interview."

[0365] Furthermore, the output control unit 2913 controls the avatar 100 to output music that corresponds to the situation of the user 10 when an event occurs that requires relaxation or stress reduction. For example, the output control unit 2913 controls the avatar 100 to output music that corresponds to the emotions and needs of the user 10. As a specific example, if the user 10 is seeking relaxation before going to bed, the output control unit 2913 controls the avatar 100 to play relaxing music that induces sleep. Note that the music may be existing music, or may be improvised music created by a music generation AI installed in the control device 200.

[0366] Furthermore, the output control unit 2913 controls the avatar 100 to behave in accordance with the situation of the user 10 when an event occurs that causes the user 10 to request tourist information. For example, the output control unit 2913 controls the avatar 100 to suggest recommended spots and activities in accordance with the interests and situation of the user 10.

[0367] Furthermore, the output control unit 2913 controls the avatar 100 to perform an action that corresponds to the situation of the user 10 when an event occurs in which the user 10 needs cooking support. For example, the output control unit 2913 controls the avatar 100 to suggest dishes and recipes that correspond to the situation of the user 10 (cooking skills, ingredients, available time for cooking, etc.) with the aim of improving the cooking skills of the user 10. In this case, the output control unit 2913 controls the avatar 100 to suggest recipes that match the preferences and mood of the user 10.

[0368] 31 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The output unit 291 determines whether or not the occurrence of a predetermined event has been detected (step S1200). If the occurrence of the predetermined event has not been detected (step S1200; No), the output unit 291 waits until the occurrence of the predetermined event is detected.

[0369] On the other hand, if the occurrence of a predetermined event is detected (step S1200; Yes), the output unit 291 collects situation information indicating the situation of the user 10 (step S1201). Next, the output unit 291 controls the avatar 100 controlled by the control device 200 equipped with the sentence generation model to respond to the user 10 in accordance with the occurred event and the situation information (step S1202), and ends the process.

[0370] (Thirteenth Embodiment) Next, a thirteenth embodiment will be described. Here, the differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0371] The appearance of the avatar may be a human figure, such as avatar 100 and avatar 101, or a character, such as avatar 102. For example, avatars 100 and 101 may pose as insurance company salespeople and ask questions of user 10 about the contents of documents (application forms, etc.) used to enroll in insurance.

[0372] Here, each control device has an output function that recognizes the emotion of the user 10 and causes each avatar to output information about the insurance that the user 10 will subscribe to, according to the emotion of the user 10. For example, each avatar can output information according to the emotion of the user 10, thereby providing emotionally sensitive support to the user 10 regarding the subscription of insurance.

[0373] 2, the control device 200 has a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 291. The data structure of the character data 233 is shown schematically in FIG.

[0374] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0375] (Output of Information) The output unit 291 will be described in detail. As shown in Fig. 26, the output unit 291 has a recognition unit 2916 and an output control unit 2913. The output unit 291 also stores detection information 2923.

[0376] For example, the recognition unit 2916 recognizes the emotion of the user 10 based on detection information detected from the user 10 (e.g., detection information accumulated in the detection information 2923). Then, the output control unit 2913 causes the avatar 100, controlled by the control device 200 equipped with the sentence generation model, to output information about the insurance to be subscribed to by the user 10, to the user 10, in accordance with the emotion recognized by the recognition unit 2916. As a specific example, the output control unit 2913 causes the avatar 100 to output information for assisting in the preparation of documents to be used for enrolling in insurance. As an even more specific example, the output control unit 2913 causes the avatar 100 to output information for support in the preparation and filling out of an insurance contract or application form in accordance with the emotion of the user 10. In this way, the avatar 100 can reduce the hassle of procedures related to an insurance contract while being sensitive to the emotion of the user 10.

[0377] Furthermore, for example, the output control unit 2913 causes the avatar 100 to output information about insurance that is proposed for enrollment to the user 10. To give a specific example, the output control unit 2913 causes the avatar 100 to output information about consulting support when enrolling in insurance, in accordance with the emotions of the user 10. This allows the avatar 100 to explain the types and details of insurance in an easy-to-understand manner while being sensitive to the emotions of the user 10, and to propose insurance that meets the needs of the user 10, thereby increasing the trust of the user 10.

[0378] Furthermore, for example, the output control unit 2913 causes the avatar 100 to output a response to a question or inquiry about insurance from the user 10. To give a specific example, the output control unit 2913 responds to the question or inquiry about insurance in accordance with the emotions of the user 10. This allows the avatar 100 to provide support as an insurance consultation center while being sensitive to the emotions of the user 10, thereby increasing the trust of the user 10.

[0379] 32 is a diagram schematically illustrating an example of an operation flow by the output unit 291. The output unit 291 determines whether it is a predetermined timing for outputting information to the user 10 (for example, a timing at which a question or inquiry from the user is detected) (step S1300). If it is not the predetermined timing (step S1300; No), the output unit 291 waits until the predetermined timing arrives.

[0380] On the other hand, if it is the predetermined timing (step S1300; Yes), the output unit 291 recognizes the emotion of the user 10 based on the detection information detected from the user 10 (step S1301). Next, the output unit 291 causes the avatar 100 to output information about the insurance subscribed to by the user 10 to the user 10 in accordance with the emotion of the user 10 (step S1302), and ends the process.

[0381] Fourteenth Embodiment Next, a fourteenth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a system 5 according to this embodiment is shown schematically in FIG. 1.

[0382] The appearance of the avatar may be a human figure, such as avatar 100 and avatar 101, or a character, such as avatar 102. For example, avatars 100 and 101 appear only to give advice to the user when a predetermined event (e.g., a disaster) occurs, and output various individually optimized information such as "Run quickly," "Put out the fire," "Evacuate," "Be careful of aftershocks," and "The evacuation shelter is here."

[0383] Here, each control device has an event detection function that detects the occurrence of a predetermined event and causes each avatar to output information corresponding to the event. For example, each control device can detect the occurrence of a disaster and cause each avatar to output information regarding emergency responses to the disaster and evacuation guidance, thereby outputting appropriate information corresponding to the disaster to the user.

[0384] 33 is a diagram schematically illustrating the functional configuration of a control device 200 that controls the avatar 100. The control device 200 includes a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an event detection unit 293. The data structure of the character data 233 is schematically illustrated in FIG.

[0385] The event detection unit 293 realizes the above-mentioned output function. Details of the event detection unit 293 will be described later.

[0386] An example of an operational flow relating to the operation of determining the behavior of the avatar 100 is shown schematically in FIG.

[0387] (Event Detection) The event detection unit 293 will now be described in detail. Fig. 34 is a diagram illustrating a schematic functional configuration of the event detection unit 293. Here, the event detection unit 293 is provided in the control device 200, and causes the avatar 100 to output information corresponding to the detected event.

[0388] 34, the event detection unit 293 includes a detection unit 2911, a collection unit 2914, and an output control unit 2913. The event detection unit 293 also stores handling information 2922.

[0389] Each component of the event detection unit 293 is realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs running on the OS.

[0390] The detection unit 2911 detects the occurrence of a predetermined event. The output control unit 2913 then controls the avatar 100, which is controlled by the control device 200 equipped with the sentence generation model, to output information corresponding to the event detected by the detection unit 2911 to the user 10.

[0391] The collection unit 2914 also collects situation information indicating the situation of the user 10. The output control unit 2913 then controls the avatar 100 to output information corresponding to the situation information. For example, when an event occurs, the collection unit 2914 collects information indicating the situation of the site where the user 10 is located (such as the voice of the user 10). This allows the avatar 100 to recognize the emotions of the user 10 from the voice of the user 10, thereby understanding the situation of the site where the user 10 is located, and to give appropriate instructions in response to the event that has occurred through gesture control or the like.

[0392] The detection unit 2911 detects the occurrence of a disaster, and the output control unit 2913 controls the avatar 100 to output information corresponding to the disaster that has occurred.

[0393] The output control unit 2913 also controls the avatar 100 to output information about how to deal with an event that has occurred, which information is stored in the response information 2922. This allows the avatar 100 to support emergency response and evacuation guidance in the event of a disaster.

[0394] The detection unit 2911 may detect events such as earthquakes and fires from various sensor information. The detection unit 2911 may also communicate with an external server to detect events such as sudden heavy rain, tornadoes, and typhoons. The detection unit 2911 may also analyze surrounding sounds and detect an event from user speech or sounds generated from a television or the like.

[0395] 35 is a diagram schematically illustrating an example of an operation flow by the event detection unit 293. The event detection unit 293 determines whether or not the occurrence of a predetermined event has been detected (step S1400). If the occurrence of the predetermined event has not been detected (step S1400; No), the event detection unit 293 waits until the occurrence of the predetermined event is detected.

[0396] On the other hand, if the occurrence of a predetermined event is detected (step S1400; Yes), event detection unit 293 collects situation information indicating the situation of user 10 (step S1401). Next, event detection unit 293 controls avatar 100, which is controlled by control device 200 equipped with a sentence generation model, to output information corresponding to the occurred event and the situation information to user 10 (step S1402), and ends the process.

[0397] Fifteenth Embodiment Next, a fifteenth embodiment will be described. Here, differences from the above embodiments will be mainly described, and a description of the same configuration and processing as the first embodiment will be omitted. Fig. 36 is a diagram schematically illustrating an example of a system 5 according to the fifteenth embodiment.

[0398] The user 10a, the user 10b, and the user 10c are users who communicate with the avatar 100. In the description of this embodiment, the user 10a, the user 10b, and the user 10c may be collectively referred to as the user 10.

[0399] 36(3), the avatar 103 may be displayed as a character rather than a person, and may communicate with the user 12 as the character. Note that the avatar 103 displayed as such a character may have a form resembling a pet 13, as will be described later.

[0400] Here, user 10a, user 10b, and user 10c make up a family. In other words, user 10a, user 10b, and user 10c are members of a family. User 10 also keeps a pet 13. Note that while FIG. 36 shows an example in which the pet 13 is a dog, this is merely an example and is not limiting, and the pet may be, for example, a cat, rabbit, hamster, or other type of animal.

[0401] As will be described later, the avatar 100 provides advice information about the pet 13 to the user 10, but the user 10 at this time does not have to be a member of the family. Also, as will be described later, the avatar 100 provides advice information about the family to the user 10, but the user 10 at this time does not have to keep a pet 13.

[0402] The appearance of the avatar may be a human figure, as in avatar 100 and avatar 101, or may have a form that resembles the user's pet 13, as in avatar 103. By imitating the appearance of the pet 13, the user will be more likely to accept the advice information and the like that is provided to the avatar 103, and it is also thought that the avatar 103 will be particularly easy for children to relate to.

[0403] The server 300 stores user reaction information regarding user reactions to the avatar 100, received from the control device 200. The server 300 receives and stores user reaction information not only from the control device 200 that controls the avatar 100, but also from the control devices 201 and 202 that control the avatars 101 and 103. The server 300 then analyzes the user reaction information from the control devices 200, 201, and 202 and updates the reaction rules.

[0404] The control device 200 also has a function of recognizing the state (behavior and emotions) of the pet 13. The control device 200 recognizes the behavior, etc. of the pet 13 by analyzing facial images of the pet 13 acquired by the camera function and sounds of the pet 13 acquired by the microphone function. The control device 200 determines the behavior, etc. of the avatar 100 to be performed based on the recognized behavior, etc. of the pet 13.

[0405] The control device 200 according to this embodiment can provide advice information about the pet 13 via the avatar 100. For example, the control device 200 recognizes the state of the pet 13. The state of the pet 13 includes the behavior and emotions of the pet 13. The control device 200 provides advice information about the pet 13 according to the recognized state of the pet 13.

[0406] As an example, when the control device 200 recognizes a behavior (appearance) or emotion of the pet 13 indicating that it wants to play with the user 10, the control device 200 controls the avatar 100 so that the avatar 100 takes an action to start a conversation with the user 10. Specifically, the avatar 100 makes an utterance indicating that advice information is about to be provided, such as "I have some advice about your pet." Note that "pet" in the content of the utterance may be the name of the pet 13.

[0407] Next, the control device 200 generates advice information regarding the pet 13 based on the recognized state of the pet 13. The advice information includes, but is not limited to, information regarding the relationship between the pet 13 and the user 10 and information regarding care of the pet 13. Here, the avatar 100 provides advice information by speech that is in line with the state of the pet 13, such as "Your pet wants to play. Play with it and you'll become friends with it."

[0408] As described above, in the present embodiment, the control device 200 recognizes the state of the pet 13 and causes the avatar 100 to take an action corresponding to the recognized state of the pet 13, thereby being able to provide the user 10 with appropriate advice regarding the pet 13. In other words, the control device 200 according to the present embodiment allows the avatar 100 to take an appropriate action for the user 10. Furthermore, the control device 200 can provide advice based on a deeper understanding of the state (e.g., feelings and behavior) of the pet 13, thereby supporting the user 10 (pet owner) in building a better relationship with the pet 13.

[0409] Furthermore, the control device 200 according to this embodiment can provide advice information about the family via the avatar 100. For example, the control device 200 recognizes the state of the users 10 who make up the family. The state of the users 10 includes the behavior and emotions of the users 10. The control device 200 provides advice information about the family according to the recognized state of the users 10.

[0410] As an example, when the control device 200 recognizes a behavior (appearance) or emotion indicating that the user 10a is lonely, the avatar 100 executes an action of starting a conversation with the other users 10, 10b and 10c, of the users 10 who make up the family. Specifically, the avatar 100 makes an utterance indicating that advice information will be provided, such as "I have some advice about the user 10a." Note that the "user 10a" in the utterance content may be the name of the user 10a.

[0411] Next, the control device 200 generates advice information about the user 10a based on the recognized state of the user 10a. The advice information includes, but is not limited to, information about communication between family members. In this example, the avatar 100 provides advice information tailored to the state of the user 10a to the users 10b and 10c by speaking, for example, "The user 10a seems lonely. You might want to talk to him and try to communicate with him."

[0412] As described above, in this embodiment, the control device 200 recognizes the state of a user 10 (e.g., user 10a) who constitutes a family, and causes the avatar 100 to take an action corresponding to the state of the recognized user 10 (user 10a), thereby making it possible to provide appropriate advice about the family to users other than the recognized user 10 (e.g., users 10b and 10c). In other words, the control device 200 according to this embodiment allows the avatar 100 to take an appropriate action toward the user 10. Furthermore, the control device 200 can facilitate communication between family members by appropriately conveying the state (e.g., opinions and emotions) of the family members and providing advice.

[0413] 37 is a diagram schematically illustrating the functional configuration of a control device 200 that controls the avatar 100. The control device 200 is configured by a control unit having a sensor unit 210, a sensor module unit 220, a storage unit (memory unit) 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, and a communication processing unit 280. The data structure of the character data 233 is schematically shown in FIG.

[0414] The control object 252 includes a display device 2521, a speaker 2522, etc. The display device 2521 displays the avatar 100 itself and images related to the avatar 100. The speaker 2522 outputs sounds related to the conversation and actions of the avatar 100. The posture, gestures, and facial expressions of the avatar 100 are examples of the attitude of the avatar 100.

[0415] The history data 232 may include user information for each of the multiple users 10 associated with the identification information of the users 10. The user information includes information indicating the characteristics of the users 10, such as the user's 10 personality, interests, hobbies, and inclinations. Such user information may be estimated from the user's 10 behavioral history or may be registered by the user 10 himself / herself. The history data 232 also includes the pet's 13's past emotional values ​​and behavioral history. This emotional value and behavioral history may be recorded for each pet 13, for example, by being associated with the pet's 13 identification information. At least a portion of the storage unit 230 is implemented as a storage medium such as a memory. The storage unit 230 may include a person DB that stores the user's 10 facial image, attribute information of the user 10, etc., and a pet DB that stores the pet's 13 facial image, attribute information of the pet 13, etc.

[0416] The voice emotion recognition unit 221 of the sensor module unit 220 analyzes the voice of the user 10 detected by the microphone 211 to recognize the emotion of the user 10. The voice emotion recognition unit 221 also analyzes the voice (cry) of the pet 13 detected by the microphone 211 to recognize the emotion of the pet 13. For example, the voice emotion recognition unit 221 extracts feature amounts such as frequency components of the voice, and recognizes the emotion of the pet 13 based on the extracted feature amounts.

[0417] The facial expression recognition unit 223 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 213. The facial expression recognition unit 223 also recognizes the facial expression and emotion of the pet 13 from the image of the pet 13 captured by the 2D camera 213. For example, the facial expression recognition unit 223 recognizes the facial expression and emotion of the pet 13 based on the shape, positional relationship, etc. of the eyes and mouth.

[0418] The face recognition unit 224 recognizes the face of the user 10. The face recognition unit 224 also recognizes the face of the pet 13. The face recognition unit 224 recognizes the pet 13 by matching a face image stored in a pet DB (not shown) with a face image of the pet 13 captured by the 2D camera 213.

[0419] The state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220. The state recognition unit 240 also recognizes the state of the pet 13 based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "Your pet is wagging its tail" or "There is an 80% chance that your pet does not look angry." The state recognition unit 240 performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "Your pet wants to play."

[0420] The emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the state recognition unit 240. The emotion determination unit 242 also determines an emotion value indicating the emotion of the pet 13 based on the information analyzed by the sensor module unit 220 and the state of the pet 13 recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the pet 13 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the pet 13.

[0421] Here, the emotion value indicating the emotion of the user 10 or the pet 13 (hereinafter sometimes referred to as "user, etc.") is a value indicating the positive or negative emotion of the user, etc. For example, if the emotion of the user, etc. is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the emotion value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user, etc. is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the emotion value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user, etc. is none of the above ("neutral"), the value indicates 0. Note that the type of emotion of the user 10 and the type of emotion of the pet 13 may be different or the same. Furthermore, the emotion value of the emotion of the user 10 may be different or the same as the emotion value of the emotion of the pet 13.

[0422] Furthermore, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user or the like recognized by the state recognition unit 240 .

[0423] Specifically, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 in accordance with rules for updating the emotion value of the avatar 100, which are determined in correspondence with the information analyzed by the sensor module unit 220 and the state of the user, etc., recognized by the state recognition unit 240.

[0424] For example, when the state recognition unit 240 recognizes that the user or the like looks lonely, the emotion determination unit 242 increases the emotion value of "sadness" of the avatar 100. When the state recognition unit 240 recognizes that the user or the like is smiling, the emotion determination unit 242 increases the emotion value of "joy" of the avatar 100.

[0425] The behavior recognition unit 244 recognizes the behavior of the user, etc., based on the information analyzed by the sensor module unit 220 and the state of the user, etc., recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the user, etc., are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (for example, "laughing," "angry," "asking a question," "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user, etc.

[0426] The behavior determination unit 246 determines a behavior corresponding to the behavior of the user, etc., recognized by the behavior recognition unit 244, based on the current emotion value of the user, etc., determined by the emotion determination unit 242, history data 232 of past emotion values ​​determined by the emotion determination unit 242 before the current emotion value of the user, etc., was determined, and the emotion value of the avatar 100. In this embodiment, the behavior determination unit 246 uses one most recent emotion value included in the history data 232 as the past emotion value of the user, etc., but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 246 may use multiple most recent emotion values ​​as the past emotion value of the user, etc., or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 246 may determine a behavior corresponding to the behavior of the user, etc., by taking into consideration not only the current emotion value of the avatar 100 but also the history of the past emotion values ​​of the avatar 100. The behavior determined by the behavior determination unit 246 includes gestures performed by the avatar 100 or speech content of the avatar 100.

[0427] The behavior determination unit 246 according to this embodiment determines the behavior of the avatar 100 as a behavior corresponding to the behavior of the user, etc., based on a combination of the past and current emotional values ​​of the user, etc., the emotional value of the avatar 100, the behavior of the user, etc., and the reaction rules 231. For example, if the past emotional value of the user, etc. is a positive value and the current emotional value is a negative value, the behavior determination unit 246 determines the behavior corresponding to the behavior of the user, etc., as a behavior for changing the emotional value of the user, etc. to a positive value.

[0428] The reaction rules 231 may include rules for determining the behavior of the avatar 100 based on not only the emotions of the user 10 but also the emotions of the pet 13, the emotions of the avatar 100, and the behavior of the pet 13. For example, the reaction rules 231 may determine the behavior of the avatar 100 based on a combination of the past and current emotional values ​​of the pet 13, the emotional value of the avatar 100, and the behavior of the pet 13.

[0429] For example, the reaction rules 231 prescribe behaviors of the avatar 100 corresponding to behavioral patterns when the state of the pet 13 requires advice from the user 10, when the user 10 has responded to advice information provided, etc. As an example, when the behavior determination unit 246 recognizes the behavior or emotion of the pet 13 as the state of the pet 13 based on the reaction rules 231, it determines, as the behavior of the avatar 100, behavior that provides advice information about the pet 13 to the user 10 according to the state of the pet 13.

[0430] Furthermore, for example, the reaction rules 231 prescribe behaviors of the avatar 100 corresponding to behavioral patterns such as when the state of the user 10a who is a member of the family requires advice from the other users 10b and 10c, or when there is a reaction from the user 10 to the advice information provided. As an example, when the behavior determining unit 246 recognizes the behavior or emotion of the user 10a as the state of the user 10a based on the reaction rules 231, it determines, as the behavior of the avatar 100, behavior that provides advice information about the family that corresponds to the state of the user 10a to the users 10b and 10c.

[0431] The memory control unit 248 determines whether or not to store data including the behavior of the user, etc. in the history data 232 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0432] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the avatar 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 246, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 246, is equal to or greater than a threshold value, it is determined that data including the behavior of the user, etc. is to be stored in the history data 232.

[0433] When the memory control unit 248 decides to store data including the behavior of the user or the like in the history data 232, the memory control unit 248 stores in the history data 232 the behavior determined by the behavior determination unit 246, information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user or the like recognized by the state recognition unit 240 (for example, the facial expression, emotions of the user 10, the facial expression, emotions of the pet 13, etc.).

[0434] As described above, in the control device 200 according to this embodiment, the state recognition unit 240 and the behavior recognition unit 244 recognize the states (behaviors and emotions) of the user 10 and the pet 13 that make up the family. The behavior determination unit 246 determines the behavior of the avatar 100 that corresponds to the recognized states of the user 10 and the pet 13. The behavior control unit 250 controls the avatar 100 (controls the control target 252) based on the determined behavior of the avatar 100.

[0435] Specifically, when the behavior control unit 250 recognizes the behavior of the pet 13 as the state of the pet 13, it determines the behavior of the avatar 100 to be an behavior that provides advice information about the pet 13 in accordance with the behavior of the pet 13, and controls the avatar 100 (controls the control object 252).

[0436] The advice information includes information regarding the relationship between the pet 13 and the user 10 who keeps the pet 13 (hereinafter, this may be referred to as "relationship information"), and information regarding the care of the pet 13 (hereinafter, this may be referred to as "care information"). The relationship information includes information for improving or enhancing the relationship between the pet 13 and the user 10, but these are examples and are not limited thereto. The care information includes information on how to deal with emotional concerns regarding the care of the pet 13, information on the daily care of the pet 13, etc., but these are examples and are not limited thereto. The advice information may include both the relationship information and the care information, or may include one of them. In other words, the advice information includes at least one of the relationship information and the care information. The advice information is not limited to the relationship information and the care information, but may also include other information, such as characteristic information of the pet 13.

[0437] For example, when the behavior control unit 250 recognizes that the pet 13 is wagging its tail, making a whining noise, or other behavior (sign) that indicates that it wants to play with the user 10, the behavior control unit 250 controls the avatar 100 so that the avatar 100 takes an action to start a conversation with the user 10. Specifically, the avatar 100 makes an utterance indicating that advice information will be provided, such as "I have some advice for you about your pet."

[0438] Next, the behavior control unit 250 generates advice information regarding the pet 13 based on the recognized behavior of the pet 13, and provides the generated advice information by speech from the avatar 100. For example, the behavior control unit 250 provides advice information (here, related information) by speech from the avatar 100 that is in line with the state of the pet 13, such as "Your pet wants to play. Play with it and you'll become friends with it."

[0439] In the above description, the behavior control unit 250 recognizes the behavior of the pet 13 as the state of the pet 13. However, the behavior control unit 250 may also recognize the emotion of the pet 13. For example, if the behavior control unit 250 recognizes that the emotion of the pet 13 is "unpleasant" or "angry," such as when the pet 13 is barking or baring its teeth, the behavior control unit 250 generates advice information about the pet 13 based on the recognized emotion of the pet 13, and the generated advice information is provided by the avatar 100 through speech. For example, the behavior control unit 250 provides advice information (here, care information) that is appropriate to the state of the pet 13 through speech by the avatar 100, such as "Your pet seems to be unhappy. You will be happy if you feed it" or "Your pet seems to be unhappy. You will be happy if you take it for a walk." Furthermore, since the emotion value of the avatar 100 is determined by the emotion determination unit 242, the behavior control unit 250 may provide advice information, based on such emotion value, by having the avatar 100 speak advice information that is responsive to the feelings (concerns) of the user 10, such as, "Walking is hard, but I recommend walking to maintain your health."

[0440] In this way, the behavior control unit 250 according to this embodiment can provide the user 10 with appropriate advice regarding the pet 13 via the avatar 100. Furthermore, the behavior control unit 250 can provide advice based on a deeper understanding of the state of the pet 13 (e.g., feelings and behavior), thereby supporting the user 10 (pet owner) in building a better relationship with the pet 13.

[0441] In addition, when the behavior control unit 250 recognizes the state of a user 10 who is a member of a family, it determines the behavior of the avatar 100 to be an action that provides advice information about the family in accordance with the recognized state of the user 10, and controls the avatar 100 (controls the control object 252).

[0442] Specifically, when the behavior control unit 250 recognizes the behavior of a user 10 who is a member of a family as the state of the user 10, it determines the behavior of the avatar 100 to be an behavior that provides advice information about the family in accordance with the behavior of the user 10, and controls the control object 252.

[0443] For example, when the behavior control unit 250 receives a voice such as "I feel lonely" or "I feel bored" from the user 10a and recognizes a behavior (appearance) indicating that the user 10a is lonely, the behavior control unit 250 controls the avatar 100 to take an action to start a conversation with users 10b and 10c other than the user 10a among the multiple users 10 that make up the family. Specifically, the avatar 100 makes an utterance indicating that advice information will be provided, such as "I have some advice for user 10a."

[0444] Next, the behavior control unit 250 generates advice information about the family based on the recognized behavior of the user 10a, and provides the generated advice information by speech from the avatar 100. For example, the behavior control unit 250 provides advice information tailored to the state of the user 10a by speech from the avatar 100 to the users 10b and 10c, such as "The user 10a seems lonely. You might want to talk to him and try to communicate with him" or "The user 10a seems lonely. Why don't you go out together?"

[0445] Here, the behavior control unit 250 may generate advice information based on user information indicating the characteristics of the users 10. For example, the behavior control unit 250 generates advice information based on at least one of the personalities, interests, hobbies, and inclinations of each of the multiple users 10 included in the user information. As an example, if the interest or preoccupation of user 10a is "shopping," the behavior control unit 250 provides advice information in line with the interest or preoccupation of user 10a by having the avatar 100 speak, such as "User 10a seems lonely. Why don't you go shopping together?"

[0446] The user information is not limited to the personality, interests, hobbies, and inclinations of the user 10, but may also include other characteristics such as hobbies and preferences.

[0447] In the above description, the behavior control unit 250 recognizes the behavior of the user 10 as the state of the user 10. However, the behavior control unit 250 may also recognize the emotions (opinions) of the user 10. For example, if the behavior control unit 250 receives speech from the user 10a, such as "The user 10b gets angry easily" or "The user 10b is noisy," and recognizes that the user 10a's emotions toward the user 10b are "uncomfortable" or "anxious," the behavior control unit 250 generates advice information about the family based on the recognized emotions of the user 10, and provides the generated advice information to the user 10b by speech through the avatar 100. For example, the behavior control unit 250 provides advice information tailored to the state of the user 10a, such as "It seems that what you said is making the user 10a unhappy. Why don't you try changing the way you say it?" by speech through the avatar 100 to the user 10b. Furthermore, since the emotion value of the avatar 100 is determined by the emotion determination unit 242, the behavior control unit 250 may provide advice information, based on such emotion value, by having the avatar 100 speak such advice information that is sympathetic to the feelings (concerns) of the user 10b, such as, "I understand that you are angry, but if you change the way you say it, your feelings will be conveyed more effectively."

[0448] In this way, the behavior control unit 250 according to this embodiment can provide appropriate advice about family members to the user 10 via the avatar 100. Furthermore, the behavior control unit 250 can facilitate smooth communication between family members by appropriately conveying the state of the family members (for example, opinions and emotions) and providing advice, or in other words, can support deepening mutual understanding between family members.

[0449] In addition, the provision of advice information to the user 10 is not limited to being done by speech using the speaker 2522, which is the control object 252, but may also be done through the display of the avatar 100 on the display device 2521, which is the control object 252, etc.

[0450] Furthermore, the behavior control unit 250 changes the content of the advice information according to the emotion of the avatar 100. For example, the behavior control unit 250 changes the content of the advice information to be generated according to the emotion value of the avatar 100. Specifically, when the behavior control unit 250 receives voice from a user indicating an arrogant attitude, the emotion value of "anger" or "sadness" of the avatar 100 increases, and the behavior control unit 250 generates and provides advice information according to the increased emotion value. The content of such advice information differs from advice information provided to a user who does not receive voice indicating abusive language or an arrogant attitude (i.e., a user who makes good comments and has a good attitude). As an example, advice information provided to a user who receives voice indicating abusive language, etc., may contain less advice or omit part of the advice content compared to advice information provided to a user who does not receive voice indicating abusive language, etc.

[0451] Furthermore, if the emotional value of "anger" or "sadness" of the avatar 100 increases due to loud ambient noise and the user's voice cannot be detected, the behavior control unit 250 generates and provides advice information according to the increased emotional value. The content of such advice information differs from the advice information provided to the user when the user's voice can be detected. For example, advice information provided to the user when the user's voice cannot be detected may contain less advice or omit some of the advice content compared to advice information provided to the user when the user's voice can be detected.

[0452] Furthermore, when the behavior control unit 250 receives voice from the user indicating a friendly attitude toward the avatar 100, such as "Thank you for your advice," the emotional value of "happiness" or "pleasure" of the avatar 100 increases, and advice information corresponding to the increased emotional value is generated and provided. The content of such advice information differs from that of advice information provided to a user who does not receive voice indicating a friendly attitude (i.e., a user with a normal attitude). As an example, advice information provided to a user who receives voice indicating a friendly attitude may include more extensive advice suggestions or additional advice content than advice information provided to a user who does not receive voice indicating a friendly attitude.

[0453] The behavior control unit 250 may change the frequency with which advice information is provided to the user based on the emotional value of the avatar 100. For example, the behavior control unit 250 may increase the frequency with which advice information is provided to the user if the user's reaction to the provided advice information is not negative (specifically, the user gives a positive response such as "Like") and the emotional value of "Happy" of the avatar 100 increases. Furthermore, the behavior control unit 250 may decrease the frequency with which advice information is provided to the user if the user's reaction to the provided advice information is negative (specifically, the user gives a negative response such as "That's not right") and the emotional value of "Sad" of the avatar 100 increases.

[0454] Fig. 38 is a diagram illustrating an example of an operational flow relating to an operation for determining the behavior of avatar 100. The operational flow illustrated in Fig. 38 is repeatedly executed. At this time, it is assumed that information analyzed by sensor module unit 220 has been input.

[0455] First, in step S1500, the state recognition unit 240 recognizes the state of the user, etc., based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling," and performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "Daddy is alone and looks lonely." The state recognition unit 240 also generates perceptual information such as "Your pet is wagging its tail" or "There is an 80% chance that your pet is not angry," and generates semantic information such as "Your pet wants to play."

[0456] In step S1501 , the emotion determining unit 242 determines an emotion value indicating the emotion of the user, etc., based on the information analyzed by the sensor module unit 220 and the state of the user, etc., recognized by the state recognizing unit 240 .

[0457] In step S1502, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user or the like recognized by the state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user or the like to the history data 232.

[0458] In step S1503 , the behavior recognition unit 244 recognizes a behavior classification of the user or the like based on the information analyzed by the sensor module unit 220 and the state of the user or the like recognized by the state recognition unit 240 .

[0459] In step S1504, the behavior determination unit 246 determines the behavior of the avatar 100 based on the usage age acquired in step S52 of FIG. 4, the combination of the current emotional value of the user 10, etc. determined in step S15101 of FIG. 38 and the past emotional value included in the history data 232, the emotional value of the avatar 100, the behavior of the user 10, etc. recognized by the behavior recognition unit 244, and the reaction rules 231.

[0460] In step S1505, the behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior determined by the behavior determination unit 246. For example, the behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior determined according to at least one of the state of the user 10 who constitutes the family and the state of the pet 13.

[0461] In step S1506, the memory control unit 248 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0462] In step S1507, the storage control unit 248 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10, etc. in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S1508.

[0463] In step S1508, the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the status of the user 10, etc. recognized by the status recognition unit 240 are stored in the history data 232.

[0464] As described above, the control device 200 includes a control unit that recognizes at least one of the states of the users 10 who make up a family and the states of the pets 13 kept by the users 10, determines the behavior of the avatar 100 corresponding to the recognized state, and controls the avatar 100 based on the determined behavior of the avatar 100. This allows the control device 200 to perform appropriate actions, such as providing appropriate advice to the user 10 regarding the family or pets 13.

[0465] Furthermore, when the control unit of the control device 200 recognizes the behavior of the pet 13 as the state of the pet 13, it determines, as the behavior of the avatar 100, a behavior that provides advice information about the pet 13 in accordance with the behavior of the pet 13. This allows the control device 200 to provide, via the avatar 100, appropriate advice information about the pet 13 that is in line with the behavior of the pet 13.

[0466] Furthermore, when the control unit of the control device 200 recognizes the emotion of the pet 13 as the state of the pet 13, the control unit determines, as the behavior of the avatar 100, an action that provides advice information about the pet 13 in accordance with the emotion of the pet 13. This allows the control device 200 to provide, via the avatar 100, appropriate advice information about the pet 13 that is in line with the emotion of the pet 13.

[0467] Furthermore, the advice information regarding the pet 13 includes at least one of information regarding the relationship between the pet 13 and the user 10 who keeps the pet 13, and information regarding the care of the pet 13. This enables the control device 200 to provide, as advice information, information for improving the relationship between the pet 13 and the user 10, methods for dealing with emotional concerns regarding the care of the pet 13, and the like.

[0468] Furthermore, when the control unit of the control device 200 recognizes the behavior of a user 10 who is a member of a family as the state of the user 10, the control unit determines, as the behavior of the avatar 100, behavior that provides advice information about the family in accordance with the behavior of the user 10. This allows the control device 200 to provide, via the avatar 100, appropriate advice information about the family that is in line with the behavior of the user 10 who is a member of the family.

[0469] Furthermore, when the control unit of the control device 200 recognizes the emotion of a user 10 constituting a family as the state of the user 10, the control unit determines, as the behavior of the avatar 100, an action that provides advice information about the family in accordance with the emotion of the user 10. This allows the control device 200 to provide, via the avatar 100, appropriate advice information about the family that is in line with the emotion of the user 10 constituting the family.

[0470] Furthermore, the control unit of the control device 200 generates advice information based on at least one of the personality, interests, hobbies, and inclinations of each of the multiple users 10 who make up a family. This allows the control device 200 to provide appropriate advice information about the family that is suited to the personality, etc. of each user 10.

[0471] Sixteenth Embodiment Next, a sixteenth embodiment will be described. Here, the differences from the above embodiments will be mainly described, and a description of the same configurations and processes as the above embodiments will be omitted.

[0472] FIG. 39 is a diagram schematically illustrating an example of a control system 1 according to a sixteenth embodiment. As illustrated in FIG. 39, the control system 1 includes a control device 400 that controls multiple avatars 120, a linked device 500, and a server 300. The multiple control devices 400a to 400d are each managed by a user. The appearance of the multiple avatars 120 may be a human-like appearance, as with the avatars 120a to 120c, or a stuffed animal-like appearance, as with the avatar 120d. In the description of this embodiment, when the avatars 120a to 120d are not distinguished from each other, they may simply be referred to as avatars 120. Similarly, when the control devices 400a to 400d are not distinguished from each other, they may simply be referred to as control devices 400. The control device 400 that controls the avatars 120 is connected wirelessly or wired to a display device such as a PC monitor or a smartphone display, or to a wearable device or the like that supports at least one of augmented reality, virtual reality, and mixed reality. The control device 400 is an example of an electronic device.

[0473] As shown in FIG. 39 , the avatar 120 controlled by the control device 400 communicates with users in various forms. For example, as shown on screens 110a to 110c in FIG. 39 , the avatars 120a to 120c are represented as people or the like displayed on a display device or the like, and communicate with users via input / output devices provided in the control devices 400a to 400d. Also, as shown on screen 110d in FIG. 39 , the avatar 120d communicates in a virtual space with users 13a and 13b, who are displayed as people or characters on a display device or the like. Also, as shown on screen 110d in FIG. 39 , the avatar 120d can be displayed not as a person but as a character, such as a stuffed animal, and communicate with users 13a and 13b as that character.

[0474] The appearance of the avatar may be a human figure like avatars 120a to 120c, or a stuffed toy character like avatar 120d. For example, avatars 120a to 120c may be dressed as lawyers or counselors, and may nod and listen to users' stories just like real lawyers or counselors, thereby providing emotional support and various information to users facing issues such as divorce.

[0475] The same description of the avatar 120 of this embodiment as that of the avatar 100 of the above embodiment will be omitted. Also, the same description of the control device 400 of this embodiment as that of the control device 200 of the above embodiment will be omitted.

[0476] The control device 400 can execute actions in cooperation with the linked devices 500. The linked devices 500 are, for example, a thermometer, a blood pressure monitor, a smartwatch (including those capable of measuring heart rate, etc.), a body composition monitor, a terminal device (a personal computer (PC), a smartphone, a tablet, etc.), a television, a display, and training equipment (a road runner, a fitness bike, etc.). These linked devices 500 are communicably connected to the control device 400 via the communication network 20, and transmit and receive information to and from the control device 400. With this configuration, the linked devices 500 control themselves, converse with the user, provide information, etc., in accordance with instructions from the control device 400.

[0477] This disclosure describes examples of providing appropriate information and advice as various actions to users who wish to consider various issues. Here, the contents of various issues include, for example, matters related to divorce, legal issues and concerns, procedures at local governments, consumer issues such as consumer troubles, matters related to debt (debt issues), and matters related to custody.

[0478] Specifically, control device 400 is placed in a government office, a counseling center, or the like, and provides various information through conversations, images, etc. in accordance with the contents of various considerations and emotions of the user acquired through conversations between avatar 120 and the user. The contents of various considerations of the user are an example of consideration information.

[0479] For example, the control device 400 recognizes a conversation from the user regarding divorce and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized divorce-related matters, for example, through a conversation with the avatar 120 and by displaying the information on its display device or the linked device 500 (terminal device). At this time, the avatar 120 appears, for example, as a character modeled after a lawyer, plays the role of a conversation partner, and provides appropriate advice while sharing the user's worries and anxieties based on the content of the conversation and the user's emotions (the user / the user). For example, the avatar 120 explains how to file for divorce and examples of property division upon divorce. Furthermore, for example, the avatar 120 appears as a character modeled after a fortune teller, and plays the role of a conversation partner until the user's emotions calm down. In other words, the control device 400 provides the user with at least one of legal information and administrative procedure information related to divorce proceedings via the avatar 120 or the like.

[0480] The control device 400 also recognizes legal issues and concerns from the user through conversation and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized legal issues and concerns, for example, through conversation using the avatar 120 and by displaying information on its display device or the linked device 500 (terminal device). At this time, the avatar 120 appears, for example, as a character modeled after a lawyer and provides advice regarding the legal issues and concerns based on the content of the conversation and the user's emotions (the user / the user). The avatar 120 also introduces the user to an appropriate legal expert and provides information regarding legal procedures. In other words, the control device 400 provides the user with at least one of information regarding an expert corresponding to the information regarding the legal issue or concern and information regarding legal procedures via the avatar 120 or the like. The user may be not only an individual but also a corporate (corporation) representative.

[0481] The control device 400 also recognizes conversations from the user regarding procedures at a local government (such as a city hall) and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized local government procedures, for example, through conversations with the avatar 120 and screen displays on its own display device and the linked device 500 (terminal device). The avatar 120, for example, appears as a person modeled after a local government employee and provides information and advice regarding the local government procedures according to the content of the conversation and the user's emotions (the user / the control device 400). The avatar 120 also provides support regarding the preparation of necessary documents and how to proceed with the procedures. For example, if the control device 400 is requesting a moving-in notification for moving procedures, the avatar 120 displays the notification while explaining the required fields and providing guidance on which counter to go to. In other words, the control device 400 provides the user with at least one of information regarding the preparation of documents for the procedures and information regarding how to proceed with the procedures via the avatar 120 or the like.

[0482] The control device 400 also recognizes consumer-related matters, such as consumer troubles, through conversation and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized consumer-related matters, for example, through conversation with the avatar 120 and through screen display on its own display device or the linked device 500 (terminal device). At this time, the avatar 120 appears, for example, as a character modeled after a customer service representative and provides information and advice regarding laws and regulations related to consumer rights depending on the content of the conversation and the user's emotions (user / self). The avatar 120 also provides support for dealing with consumer troubles and complaints. For example, the avatar 120 introduces consumer organizations or customer service centers depending on the case. In other words, the control device 400 provides the user with at least one of information regarding laws and regulations related to consumer rights via the avatar 120, etc.

[0483] The control device 400 also recognizes matters related to debt (debt problems) through conversation and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized debt (debt problems), for example, through conversation with the avatar 120 and through screen display on its own display device or the linked device 500 (terminal device). At this time, the avatar 120 appears, for example, as a character modeled after a counselor at a consultation center. Depending on the content of the conversation and the user's emotions (the user / the user), the avatar 120 provides mental support, for example, by understanding the user's anxiety or fear and suggesting appropriate emotional support and coping methods. The avatar 120 also provides information on laws and procedures related to debt problems and referrals to specialists. For example, the avatar 120 may refer a user with multiple debts to a debt consolidation specialist. In other words, the control device 400 engages in conversation to provide mental support to the user via the avatar 120 or the like, and provides the user with at least one of information on debt-related laws, debt-related administrative procedures, and specialists according to the debt information.

[0484] The control device 400 also recognizes matters related to parental authority through conversation and determines the user's emotions. The control device 400 provides the user with information corresponding to the recognized matters related to parental authority, for example, through conversation with the avatar 120 and through screen display on its own display device or the linked device 500 (terminal device). At this time, the avatar 120 appears, for example, as a character modeled after a lawyer, and provides support by understanding the user's emotional expression based on the content of the conversation and emotions (user / self) and giving advice that takes into consideration the user's position and emotions. The avatar 120 also provides information on laws and procedures related to parental authority. For example, the avatar 120 introduces relationships with children and custody cases. In other words, the control device 400 engages in conversation to support the user's position through the avatar 120, etc., and provides the user with at least one of information on laws related to parental authority and information on administrative procedures related to parental authority.

[0485] In this manner, in the present disclosure, the control device 400 can provide various information through conversations, images, etc., in accordance with the content of various considerations (consideration information) and emotions of the user via the avatar 120, etc. In other words, the control device 400 according to the present disclosure can provide information in accordance with the consideration information and emotions of the user.

[0486] 40 is a diagram schematically illustrating the functional configuration of a control device 400 that controls the avatar 120. The control device 400 is configured by a control unit having a sensor unit 210, a sensor module unit 220, a storage unit 230, a state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, and a communication processing unit 280.

[0487] The control object 252 includes a display device, a speaker, and the like. The display device displays the avatar 120 itself and images related to the avatar 120. The speaker outputs sounds related to the conversation of the avatar 120 and the actions of the avatar 120. The display device may also display the content of the conversation with the user as text. The gestures and facial expressions of the avatar 120 are examples of the attitude of the avatar 120.

[0488] The sensor unit 210 includes a microphone 211, a 3D depth sensor 212, a 2D camera 213, a distance sensor 214, an acceleration sensor 215, a thermosensor 216, and a touch sensor 217. The acceleration sensor 215 is, for example, a gyro sensor, and detects the acceleration of the control device 400. The thermosensor 216 detects the ambient temperature of the control device 400. The touch sensor 217 is a sensor that detects a touch operation by the user, and is disposed, for example, on a display device connected to the control device 400. Note that the sensor unit 210 may also include a clock, a sensor for motor feedback, etc.

[0489] The storage unit 230 includes reaction rules 231 and history data 232 .

[0490] The speech understanding unit 222 analyzes the user's voice detected by the microphone 211 and outputs text information representing the content of the user's speech. For example, the speech understanding unit 222 outputs text information representing the content of various considerations made by the user.

[0491] The facial expression recognition unit 223 recognizes the facial expression and emotions of the user from the image of the user captured by the 2D camera 213. For example, the facial expression recognition unit 223 recognizes the facial expression and emotions of the user based on the shapes, positional relationships, etc. of the eyes and mouth. For example, the facial expression recognition unit 223 recognizes the facial expression and emotions of the user during a conversation with the user.

[0492] The state recognition unit 240 recognizes the state of the user based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 mainly performs processing related to perception using the analysis results of the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is a 90% chance that the user is still anxious" or "There is a 50% chance that the user has organized their thoughts," and performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "The user is still anxious and appears to require a more digestible explanation."

[0493] The behavior recognition unit 244 recognizes the user's behavior based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized user's state are input into a pre-trained neural network to obtain the probability of each of a plurality of predetermined behavior categories (e.g., "laughing," "angry," "asking a question," "sad"), and the behavior category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 244 recognizes user behavior such as "not sure whether to speak up," "speaking up," and "asking a question" at a government office, a counseling center, etc.

[0494] The behavior determination unit 246 according to this embodiment determines, as a behavior corresponding to a user behavior, a behavior of the avatar 120 that corresponds to the reaction rule 231 when a combination including at least one of the user's past emotion value and current emotion value, the emotion value of the avatar 120, and the user behavior satisfy the conditions of a preset reaction rule. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, the behavior determination unit 246 determines, as a behavior corresponding to the user behavior, a behavior that changes the user's emotion value to a positive value.

[0495] For example, the reaction rules 231 define the behavior of the avatar 120 corresponding to behavioral patterns such as when the user is unsure whether to speak up, when the user speaks up, when the user asks a question, and so on.

[0496] The behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 246. Specifically, the behavior control unit 250 controls the movement of the avatar 120 and the operation of the control target related to the movement of the avatar 120 based on the behavior of the avatar 120 associated with the reaction rule 231 determined by the behavior determination unit 246. For example, in a case where the avatar 120 and the user are having a conversation, if the behavior determination unit 246 determines an action that includes speaking, the behavior control unit 250 controls the avatar 120 to take a predetermined movement and causes the speaker included in the control target 252 to output the voice of the avatar 120. At this time, the behavior control unit 250 may determine the speech rate of the voice based on the emotion value of the avatar 120. For example, the behavior control unit 250 determines a faster speech rate as the emotion value of the avatar 120 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 246 based on the emotion value determined by the emotion determination unit 242. For example, when the user is unsure whether to speak up, the behavior control unit 250 controls the avatar 120 to utter an utterance such as "Can I help you?". Furthermore, when the user speaks up, the behavior control unit 250 controls the avatar 120 to utter an utterance such as "What do you need?" to find out the user's needs. Furthermore, when the user asks a question, the behavior control unit 250 controls the avatar 120 to utter an utterance such as "In that case, we can handle the procedure for you," thereby guiding the user. Furthermore, when the user's emotion is a negative emotion such as "sorrow," "discomfort," "anxiety," "sorrow," "worry," or "emptiness," the behavior control unit 250 controls the avatar 120 to utter an utterance such as "It's okay" or "Leave it to us," which alleviates the user's negative emotion.

[0497] The communication processing unit 280 is responsible for communication with the server 300. The communication processing unit 280 can send and receive information to and from the linked device 500.

[0498] 41 is a diagram schematically illustrating an example of an operational flow relating to the operation of the control device 400 to determine the behavior of the avatar 120. The operational flow shown in FIG. 41 is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 220 has been input.

[0499] First, in step S1600, the state recognition unit 240 recognizes the state of the user based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is a 90% chance that the user is still anxious" or "There is a 50% chance that the user has organized their thoughts," and performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "The user is still anxious and appears to require a more digestible explanation."

[0500] In step S1601 , the emotion determining unit 242 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognizing unit 240 .

[0501] In step S1602, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 120 based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user to the history data 232.

[0502] In step S1603, the behavior recognition unit 244 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognition unit 240. For example, the behavior recognition unit 244 recognizes user behavior such as "not sure whether to speak up," "speaking up," and "asking a question" at a government office, a counseling center, etc.

[0503] In step S1604, the behavior determination unit 246 determines the behavior of the avatar 120 that corresponds to the reaction rule 231 if a combination including at least one of the user's current emotional value determined in step S1601 and the past emotional values ​​included in the history data 232, the emotional value of the avatar 120, and the user's behavior recognized by the behavior recognition unit 244 satisfy the conditions of the preset reaction rule.

[0504] In step S1605, the behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 246. For example, the behavior control unit 250 controls the avatar 120 to take an action when the user is unsure whether to speak to the user, to find out requirements from the user, to respond to the user, or to guide the user to a counter where procedures can be carried out.

[0505] In step S1606, the memory control unit 248 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 246 and the emotion value of the avatar 120 determined by the emotion determination unit 242.

[0506] In step S1607, the storage control unit 248 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the user's behavior in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S1608.

[0507] In step S1608, the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the user's status recognized by the status recognition unit 240 are stored in the history data 232.

[0508] As described above, the control device 400 includes a control unit that recognizes the user's consideration information, determines the user's emotion, and controls the avatar 120 to converse with the user based on the determined emotion and provide the user with information corresponding to the recognized consideration information. In this way, the control device 400 can use the avatar 120 to provide information according to the user's consideration information and emotion.

[0509] The consideration information is information about divorce, and the control device 400 provides the user with at least one of information about laws and administrative procedures related to divorce proceedings. This allows the control device 400 to use the avatar 120 to provide information about laws and the like related to divorce proceedings while being sensitive to the feelings of the user who is considering divorce.

[0510] The review information is information on at least one of legal issues and concerns, and the control device 400 provides the user with at least one of information on legal experts corresponding to the information on the legal issues or concerns and information on legal procedures. This allows the control device 400 to use the avatar 120 to provide advice on the legal issues or concerns, introduce the user to an appropriate legal expert, and provide information on legal procedures, while being sensitive to the feelings of the user who has legal issues or concerns.

[0511] The consideration information is information about procedures at the local government, and the control device 400 provides the user with at least one of information about the preparation of documents for the procedures and information about how to proceed with the procedures. This allows the control device 400 to use the avatar 120 to provide information and advice about procedures at the local government, as well as support for preparing the necessary documents and how to proceed with the procedures, while being sensitive to the feelings of the user who is considering the procedures at the local government.

[0512] The consideration information is information about consumers, and the control device 400 provides the user with at least one of information about laws and regulations related to consumer rights. This allows the control device 400 to use the avatar 120 to provide information and advice about laws and regulations related to consumer rights, support for dealing with consumer problems and complaints, and the like, while empathizing with the feelings of the user who is experiencing consumer problems, etc.

[0513] The consideration information is information about the debt, and the control device 400 holds a conversation to provide the user with mental support and provides the user with at least one of information about laws related to the debt, information about administrative procedures related to the debt, and information about experts according to the information about the debt. In this way, the control device 400 can use the avatar 120 to empathize with the feelings of the user who is in debt, suggest ways to deal with the debt, and provide information about laws and procedures related to debt problems and referrals to experts.

[0514] The consideration information is information related to parental authority, and the control device 400 holds a conversation to support the user's position and provides the user with at least one of information on laws related to parental authority and information on administrative procedures related to parental authority. As a result, the control device 400 can use the avatar 120 to empathize with the feelings of the user who is worried about parental authority, give advice that takes into consideration the user's position and feelings, and provide information on laws and procedures related to parental authority.

[0515] Seventeenth Embodiment Next, a seventeenth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a control system 1 according to this embodiment is shown schematically in FIG. 39.

[0516] The appearance of the avatar may be a human figure, as in avatars 120a to 120c, or a stuffed toy character, as in avatar 120d. For example, avatars 120a to 120c may be disguised as psychosomatic physicians or trainers, and may provide counseling by nodding and listening to what the real psychosomatic physicians or trainers say, thereby providing various information, such as information on emotional management and training, to athletes or users on a diet.

[0517] In the present disclosure, examples will be described in which appropriate information and advice are provided to a user who is undergoing training as various actions. Note that training includes mental training, physical training, dietary therapy, and the like.

[0518] Specifically, the control device 400 is placed in a training facility, the user's home, or the like, and the avatar 120 converses with the user and provides various information through conversation, gestures, image displays, etc. depending on the training situation. The control device 400 may also be linked to linked devices 500 such as a thermometer, a blood pressure monitor, a smartwatch, a body composition monitor, and training equipment.

[0519] For example, the control device 400 recognizes the behavior of a user who is an athlete and determines the user's emotions. The control device 400 displays and provides situations that put the user under pressure, such as footage of the opening ceremony or entrance of a tournament, or a tournament bracket for a tournament the user is scheduled to participate in, on a display. The avatar 120 appears, for example, as a person modeled after a psychosomatic physician, and provides the user with advice on how to deal with pressure by uttering, for example, "Take a deep breath," based on the behavior recognized and emotions determined by the control device 400. The avatar 120 also appears, for example, as a person modeled after a trainer, and provides a method for increasing self-confidence by uttering, for example, "Don't worry, you will definitely succeed," based on the behavior recognized and emotions determined by the control device 400. At this time, the avatar 120 reproduces facial expressions, hand movements, and the like, as in an actual medical examination or training session. In other words, the control device 400 provides the user with at least one piece of training-related information, such as information on emotional management and information on mental training, via the avatar 120, etc.

[0520] The control device 400 also recognizes the behavior of a user who is dieting and determines the user's emotions. For example, the control device 400 recognizes the user's diet goal through conversation with the avatar 120 and recognizes the user's body shape, weight, body fat percentage, etc. The avatar 120, for example, appears as a person imitating a trainer and, based on the behavior recognized and the emotion determined by the control device 400, utters, for example, "We will provide you with a reasonable diet plan," and provides meal plans, exercise programs, etc. by displaying them on a display. The avatar 120 also appears as a person imitating a trainer and, based on the behavior recognized and the emotion determined by the control device 400, utters, for example, "You're doing well" or "Let's try a little harder" at predetermined times, such as 25%, 50%, and 75% of the diet plan, thereby supporting the user in maintaining motivation and overcoming setbacks. In other words, the control device 400 provides an effective approach to successful dieting via the avatar 120, etc. That is, the control device 400 provides at least one of information related to diet and information related to exercise as information related to training via the avatar 120 or the like.

[0521] The control device 400 also recognizes the behavior of a user training at home (e.g., self-training) and determines the user's emotions. For example, the control device 400 recognizes the user's body type and muscle strength (e.g., acquired in cooperation with training equipment). The avatar 120 appears, for example, as a person imitating a trainer, and based on the behavior recognized and the emotion determined by the control device 400, utters, for example, "Try to imitate the correct form," and demonstrates the correct form with expressive gestures or displays a video of the correct form on a display. The avatar 120 also appears, for example, as a person imitating a trainer, and based on the behavior recognized and the emotion determined by the control device 400, praises the user by uttering, for example, "You're doing a great job," when the control device 400 recognizes that the user is training with the correct form. Furthermore, the avatar 120 appears as, for example, a person modeled after a trainer, and praises the user's change in physique by saying something like, "You've gained muscle," after a predetermined period of time, such as one week or one month, based on the behavior recognized and the emotion determined by the control device 400. In other words, the control device 400 provides, via the avatar 120, etc., information about a training method suited to at least one of the user's physique and muscle strength, as information about training.

[0522] In this manner, in the present disclosure, the control device 400 can provide appropriate information and advice to a user performing training via the avatar 120, etc. In other words, the control device 400 according to the present disclosure can provide information according to the training situation.

[0523] The functional configuration of the control device 400 that controls the avatar 120 is shown schematically in FIG.

[0524] The speech understanding unit 222 analyzes the user's voice detected by the microphone 211 and outputs text information representing the content of the user's speech. For example, the speech understanding unit 222 outputs text information representing the content of a conversation between the avatar 120 and the user.

[0525] The state recognition unit 240 recognizes the state of the user based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 mainly performs processing related to perception using the analysis results of the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is a 50% chance that the user is overcoming pressure" or "There is a 60% chance that the user is dissatisfied with the meal plan," and performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "The training goal has not been reached, and the user's motivation is declining."

[0526] The behavior recognition unit 244 recognizes the user's behavior based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized user's state are input into a pre-trained neural network to obtain the probability of each of a plurality of predetermined behavior categories (e.g., "laughing," "angry," "asking a question," and "sad"), and the behavior category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 244 recognizes user behavior such as "taking a deep breath," "measuring weight and body fat percentage," "training," etc. at a training facility, the user's home, etc.

[0527] For example, the reaction rules 231 define the behavior of the avatar 120 corresponding to behavioral patterns such as when the user is taking a deep breath, when the user is measuring their weight and body fat percentage, when the user is training, etc.

[0528] The behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 246. Specifically, the behavior control unit 250 controls the movement of the avatar 120 and the operation of the control target related to the movement of the avatar 120 based on the behavior of the avatar 120 associated with the reaction rule 231 determined by the behavior determination unit 246. For example, in a case where the avatar 120 and the user are having a conversation, if the behavior determination unit 246 determines an action that includes speaking, the behavior control unit 250 controls the avatar 120 to take a predetermined movement and causes the speaker included in the control target 252 to output the voice of the avatar 120. At this time, the behavior control unit 250 may determine the speech rate of the voice based on the emotion value of the avatar 120. For example, the behavior control unit 250 determines a faster speech rate as the emotion value of the avatar 120 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 246 based on the emotion value determined by the emotion determination unit 242. For example, when the user is taking a deep breath, the behavior control unit 250 controls the avatar 120 to utter utterances such as "Are you calm?" or "Don't worry, you'll definitely succeed." Furthermore, when the user is measuring their weight and body fat percentage, the behavior control unit 250 controls the avatar 120 to utter utterances such as "We'll provide you with a reasonable diet plan," and provides a meal plan, exercise program, etc. on the display. Furthermore, when the user is measuring their weight and body fat percentage at a predetermined timing during the diet plan period, the behavior control unit 250 controls the avatar 120 to utter utterances such as "You're doing well" or "Let's try a little harder." Furthermore, when the user is doing physical training, the behavior control unit 250 controls the avatar 120 to make utterances such as "Try to imitate the correct form," and also demonstrates the correct form through gestures and provides an image of the correct form on the display. Furthermore, when the user is doing physical training, the behavior control unit 250 praises the user by controlling the avatar 120 to make utterances such as "You're doing a great job."Furthermore, when a predetermined period of time has elapsed since the start of training and the user is performing physical training, the behavior control unit 250 controls the avatar 120 to make utterances such as "You've gained muscle," thereby praising the user. Furthermore, when the user is feeling negative emotions such as "sorrow," "discomfort," "anxiety," "grief," "worry," or "emptiness," the behavior control unit 250 controls the avatar 120 to make utterances such as "Just a little more!" or "Don't give up on yourself!" to ease the user's negative emotions and encourage them.

[0529] An example of an operational flow relating to the operation of determining the behavior of the avatar 120 in the control device 400 is shown schematically in Fig. 41. Here, processing that differs from the sixteenth embodiment will be described, and a description of processing that is the same as in the sixteenth embodiment will be omitted.

[0530] First, in step S1600, the state recognition unit 240 recognizes the state of the user based on the information analyzed by the sensor module unit 220. For example, the state recognition unit 240 generates perceptual information such as "There is a 50% chance that the user is overcoming pressure" or "There is a 60% chance that the user is dissatisfied with the meal plan," and performs processing to understand the meaning of the generated perceptual information. For example, the state recognition unit 240 generates semantic information such as "The training goal has not been reached, and the user's motivation is declining."

[0531] In step S1603, the behavior recognition unit 244 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 220 and the user's state recognized by the state recognition unit 240. For example, the behavior recognition unit 244 recognizes user behavior such as "taking deep breaths," "measuring weight and body fat percentage," and "training" at a training facility, the user's home, etc.

[0532] In step S1605, the behavior control unit 250 controls the avatar 120 and the control target 252 based on the behavior of the avatar 120 determined by the behavior determination unit 246. For example, the behavior control unit 250 controls the avatar 120 to perform actions such as when the user is taking a deep breath or when measuring weight and body fat percentage, demonstrate correct form using gestures, display an image of correct form on a display, or praise the user.

[0533] As described above, the control device 400 includes a control unit that recognizes the behavior of the user performing training, determines the user's emotions, and controls the avatar 120 to provide the user with information related to the training based on the recognized behavior and the determined emotions. This allows the control device 400 to provide information according to the training situation using the avatar 120.

[0534] Furthermore, the control device 400 provides the user, who is an athlete, with at least one of information on emotion management and information on mental training, thereby enabling the control device 400 to support the improvement of athletic ability using the avatar 120.

[0535] Furthermore, the control device 400 provides at least one of information about diet and information about exercise to a user who is undergoing training related to a diet, thereby enabling the control device 400 to provide an effective approach to succeeding in dieting using the avatar 120.

[0536] The control device 400 also provides the user with information about a training method suited to at least one of the user's body type and muscle strength, allowing the control device 400 to demonstrate correct form through gestures and praise the user using the avatar 120.

[0537] (Eighteenth embodiment) Next, an eighteenth embodiment will be described. Here, differences from the above-described embodiments will be mainly described, and a description of the same configurations and processes as the above-described embodiments will be omitted. An example of a control system 1 according to this embodiment is shown schematically in FIG. 39.

[0538] The appearance of the avatar may be a human figure, as in avatars 120a to 120c, or a stuffed toy character, as in avatar 120d. For example, avatars 120a to 120c may be disguised as the child or grandchild of an elderly user, and may nod and listen to the elderly user's story as if they were a real child or grandchild, thereby providing mental care, support for daily life, health advice, and the like to the elderly user through conversation.

[0539] The linked device 500 is communicably connected to the control device 400 via the communication network 20, and transmits and receives information to and from the control device 400. With this configuration, the linked device 500, in accordance with instructions from the control device 400, controls itself, converses with the user, provides mental care for the user through conversation, supports daily life, and gives health management advice.

[0540] In the present disclosure, examples will be described in which various actions are performed on an elderly user, such as providing mental care through conversation, support for daily life, and advice on health management.

[0541] Specifically, the control device 400 is placed in a care facility, a hospital, or the like, and performs actions such as providing care according to the user's mental state (mental state), supporting daily life, and giving health management advice through conversation with the user and vital signs, etc., using the avatar 120. The control device 400 may also be linked to linked devices 500 such as a thermometer, a blood pressure monitor, a smartwatch, a body composition monitor, and training equipment.

[0542] For example, the avatar 120 may appear as a person modeled after the user's (elderly) child or grandchild, and may utter utterances such as "Let's exercise" or "Let's watch a relaxing video" depending on the user's mental state, thereby performing an action to care for the user's mental state. At this time, the control device 400 performs the care based on the content of the conversation by the avatar 120 and the emotions (of the user / themselves). For example, if the user's emotions are positive, such as "joy," "pleasure," or "excitement," the control device 400 may utter "Let's exercise" through the avatar 120 and perform an action to encourage exercise in cooperation with training equipment. On the other hand, if the user's emotions are negative, such as "sorrow," "discomfort," "anxiety," "grief," "worry," or "emp...

Claims

1. A behavior control system comprising: a behavior determination unit that recognizes the behavior of an artist and determines the behavior of an avatar that corresponds to the recognized behavior of the artist; and a behavior control unit that controls the avatar based on the behavior of the avatar determined by the behavior determination unit.

2. The behavior control system according to claim 1, wherein the behavior control unit determines behavior related to the artist's performance.

3. The behavior control system according to claim 1, wherein the behavior control unit determines behavior related to the artist's creativity.

4. The behavior control system according to claim 1, wherein the behavior control unit determines behavior related to the artist's expressiveness.

5. The behavior control system of claim 1, wherein the behavior determination unit determines the behavior of the avatar corresponding to a predetermined reaction rule when a combination including at least one of the user's past emotion value and current emotion value, the emotion value of the avatar, and the user's behavior satisfy the conditions of the reaction rule.

6. The behavior control system according to claim 5, wherein the behavior control unit controls the movement of the avatar and the behavior of a control object related to the movement of the avatar based on the behavior of the avatar that corresponds to the reaction rule determined by the behavior determination unit.

7. A behavior control system as described in any one of claims 1 to 6, wherein a control device that controls the avatar is connected wirelessly or wired to a specified wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality.

8. A behavior control system according to any one of claims 1 to 6, wherein a control device that controls the avatar is mounted on a specific robot, or is connected wirelessly or by wire to a controlled device mounted on the robot.

9. A behavior control system including: a detection unit that detects a user's learning status in a specified learning field; and an output control unit that controls a control device equipped with a sentence generation model to output information corresponding to the learning status to the user via an avatar.

10. A behavior control system including: a detection unit that detects the occurrence of an event related to a user's circumstances that causes negative emotions in the user; and an output control unit that outputs information corresponding to the event whose occurrence is detected by the detection unit to the user via an avatar controlled using a sentence generation model.

11. A behavior control system comprising: a behavior determination unit that determines an avatar's behavior corresponding to situation information based on an emotion value of an avatar and an emotion value of a user corresponding to situation information indicating the situation of a user including family members; and a behavior control unit that controls the avatar based on the avatar's behavior determined by the behavior determination unit.

12. A control device for controlling an avatar, comprising: an estimation unit for estimating emotions of a user communicating with the avatar; an acquisition unit for acquiring health and medical information of at least one of the user and the user's family; and an avatar control unit for controlling the avatar to provide assistance in accordance with the emotions of the user estimated by the estimation unit based on the health and medical information of at least one of the user and the user's family acquired by the acquisition unit.

13. A behavior control system comprising: a behavior determination unit that determines an avatar's behavior corresponding to a user's behavior based on a user's emotion value and an avatar's emotion value; and a behavior control unit that controls the avatar based on the avatar's behavior determined by the behavior determination unit.

14. A behavior control system comprising: a collection unit that collects situational information indicating the situation of a user while viewing a work of art; and an output control unit that controls an avatar controlled by a control device equipped with a sentence generation model to perform behavior in accordance with the situational information collected by the collection unit.

15. A behavior control system including: a recognition unit that recognizes a state of a user based on detection information detected from the user; and an output control unit that causes an avatar controlled by a control device equipped with a sentence generation model to output a response to the user in accordance with the state of the user recognized by the recognition unit.

16. A behavior control system including: an evaluation unit that evaluates a user's state based on user information about the user; and an output control unit that outputs proposal information regarding proposals for the user's state from an avatar controlled by a control device equipped with a sentence generation model to a predetermined output destination in accordance with the evaluation by the evaluation unit.

17. A behavior control system including: a detection unit that detects the occurrence of an event related to a user's circumstances; and an output control unit that outputs information corresponding to the event whose occurrence is detected by the detection unit to the user via an avatar controlled using a sentence generation model.

18. A behavior control system including: a detection unit that detects a user's learning status in a specified language; and an output control unit that controls a control device equipped with a sentence generation model to output information corresponding to the learning status to the user via an avatar.

19. A behavior control system comprising: a collection unit that collects situational information indicating a user's situation; and an output control unit that controls an avatar controlled by a control device equipped with a sentence generation model to make suggestions to the user regarding coordination based on the situational information collected by the collection unit.

20. A behavior control system including: a recognition unit that recognizes the emotions of a user based on detection information detected from the user; and an output control unit that outputs information regarding the insurance that the user subscribes to to the user from an avatar controlled by a control device equipped with a sentence generation model, in accordance with the emotion recognized by the recognition unit.

21. A behavior control system including: a detection unit that detects the occurrence of a specified event; and an output control unit that controls an avatar controlled by a control device equipped with a sentence generation model to output information corresponding to the event detected by the detection unit to a user.

22. A control device for controlling an avatar, comprising: a control unit that recognizes at least one of the state of a user who is a member of a family and the state of a pet kept by the user, determines an action of the avatar corresponding to the recognized state, and controls the avatar based on the determined action of the avatar.

23. An electronic device comprising: a control unit that recognizes a user's consideration information and determines the user's emotions, and controls an avatar to converse with the user based on the determined emotions and provide the user with information corresponding to the recognized consideration information.

24. An electronic device comprising: a control unit that recognizes the behavior of a user performing training and determines the user's emotions, and controls an avatar to provide the user with information related to the training based on the recognized behavior and the determined emotions.

25. An electronic device comprising: a control unit that recognizes a user's mental state, determines the user's emotions, and controls an avatar to converse with the user based on the determined emotions and provide care in accordance with the recognized mental state.

26. An avatar display device comprising: a control unit that analyzes the emotions of a user and other users, determines an avatar behavior corresponding to the emotions of the user and the emotions of the other users, and controls a control object based on the determined avatar behavior.

27. A control device for controlling an avatar, comprising: a recognition unit that recognizes a child's state; an emotion estimation unit that estimates the child's emotion based on the child's state recognized by the recognition unit; and an avatar control unit that controls the avatar to provide assistance according to the child's emotion estimated by the emotion estimation unit.

Images