Action control system

Abstract

To enable provision of appropriate support according to the condition of users including family members.SOLUTION: An action control system disclosed herein comprises an action determination unit configured to determine an action to be taken by an avatar in accordance with condition information on the basis of an emotion value of a user corresponding to the condition information representing the condition of users including family members and an emotion value of the avatar, and an action control unit configured to control the avatar in accordance with the action to be taken by the avatar determined by the action determination unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a behavior control system. [Background technology]

[0002] Patent Document 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot executes a specific action, and if the robot is unable to determine an action for the recognized user's reaction, it updates the robot's behavior by receiving information about an action suitable for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has room for improvement in terms of providing appropriate support according to the situation of the user, including the family members. [Means for solving the problem]

[0005] According to a first aspect of the present invention, there is provided a behavior control system comprising: a behavior determination unit that determines an avatar's behavior corresponding to situation information based on an emotion value of the user and an emotion value of an avatar corresponding to situation information indicating a situation of the user including a family member, and a behavior control unit that controls the avatar based on the avatar's behavior determined by the behavior determination unit. Effect of the Invention

[0006] According to one aspect of the embodiment, appropriate support can be provided according to the situation of the user, including the family members. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 shows a schematic diagram of an example of a system 5 according to the present embodiment. [Diagram 2] FIG. 2 shows a schematic functional configuration of a control device 200 that controls the avatar 100. As shown in FIG. [Diagram 3] FIG. 3 shows a schematic diagram of the data structure of the character data 233. [Figure 4] FIG. 4 shows an example of an operational flow for setting a character. [Diagram 5] FIG. 5 shows an example of an operational flow for determining the behavior of avatar 100. In FIG. [Figure 6] FIG. 6 shows an outline of the functional configuration of the output unit 290. [Figure 7] FIG. 7 shows an example of an operation flow of the output unit 290. In FIG. [Figure 8] FIG. 8 illustrates an example of a hardware configuration of a computer 1200 that functions as the control device 200 and the server 300. [Figure 9] FIG. 9 is a diagram showing an emotion map 400 onto which multiple emotions are mapped. [Figure 10] FIG. 10 is a diagram showing another example of the emotion map. [Figure 11] FIG. 11 is a diagram illustrating an example of an emotion table. [Figure 12] FIG. 12 is a diagram illustrating an example of an emotion table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] 1 is a schematic diagram illustrating an example of a system 5 according to the present embodiment. The system 5 includes a control device 200 that controls avatars such as avatar 100, avatar 101, and avatar 102, and a server 300. The control device 200 that controls the avatar 100 is connected wirelessly or wired to a display device such as a PC monitor or a smartphone display, or to a predetermined wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality.

[0010] The user 10a, the user 10b, the user 10c, and the user 10d are users who communicate with the avatar 100. The user 11a, the user 11b, and the user 11c are users who communicate with the avatar 101. The user 12a and the user 12b are users who communicate with the avatar 102. In the description of this embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The avatar 101 and the control device 201, and the avatar 102 and the control device 202 have substantially the same functions as the avatar 100 and the control device 200. Therefore, the system 5 will be described by mainly focusing on the functions of the avatar 100.

[0011] As shown in FIG. 1, avatars 100-102 controlled by control devices 200-202 communicate with users 10-12 based on various forms. For example, as shown in FIG. 1(1), the avatar 100 is expressed as a person or the like displayed on a display device or the like, and communicates with the user 10 via an input / output device provided in the control device 200. Also, as shown in FIG. 1(2), the avatar 101 communicates with the user 11 displayed as a person or character on a display device or the like in a virtual space. Also, as shown in FIG. 1(3), the avatar 102 can be displayed as a character rather than a person, and communicates with the user 12 as the character. Note that, as for the above-mentioned display device or the like, a display monitor has been described in FIG. 1 as an example, but is not limited thereto. For example, in this embodiment, the display device may include a monitor, a display unit of a notebook PC or a mobile terminal, and a wearable terminal device related to augmented reality, virtual reality, mixed reality, etc., such as VR / AR goggles, VR / AR glasses, and other wearable devices that provide information perceived by the five senses of the user.

[0012] The appearance of the avatar may be a human figure like avatar 100 and avatar 101, or a character like avatar 102. For example, avatars 100 and 101 are avatars dressed as hosts or announcers who narrate a memorial video about the bride and groom (users) at a wedding. For example, avatar 102 is considered to be easily familiar to children in particular because it has a character appearance.

[0013] The avatar 100 is controlled by the control device 200, and converses with the user 10 and provides images to the user 10 via an input / output device provided in the control device 200. At this time, the control device 200 cooperates with a server 300 or the like that can communicate with the control device 200 via the communication network 20 to conduct a conversation between the avatar 100 and the user 10 and provide images or the like to the user 10. For example, the control device 200 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn so as to be able to have a more appropriate conversation with the user 10. In addition, the control device 200 causes the server 300 to record the captured image data of the user 10, and requests the server 300 for the image data or the like as necessary, and provides the image data or the like to the user 10.

[0014] The control device 200 also stores emotion values ​​that represent the types of emotions of the avatar 100 itself. For example, the control device 200 stores emotion values ​​that represent the strength of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral" as emotion values ​​of the avatar 100. For example, when the user 10 and the avatar 100 converse in a state where the emotion value of excitement is high, the control device 200 controls the avatar 100 so that the avatar 100 speaks at a fast speed. In this way, the avatar 100 can express its own emotions through its actions under the control of the control device 200.

[0015] Furthermore, the control device 200 may be configured to determine the behavior of the avatar 100 corresponding to the emotion of the user 10 by matching a sentence generation model (so-called an AI (Artificial Intelligence) chat engine) with an emotion engine. Specifically, the control device 200 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the behavior of the user, and determine the behavior of the avatar 100 corresponding to the determined emotion.

[0016] More specifically, when the control device 200 recognizes the behavior of the user 10, the control device 200 automatically generates behavioral content that the avatar 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is publicly known as disclosed in, for example, JP 2018-081444 A, and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured by a large-scale language model (LLM: Large Language Model). As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 or the avatar 100 and various linguistic information in the behavior of the avatar 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model and an emotion engine.

[0017] Here, each control device has an output function for outputting information according to the social situation of the user through each avatar. For example, each control device can output information on emotional support for a businessperson having a labor problem through the avatar controlled by the control device, thereby outputting appropriate information according to the user's situation.

[0018] The control device 200 also has a function of recognizing the behavior of the user 10. The control device 200 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The control device 200 determines the behavior to be performed by the avatar 100 based on the recognized behavior of the user 10, etc.

[0019] The control device 200 stores rules that define actions to be taken by the avatar 100 based on the emotions of the user 10, the emotions of the avatar 100 itself, and the actions of the user 10, and performs various actions in accordance with the rules.

[0020] Specifically, the control device 200 stores reaction rules for determining the behavior of the avatar 100 based on the emotion of the user 10, the emotion of the avatar 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the avatar 100 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the avatar 100 as "apologizing" when the behavior of the user 10 is "angry". The reaction rules also define the behavior of the avatar 100 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the avatar 100 as "calling out" when the behavior of the user 10 is "sad".

[0021] When the control device 200 recognizes the behavior of the user 10 as "angry" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be performed by the avatar 100. For example, when the avatar 100 selects the behavior of "apologizing", it performs the motion of "apologizing" and outputs a voice expressing the words "apologize".

[0022] In addition, when the emotion of avatar 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the state of user 10 is "alone and looks lonely" is met, it is determined that the emotion of avatar 100 will change to "worried" and the action of "calling out" can be executed.

[0023] When the control device 200 recognizes based on the reaction rule that the current emotion of the avatar 100 is "normal" and that the user 10 is alone and appears lonely, the control device 200 increases the emotion value of "sadness" of the avatar 100. Furthermore, the control device 200 selects the action of "calling out" defined in the reaction rule as the action to be performed by the avatar 100 toward the user 10. For example, when the control device 200 selects the action of "calling out", the control device 200 converts the words "What's wrong?", which express concern, into a worried voice of the avatar 100 and outputs it.

[0024] Furthermore, the control device 200 transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the avatar 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.

[0025] The control device 200 stores user reaction information regarding the user's reaction to the avatar 100. The server 300 receives and stores user reaction information not only from the control device 200 that controls the avatar 100, but also from the control devices 201 and 202 that control the avatar 101 and 102. The server 300 analyzes the user reaction information from the control devices 200, 201, and 202, and updates the reaction rules.

[0026] The control device 200 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The control device 200 incorporates the updated reaction rules into the reaction rules stored in the control device 200. This allows the control device 200 to incorporate the reaction rules acquired by the control device 201 and the control device 202 of the avatar 101, the avatar 102, etc. into the reaction rules of the control device 200 itself.

[0027] 2 shows a schematic functional configuration of a control device 200 that controls the avatar 100. The control device 200 includes a sensor unit 210, a sensor module unit 220, a storage unit 230, a user state recognition unit 240, an emotion determination unit 242, a behavior recognition unit 244, a behavior determination unit 246, a memory control unit 248, a behavior control unit 250, a control target 252, a communication processing unit 280, and an output unit 290.

[0028] The controlled object 252 includes a display device 2521 and a speaker 2522. The display device 2521 displays the avatar 100 itself and images related to the avatar 100. The speaker 2522 outputs sounds related to the conversation of the avatar 100 and the actions of the avatar 100.

[0029] The sensor unit 210 includes a microphone 211, a 3D depth sensor 212, a 2D camera 213, and a distance sensor 214. The microphone 211 continuously detects sound and outputs sound data. The microphone 211 may be provided on a dummy head, a simulator, or the like, and may have a function of performing binaural recording. The 3D depth sensor 212 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 213 is an example of an image sensor. The 2D camera 213 captures an image with visible light and generates visible light video information. The distance sensor 214 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 210 may also include a clock, a gyro sensor, a touch sensor, and the like.

[0030] The storage unit 230 includes a reaction rule 231, history data 232, and character data 233. The history data 232 includes the emotion value and behavior history of the user 10 in the past. The emotion value and behavior history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 230 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. Note that, among the components of the control device 200 shown in FIG. 2, the functions of the components other than the control target 252, the sensor unit 210, and the storage unit 230 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program operating on the OS.

[0031] The character data 233 is data that associates a character with an age. For example, the character is a person who appears in existing content such as animation, video games, comics, movies, etc. Also, the character may be an animal or plant with a personality, or an inanimate object (such as a robot).

[0032] For example, the age (user age) associated with a character in the character data 233 is determined based on the age group of viewers who are assumed to be the target audience of the content in which the character appears. FIG. 3 shows a schematic data structure of the character data 233.

[0033] For example, character "A" appears in an animation for kindergarteners. In this case, as shown in FIG. 3, character "A" is associated with a user age of "3 to 7 years old."

[0034] For example, if a movie in which character "C" appears contains violent scenes and is not suitable for viewing by young children, the user age of "12 years and older" is associated with character "C" as shown in FIG.

[0035] The age in the character data 233 may be determined based on an age rating by a rating organization such as PEGI (Pan European Game Information), the Motion Picture Ethics Organization, or CERO (Computer Entertainment Rating Organization). The age of use may be determined by a range such as "3 to 5 years old" or "12 years old or older," or may be determined by a single value such as "10 years old" or "15 years old."

[0036] The sensor module unit 220 includes a voice emotion recognition unit 221, a speech understanding unit 222, a facial expression recognition unit 223, and a face recognition unit 224. Information detected by the sensor unit 210 is input to the sensor module unit 220. The sensor module unit 220 analyzes the information detected by the sensor unit 210, and outputs the analysis result to the user state recognition unit 240.

[0037] The voice emotion recognition unit 221 of the sensor module unit 220 analyzes the voice of the user 10 detected by the microphone 211 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 221 extracts feature amounts such as frequency components of the voice, and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 222 analyzes the voice of the user 10 detected by the microphone 211 and outputs text information representing the content of the utterance of the user 10.

[0038] The facial expression recognition unit 223 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 213. For example, the facial expression recognition unit 223 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0039] The face recognition unit 224 recognizes the face of the user 10. The face recognition unit 224 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 213.

[0040] The user state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220. For example, the user state recognition unit 240 mainly performs processing related to perception using the analysis results of the sensor module unit 220. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0041] The emotion determination unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0042] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.

[0043] Furthermore, the emotion determining unit 242 determines an emotion value indicating the emotion of the avatar 100 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognizing unit 240 .

[0044] The emotion value of the avatar 100 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "happiness."

[0045] Specifically, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100 in accordance with a rule for updating the emotion value of the avatar 100 that is determined in correspondence with the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognition unit 240.

[0046] For example, when the user state recognition unit 240 recognizes that the user 10 looks lonely, the emotion determination unit 242 increases the emotion value of "sadness" of the avatar 100. Also, when the user state recognition unit 240 recognizes that the user 10 is smiling, the emotion determination unit 242 increases the emotion value of "happy" of the avatar 100.

[0047] The emotion determination unit 242 may determine the emotion value indicating the emotion of the avatar 100 by further considering the state of the avatar 100. For example, the emotion determination unit 242 may increase the emotion value of "sadness" of the avatar 100 when the user 10 does not say anything to the avatar 100 or responds in a disorganized manner. Furthermore, the emotion determination unit 242 may increase the emotion value of "anger" when the user 10 continues to respond in a disorganized manner.

[0048] The behavior recognition unit 244 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognition unit 240. For example, the information analyzed by the sensor module unit 220 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing," "getting angry," "asking a question," and "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0049] As described above, in this embodiment, the control device 200 identifies the user 10 and then acquires the content of the user 10's speech. When acquiring and using the content of the speech, the control device 200 obtains the necessary consent in accordance with laws and regulations from the user 10, and the behavior control system for the avatar 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0050] The behavior determining unit 246 determines an avatar behavior corresponding to a user behavior based on the user emotion value and the avatar emotion value. The behavior determining unit 246 determines an action corresponding to a user behavior recognized by the behavior recognizing unit 244 based on, for example, the current emotion value of the user 10 determined by the emotion determining unit 242, history data 232 of past emotion values ​​determined by the emotion determining unit 242 before the current emotion value of the user 10 was determined, and the emotion value of the avatar 100. The action determined by the behavior determining unit 246 includes a gesture performed by the avatar 100 or the contents of the speech of the avatar 100.

[0051] When a combination including at least one of the past emotion value and the current emotion value of the user 10, the emotion value of the avatar 100, and the behavior of the user 10 satisfy the conditions of a preset reaction rule, the behavior determining unit 246 determines, as the behavior corresponding to the behavior of the user 10, the behavior of the avatar 100 corresponding to the reaction rule 231. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior determining unit 246 determines, as the behavior corresponding to the behavior of the user 10, the behavior for changing the emotion value of the user 10 to positive.

[0052] Furthermore, the behavior determining unit 246 determines the behavior of the avatar 100 corresponding to the reaction rule 231 when the situation information of the user 10 over time, the emotion value of the user 10 corresponding to the situation information, and the emotion value of the avatar 100 satisfy the conditions of the preset reaction rule 231. The situation information of the user 10 over time will be described later.

[0053] The behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 based on the emotion of the avatar 100. For example, when the avatar 100 is insulted by the user 10 or when the user 10 behaves arrogantly (i.e., when the user's reaction is poor), when the surrounding noise is loud and the voice of the user 10 cannot be detected, or when the emotion value of "anger" or "sorrow" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "anger" or "sorrow". Also, when the user's reaction is good or when the emotion value of "joy" or "pleasure" of the avatar 100 increases, the behavior determining unit 246 may determine a behavior corresponding to the behavior of the user 10 according to the increase in the emotion value of "joy" or "pleasure".

[0054] Furthermore, the behavior determining unit 246 may determine a behavior for a user 10 who has increased the emotional values ​​of "joy" or "pleasure" of the avatar 100 to be different from the behavior for the user 10 who has increased the emotional values ​​of "anger" or "sadness" of the avatar 100. In this way, the behavior determining unit 246 may determine a different behavior depending on the emotion of the avatar 100 itself or how the user 10 has changed the emotion of the avatar 100 due to the behavior of the user 10.

[0055] Reaction rule 231 defines a combination of the past emotion value and the current emotion value of user 10, the emotion value of avatar 100, and the behavior of avatar 100 according to the behavior of user 10. For example, when the past emotion value of user 10 is a positive value and the current emotion value is a negative value, and the behavior of user 10 is sad, a combination of gestures and speech content when asking a question to encourage user 10 with gestures is defined as the behavior of avatar 100.

[0056] For example, the reaction rule 231 defines the behavior of the avatar 100 for all combinations of emotion value patterns of the avatar 100 (1296 patterns, which are the fourth power of six values, i.e., "joy", "anger", "sorrow", and "pleasure", i.e., "0" to "5"); combination patterns of past emotion values ​​and current emotion values ​​of the user 10; and behavior patterns of the user 10. That is, for each emotion value pattern of the avatar 100, behavior of the avatar 100 according to the behavior pattern of the user 10 is defined for each of a plurality of combinations of past emotion values ​​and current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that the behavior determining unit 246 may determine the behavior of the avatar 100 using the history data 232 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0057] Note that reaction rules 231 may define at least one of a gesture and a statement as the behavior of avatar 100 for each of the emotion value patterns (1296 patterns) of avatar 100. Alternatively, reaction rules 231 may define at least one of a gesture and a statement as the behavior of avatar 100 for each group of emotion value patterns of avatar 100.

[0058] The strength of each gesture included in the behavior of avatar 100 defined in reaction rule 231 is determined in advance. The strength of each speech included in the behavior of avatar 100 defined in reaction rule 231 is determined in advance.

[0059] The memory control unit 248 determines whether or not to store data including the behavior of the user 10 in the history data 232 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 246 and the emotion value of the avatar 100 determined by the emotion determination unit 242.

[0060] Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the avatar 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 246, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 246, is equal to or greater than a threshold value, it is decided to store data including the behavior of the user 10 in the history data 232.

[0061] When the memory control unit 248 decides to store data including the behavior of the user 10 in the history data 232, it stores in the history data 232 the behavior determined by the behavior determination unit 246, information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 240 (e.g., the facial expression, emotions, etc. of the user 10).

[0062] The behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior of the avatar 100 determined by the behavior determination unit 246. Specifically, the behavior control unit 250 controls the movement of the avatar 100 and the movement of the control target related to the movement of the avatar 100 based on the behavior of the avatar 100 corresponding to the reaction rule 231 determined by the behavior determination unit 246. For example, in a case where the avatar 100 and the user 10 are having a conversation, when the behavior determination unit 246 determines an action including speaking, the behavior control unit 250 controls the avatar 100 to take a predetermined movement and causes the speaker 2522 included in the control target 252 to output the voice of the avatar 100. At this time, the behavior control unit 250 may determine the speech speed of the voice based on the emotion value of the avatar 100. For example, the behavior control unit 250 determines a faster speech speed as the emotion value of the avatar 100 is larger. In this way, the behavior control unit 250 determines the manner of execution of the behavior determined by the behavior determining unit 246 based on the emotion value determined by the emotion determining unit 242.

[0063] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 246. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. In addition, the change in emotion of the user 10 may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 210. When an impact is detected by the touch sensor included in the sensor unit 210, the emotion of the user 10 may be recognized as having worsened, and when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor included in the sensor unit 210, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0064] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determining unit 246 in the execution mode determined according to the emotion of the avatar 100, the emotion determining unit 242 further changes the emotion value of the avatar 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determining unit 242 increases the emotion value of "happiness" of the avatar 100 when the user's reaction to the behavior determined by the behavior determining unit 246 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Furthermore, the emotion determining unit 242 increases the emotion value of "sadness" of the avatar 100 when the user's reaction to the behavior determined by the behavior determining unit 246 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.

[0065] Furthermore, the behavior control unit 250 expresses the emotion of the avatar 100 based on the determined emotion value of the avatar 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the avatar 100, the behavior control unit 250 controls the movement of the avatar 100 to make the avatar 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the avatar 100, the behavior control unit 250 controls the movement of the avatar 100 to make the avatar 100 assume a drooping posture.

[0066] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 231.

[0067] The output unit 290 realizes the above-mentioned output function. The output unit 290 will be described in detail later.

[0068] The server 300 communicates between the server 300 and the control devices 200-202 that control the avatars 100-102, receives user reaction information transmitted from the avatar 100, and updates the reaction rules based on the reaction rules that include actions that have received positive reactions.

[0069] (Character-based action decisions) So far, a case has been described in which the behavior determining unit 246 determines the behavior of the avatar 100 based on the state recognized by the user state recognizing unit 240. On the other hand, the behavior determining unit 246 may determine the behavior of the avatar 100 based on the set character as well as the state of the user. In this case, the behavior determining unit 246 may obtain an age (user age) associated with the character from the character data 233, and determine the behavior of the avatar 100 based on the obtained user age.

[0070] That is, the behavior determining unit 246 determines the behavior of the avatar 100 based on the state recognized by the user state recognizing unit 240 and the set character or the age associated with the character. This makes it possible to cause the avatar 100 to perform an appropriate behavior according to the age of the user. In particular, it becomes possible to restrict the avatar 100 from performing actions that are inappropriate for young users (for example, outputting violent content).

[0071] Characters are set in advance in the system 5. The character settings are input as a prompt (command sentence). The prompt may be input via an input device provided in the control device 200, or via an external device such as a server communicably connected to the control device 200. In addition, the prompt may specify the name of the character, or may specify an ID defined for each character.

[0072] Also, for example, the behavior determining unit 246 determines the behavior of outputting information to the display device 2521 or the speaker 2522 (examples of an output device) provided in the control device 200 in a manner according to the user's age. For example, the behavior determining unit 246 changes the voice of the avatar 100 emitted from the speaker 2522 to the character's tone of voice.

[0073] Also, for example, the behavior determining unit 246 determines the behavior of outputting a voice or a message by using text composed of words according to the user's age. Here, it is assumed that usable words for each age are set in advance. The behavior determining unit 246 obtains the user's age from the character data 233.

[0074] For example, it is assumed that the words "What's wrong?" and "How is it going?" are stored in advance in the storage unit 230 as words to be output when the avatar 100 performs the action of "calling out" to the user 10. It is also assumed that the age "under 12 years old" is associated with "What's wrong?" and the age "12 years old or older" is associated with "How is it going?". For example, if the user age is "18 years old or older", the behavior decision unit 246 decides to output the words "How is it going?". Also, for example, if the user age is "3 to 7 years old", the behavior decision unit 246 decides to output the words "What's wrong?".

[0075] In this way, the control device 200 can increase familiarity with younger users by changing the tone of voice and the words output according to the age of the user, while restricting behavior that is particularly inappropriate for younger users.

[0076] Furthermore, the behavior determining unit 246 determines an action to output content corresponding to the character to an output device (such as the display device 2521). For example, the behavior determining unit 246 determines an action to display video content (such as a movie or animation) in which the character appears on the display device 2521.

[0077] The behavior determining unit 246 may also determine an action to output educational content according to the age of use. Here, the educational content is text, video, audio, etc. related to study subjects such as English, arithmetic, Japanese, science, and social studies. The educational content may also be interactive content in which the user inputs answers to questions. For example, the behavior determining unit 246 determines an action to display on the display device 2521 a text of a calculation problem corresponding to the grade according to the age of use. For example, if the age of use is "under 8 years old", the behavior determining unit 246 determines to display an addition problem, and if the age of use is "8 years old or older", the behavior determining unit 246 determines to display a multiplication problem.

[0078] The behavior determining unit 246 may determine the behavior of causing the output device to output content according to the age of use, rather than the character. In this case, the content may be content in which a character appears, or content that does not depend on a character, such as a commonly known folk tale or fairy tale.

[0079] In addition, content corresponding to the character and the grade and educational content corresponding to the age of the user may be pre-stored in the storage unit 230, or may be obtained from an external device such as a server connected to the control device 200 so as to be able to communicate with the control device 200.

[0080] Figure 4 shows an example of a flow of operations related to character setting. Note that "S" in the flow of operations indicates a step to be executed.

[0081] In step S50, the control device 200 accepts the setting of the character. Then, in step S51, the control device 200 outputs a screen corresponding to the character of the avatar 100 (for example, a screen displaying the character's appearance).

[0082] Furthermore, in step S52, the behavior determining unit 246 acquires from the character data 233 a usage age appropriate to the set character.

[0083] Fig. 5 shows an example of an outline of an operation flow relating to an operation for determining the behavior of the avatar 100. The operation flow shown in Fig. 5 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 220 is input. Note that "S" in the operation flow indicates a step to be executed.

[0084] First, in step S100 , the user state recognition unit 240 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 220 .

[0085] In step S102, the emotion determining unit 242 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognizing unit 240.

[0086] In step S103, the emotion determination unit 242 determines an emotion value indicating the emotion of the avatar 100, based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognition unit 240. The emotion determination unit 242 adds the determined emotion value of the user 10 to the history data 232.

[0087] In step S104 , the behavior recognition unit 244 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 220 and the state of the user 10 recognized by the user state recognition unit 240 .

[0088] In step S106, the behavior determination unit 246 determines the behavior of the avatar 100 based on the usage age acquired in step S52 of FIG. 4, a combination of the current emotion value of the user 10 determined in step S102 of FIG. 5 and the past emotion value included in the history data 232, the emotion value of the avatar 100, the behavior of the user 10 recognized by the behavior recognition unit 244, and the reaction rules 231.

[0089] In step S 108 , the behavior control unit 250 controls the avatar 100 and the control target 252 based on the behavior determined by the behavior determination unit 246 .

[0090] In step S110, the memory control unit 248 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determining unit 246 and the emotion value of the avatar 100 determined by the emotion determining unit 242.

[0091] In step S112, the storage control unit 248 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 232. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S113.

[0092] In step S113, the behavior determined by the behavior determination unit 246, the information analyzed by the sensor module unit 220 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 240 are stored in the history data 232.

[0093] As described above, according to the control device 200, an emotion value indicating the emotion of the avatar 100 is determined based on the user state, and whether or not to store data including the behavior of the user 10 in the history data 232 is determined based on the emotion value of the avatar 100. This makes it possible to reduce the capacity of the history data 232 that stores data including the behavior of the user 10. For example, when the control device 200 determines that the user state 10 years from now is the same as that 10 years ago, the control device 200 can present to the user 10 the state of the user 10 10 years ago (for example, the facial expression, emotion, etc. of the user 10), as well as all kinds of peripheral information such as data on the sound, image, smell, etc. of the place, via the avatar 100, the display device 2521, the speaker 2522, etc.

[0094] Moreover, according to the control device 200, the avatar 100 can be made to perform an appropriate action for the action of the user 10. Conventionally, the actions of the user are classified and the actions including the facial expression and the appearance of the avatar are determined. In contrast, the control device 200 determines the current emotion value of the user 10 and performs an action on the user 10 based on the past emotion value and the current emotion value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the control device 200 can make the avatar 100 speak such as "You were cheerful yesterday, but what's wrong with you today?" The control device 200 can also make the avatar 100 speak with gestures. For example, if the user 10 who was depressed yesterday is cheerful today, the control device 200 can make the avatar 100 speak such as "You were depressed yesterday, but you look cheerful today, don't you?" For example, if a user 10 who was in good spirits yesterday is in better spirits today than yesterday, the control device 200 can cause the avatar 100 to say, "You're in better spirits today than yesterday. Has anything better happened to you than yesterday?" For example, for a user 10 whose emotion value is equal to or greater than 0 and whose emotion value continues to fluctuate within a certain range, the control device 200 can cause the avatar 100 to say, "You've been feeling stable lately, which is good."

[0095] Also, for example, when the avatar 100 asks the user 10, "Did you finish the homework you told me about yesterday?" and the user 10 responds, "Yes, I did it," the control device 200 can cause the avatar 100 to utter a positive utterance such as "Great!" and make a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the control device 200 can cause the avatar 100 to utter a positive utterance such as "You did a good job!" and make the above-mentioned positive gesture. In this way, it is expected that the user 10 will feel a sense of closeness to the avatar 100 by the avatar 100 controlled by the control device 200 taking an action based on the state history of the user 10.

[0096] In the above embodiment, the control device 200 recognizes the user 10 using a facial image of the user 10, but the disclosed technology is not limited to this aspect. For example, the control device 200 may recognize the user 10 using a voice emitted by the user 10, an email address of the user 10, an SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

[0097] (Information output) The output unit 290 will be described in detail. Fig. 6 shows an outline of the functional configuration of the output unit 290. Here, the output unit 290 is provided in the control device 200, and causes information related to the avatar 100 to be output.

[0098] 6, the output unit 290 includes a detection unit 2901, a collection unit 2902, and an output control unit 2903. The output unit 290 also stores user information 2911.

[0099] Each component of the output unit 290 is realized by a CPU operating based on a program. For example, the functions of these components can be implemented as the operation of a CPU by an operating system (OS) and a program that runs on the OS. The user information 2911 is implemented by a storage medium such as a memory.

[0100] The detection unit 2901 detects the occurrence of a predetermined event. In this embodiment, such an event includes an event that requires support for recording and / or managing memorial items such as photos (images), videos, letters, and diaries that are irreplaceable memories of the family, an event that requires support for managing the family's schedule, and the like. The detection unit 2901 detects the user 10 at the user's home or at a place where the user 10 has gone out (for example, outdoors such as a park, an amusement park, the sea, and the mountains, or indoors such as a lodging facility such as an inn, a concert hall, and an aquarium). The user 10 includes a family member. The family member is a husband, a wife, a child, a husband's and / or a wife's parents (grandfathers of the child), and the like. The family member may include a pet, and the like. The output control unit 2903 controls the avatar 100 controlled by the control device 200 having a sentence generation model to output information corresponding to the event detected by the detection unit 2901 to the outside.

[0101] The collecting unit 2902 collects situation information indicating the situation of the user 10 including the family. The output control unit 2903 controls the avatar 100 to output information corresponding to the situation information. The collecting unit 2902 acquires the situation information of the user 10 over time. The situation information of the user 10 over time is situation information from a certain time in the past to a certain time after a predetermined period of time (for example, from a certain time in the past to the present). The situation information of the user 10 over time includes at least one of an image and a video of the user 10. For example, the situation information of the user 10 over time is information related to the growth and development of a child. The avatar 100 is capable of recognizing the emotions of the user 10 from the image, video, voice, etc. of the user 10.

[0102] The output control unit 2903 controls the avatar 100 to take an action corresponding to the situation information collected by the collection unit 2902. In this case, the output control unit 2903 outputs information corresponding to the situation information collected by the collection unit 2902 to the outside via the avatar 100. The output control unit 2903 controls the avatar 100 to take an action to be recorded for each selected situation information. Through such control, the avatar 100 records, for example, the growth and development of a child. That is, the avatar 100 provides appropriate support according to the situation of the user 10. This allows the user 10 to share, for example, the record of the growth and development of the child with his / her family.

[0103] Moreover, the output control unit 2903 controls the avatar 100 to take a photo (image) or video of the user 10. Moreover, the output control unit 2903 controls the avatar 100 to edit the photo (image) or video taken by the avatar 100. As an example of this case, for example, the avatar 100 on the screen of a smartphone or the like takes a photo (image) or video, and edits the taken photo (image) or video. Moreover, the output control unit 2903 controls the avatar 100 to take a behavior of saving the photo (image) or video edited by the avatar 100. In this case, for example, the avatar 100 on the screen of a smartphone or the like edits the photo (image) or video, and saves the edited photo (image) or video. Through such control, the avatar 100 creates a family album and saves the irreplaceable memories of the family.

[0104] Furthermore, the chronological situation information of the user 10 may include at least any one of memorial items of the user 10, such as photos (images), videos, letters, and diaries. The output control unit 2903 controls the avatar 100 to perform a predetermined action including at least any one of organizing, storing, and backing up the memorial items collected by the collection unit 2902. In this case, too, for example, the avatar 100 on the screen of a smartphone or the like organizes, stores, backs up, and the like, the memorial items. Through such control, the avatar 100 protects the important memories of the family. In this way, the avatar 100 can provide support that is in tune with the emotions of the user 10.

[0105] Furthermore, the time-varying situation information of the user 10 may include the schedule of the user 10. The output control unit 2903 controls the avatar 100 to perform an action of setting a reminder based on the schedule of the user 10. The output control unit 2903 may also control the avatar 100 to perform an action of setting an alert based on the schedule of the user 10. In this case, the avatar 100 performs at least one of setting a reminder and setting an alert. Through such control, the avatar 100 supports the schedule management of the family. This enables smooth schedule management of the family. In this case, for example, the avatar 100 on the screen of a smartphone or the like sets the reminder and the alert.

[0106] Furthermore, the output control unit 2903 controls the avatar 100 to act in accordance with the photograph (image) or video of the user 10, for example, at a wedding ceremony or a funeral (memorial ceremony). In this case, for example, the avatar 100 is an avatar disguised as a master of ceremonies or an announcer, and at a wedding ceremony, for example, the avatar 100 narrates memorial items (memorial videos, etc.) related to the bride and groom. Also, at a funeral ceremony, for example, the avatar 100 narrates memorial items (memorial videos, etc.) related to the deceased. When narrating, the avatar 100 should preferably do so in the voice of a professional master of ceremonies or announcer. Through such control, appropriate support can be provided according to the situation of the user 10.

[0107] 7 is a schematic diagram showing an example of an operation flow of the output unit 290. In step S200, the detection unit 2901 of the output unit 290 judges whether or not the occurrence of a predetermined event is detected (step S200). If the occurrence of the predetermined event is not detected (step S200; No), the detection unit 2901 waits until the occurrence of the predetermined event is detected.

[0108] On the other hand, when the occurrence of a predetermined event is detected (step S200; Yes), the collection unit 2902 of the output unit 290 collects situation information indicating the situation of the user 10 (step S201). Next, the output control unit 2903 of the output unit 290 controls the avatar 100 to output an action corresponding to the situation information to the outside (step S202), and ends the process.

[0109] The control device 200 is an example of an information processing device that controls the avatar 100. The behavior control by the control device 200 is not limited to the avatar 100, but can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.

[0110] 8 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the control device 200 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations associated with the device according to the present embodiment or one or more "parts" thereof, and / or cause the computer 1200 to execute a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0111] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0112] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0113] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0114] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0115] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0116] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0117] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0118] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0119] The above-mentioned programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0120] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0121] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0122] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0123] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0124] (Other embodiments) The above-mentioned control device 200 may be mounted on a stuffed toy, a robot, etc., or may be applied to a control device connected wirelessly or via a wire to a control target device (speaker or camera) mounted on a stuffed toy, a robot, etc.

[0125] The emotion determining unit 242 may determine the emotion of the user according to a specific mapping. Specifically, the emotion determining unit 242 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping.

[0126] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0127] (1) For example, if the emotion engine, which is the emotion determination unit 242 that controls the emotion of the avatar 100, detects emotions in about 100 msec, the frequency of determining the reaction action of the avatar 100 (for example, a backchannel) may be set to at least the same timing as the emotion engine's detection frequency (100 msec), or may be set to an earlier timing. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0128] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. Avatar 100 performs a reaction (backchannel, etc.) according to the directionality and the degree (strength) of the mandala in emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

[0129] (2) The directionality of emotions and the strength of their intensity may be preset in reference to emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if avatar 100 feels a sense of stability, security, etc., avatar 100 may nod and continue listening. If avatar 100 feels anxious, confused, or suspicious, avatar 100 may tilt its head or stop shaking its head.

[0130] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0131] (3) If the avatar 100 feels good after being praised, the filler "ah" may precede the line, and if the avatar 100 feels pain after receiving harsh words, the filler "ugh!" may precede the line. Also, a physical reaction such as the avatar 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 400.

[0132] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

[0133] When the avatar 100 feels a favorable feeling in its situational awareness while having an internal feeling (reaction) of a sense of satisfaction, the avatar 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the avatar 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0134] Conversely, if avatar 100 is aware of an internal sensation (reaction) of discomfort, avatar 100 may shake its head when it feels disgust, or glare at the other person when it feels hatred. These types of emotions are distributed around the 6 o'clock position on emotion map 400.

[0135] (5) The inside of emotion map 400 represents what is going on in one's mind, while the outside of emotion map 400 represents behavior. Therefore, the further out on the outside of emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[0136] (6) When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock area of ​​emotion map 400, avatar 100 may lightly nod its head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.

[0137] The emotion determination unit 242 inputs the information analyzed by the sensor module unit 220 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 220 and the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. FIG. 10 is a diagram showing another example of an emotion map. This neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, such as "peace of mind," "calm," and "encouraging," have similar emotion values.

[0138] Furthermore, the emotion determination unit 242 may determine the emotion of the avatar 100 according to a specific mapping. Specifically, the emotion determination unit 242 inputs the information analyzed by the sensor module unit 220, the state of the user 10 recognized by the user state recognition unit 240, and the state of the avatar 100 to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the avatar 100. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 220, the recognized state of the user 10, and the state of the avatar 100, and emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on learning data that indicates that when it is recognized from the output of the touch sensor that the avatar 100 is being stroked by the user 10, the emotion value becomes "3" which indicates "happiness," and learning data that indicates that when it is recognized from the output of the acceleration sensor that the avatar 100 is being hit by the user 10, the emotion value becomes "3" which indicates "anger." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0139] Furthermore, the emotion determination unit 242 may determine the emotion of the avatar 100 based on the action content of the avatar 100 generated by the sentence generation model. Specifically, the emotion determination unit 242 inputs the action content of the avatar 100 generated by the sentence generation model to a pre-trained neural network, acquires emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the acquired emotion values ​​indicating each emotion with the emotion values ​​indicating each emotion of the current avatar 100 to update the emotion of the avatar 100. For example, the emotion values ​​indicating each acquired emotion and the emotion values ​​indicating each emotion of the current avatar 100 are averaged and integrated. This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the action content of the avatar 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0140] For example, if the speech content of avatar 100 generated by the sentence generation model is "That's great. You're lucky.", then when the text representing this speech content is input into the neural network, a high emotion value for the emotion "happy" is obtained, and the emotion of avatar 100 is updated so that the emotion value of the emotion "happy" becomes higher.

[0141] The behavior determination unit 246 generates the behavior content of the avatar by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the avatar's emotions to ask about the avatar's behavior content corresponding to the user's behavior, and inputting the added sentences into a sentence generation model with an interactive function.

[0142] For example, the behavior determining unit 246 obtains text representing the state of the avatar 100 from the emotion of the avatar 100 determined by the emotion determining unit 242, using an emotion table as shown in Fig. 11. Fig. 11 is a diagram showing an example of an emotion table. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the avatar 100 is stored for each index number.

[0143] When the emotion of the avatar 100 determined by the emotion determination unit 242 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the avatar 100 correspond to multiple index numbers, multiple texts representing the state of the avatar 100 are obtained.

[0144] Also, for the emotions of the user 10, an emotion table as shown in FIG. 12 is prepared. FIG. 12 is a diagram showing an example of an emotion table. Here, if the user's behavior is speaking "It's AAA," the emotion of the avatar 100 is index number "2," and the emotion of the user 10 is index number "3," then "The avatar is in a very happy state. The user is in a normal happy state. The user spoke to me saying, "It's AAA. How would you respond as the avatar?" is input to the sentence generation model, and the content of the avatar's behavior is obtained. The behavior decision unit 246 decides the avatar's behavior from this content of the behavior.

[0145] In this way, avatar 100 can change its behavior according to the index number corresponding to the avatar's emotions, so that the user gets the impression that avatar 100 has a heart, and is encouraged to take actions such as talking to the avatar.

[0146] Furthermore, the behavior determining unit 246 may generate the behavioral content of the avatar by adding not only texts representing the user's behavior, the user's emotions, and the avatar's emotions, but also texts representing the contents of the history data 232, adding a fixed sentence for asking about the avatar's behavioral content corresponding to the user's behavior, and inputting the result into the dialogue function. This allows the avatar 100 to change the behavior of the avatar according to the history data representing the user's emotions and behavior, so that the user has the impression that the avatar has individuality, and is encouraged to take actions such as talking to the avatar. Furthermore, the history data may further include the avatar's emotions and behavior.

[0147] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0148] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0149] 5 system, 10, 11, 12 user, 20 communication network, 100, 101, 102 avatar, 200, 201, 202 control device, 210 sensor unit, 211 microphone, 212 3D depth sensor, 213 2D camera, 214 distance sensor, 220 sensor module unit, 221 voice emotion recognition unit, 222 speech understanding unit, 223 facial expression recognition unit, 224 face recognition unit, 230 storage unit, 231 reaction rule, 232 history data, 233 character data, 240 user state recognition unit, 242 emotion determination unit, 244 behavior recognition unit, 246 behavior determination unit, 248 memory control unit, 250 behavior control unit, 252 control target, 280 communication processing unit, 290 output unit, 300 server, 400, 900 emotion map, 1200 Computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip, 2521 display device, 2522 speaker, 2901 detection unit, 2902 collection unit, 2903 output control unit, 2911 user information

Claims

1. a behavior determining unit that determines an avatar's behavior corresponding to situation information based on an emotion value of the user and an emotion value of the avatar, the emotion value corresponding to the situation information indicating the situation of the user including the family; a behavior control unit that controls the avatar based on the behavior of the avatar determined by the behavior determination unit; A behavior control system comprising:

2. The action determination unit is When the situation information, the emotion value of the user corresponding to the situation information, and the emotion value of the avatar satisfy a condition of a preset reaction rule, a behavior of the avatar corresponding to the reaction rule is determined. The behavior control system according to claim 1 .

3. The action determination unit is Based on the action of the avatar corresponding to the reaction rule determined by the action determination unit, the action of the avatar and the action of a control target related to the action of the avatar are controlled. The behavior control system according to claim 2 .

4. The control device that controls the avatar is connected wirelessly or wired to a predetermined wearable terminal device related to at least one of augmented reality, virtual reality, and mixed reality. The behavior control system according to any one of claims 1 to 3.

5. A collection unit that collects situation information indicating a situation of the user including a family member; an output control unit that controls an avatar controlled by a control device having a sentence generation model to take an action according to the situation information collected by the collection unit; Further equipped The behavior control system according to claim 1 .

6. The collecting unit includes: Acquiring status information of the user over time; The output control unit is The situation information collected by the collection unit is selected according to a time when the situation information was collected by the collection unit, and the avatar is controlled to perform an action of recording for each of the selected situation information. The behavior control system according to claim 5 .

7. The user's time-based status information includes at least one of an image and a video of the user, The output control unit is The action of photographing the user is controlled so that the avatar performs the action of photographing the user in order to cause the collection unit to collect the situation information. The behavior control system according to claim 6.

8. The output control unit is The situation information captured by the avatar is edited, and the avatar is controlled to perform an action of saving the edited situation information. The behavior control system according to claim 7.

9. The user's chronological status information includes at least one of an image, a video, a letter, and a diary of the user; The output control unit is Controlling the avatar to perform a predetermined action including at least one of organizing, storing, and backing up the situation information collected by the collection unit. The behavior control system according to claim 5 .

10. the user's contextual information includes the user's schedule; The output control unit is Based on the situation information collected by the collection unit, a predetermined action including at least one of setting a reminder and setting an alert is controlled so that the avatar performs the predetermined action. The behavior control system according to claim 5 .

11. The control device that controls the avatar is mounted on a specific robot, or is connected wirelessly or by wire to a control target device mounted on the robot. The behavior control system according to claim 1 or 5.

Images