Action control system

The behavior control system for dolls addresses the lack of responsiveness in conventional technologies by using AI engines to recognize user behavior and emotions, enabling appropriate and personalized interactions through rule-based actions and learning from user reactions.

JP2024159561APending Publication Date: 2024-11-08SOFTBANK GROUP CORP

Patent Information

Application Number
JP2024064661
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-27
Filing Date
2024-04-12
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Conventional technologies fail to effectively enable a doll to perform actions appropriate to a user's actions and emotions, lacking in responsiveness and personalization.

Method used

A behavior control system for dolls that includes user state recognition, emotion determination, and interaction functions, utilizing AI engines to recognize user behavior and emotions, and determine corresponding doll behaviors through a combination of chat and emotion engines, with rule-based actions and learning from user reactions.

Benefits of technology

Enables the doll to perform actions that are appropriate and responsive to user actions and emotions, enhancing user interaction and affinity by simulating emotional responses and learning from user feedback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159561000001_ABST
    Figure 2024159561000001_ABST
Patent Text Reader

Abstract

To make a doll take an appropriate action in response to an action of a user.SOLUTION: An action control system disclosed herein comprises a user condition recognition unit for recognizing user condition including an action of the user, an emotion determination unit for determining emotions of the user or emotions of a doll, and an action determination unit configured to use a dialog function for allowing the user and the doll to have a dialog in order to determine details of an action to be taken by the doll in response to the action of the user and emotions of the user or the doll.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a behavior control system. [Background technology]

[0002] Patent Document 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Document 1 recognizes the user's reaction when the robot executes a specific action, and if the robot is unable to determine an action for the recognized user's reaction, it updates the robot's behavior by receiving information about an action suitable for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has room for improvement in terms of making the doll perform an appropriate action in response to the user's action. [Means for solving the problem]

[0005] According to a first aspect of the present invention, there is provided a behavior control system, which includes a user state recognition unit that recognizes a user state including a user's behavior, an emotion determination unit that determines an emotion of the user or an emotion of a doll, and a behavior determination unit that determines an action of the doll corresponding to the user state and the emotion of the user or the emotion of the doll based on a dialogue function that allows the user and the doll to dialogue with each other. Effect of the Invention

[0006] According to the present invention, it is possible to provide a behavior control system capable of causing a doll to perform an appropriate behavior in response to a user's behavior. [Brief description of the drawings]

[0007] [Figure 1] 1 illustrates a schematic diagram of an example of a system 5 according to the present embodiment. [Diagram 2] 1 shows a schematic functional configuration of a doll 100. [Diagram 3] 1 shows an example of a flow of actions performed by the doll 100. [Figure 4] 1 illustrates an example of a hardware configuration of a computer 1200. [Diagram 5] 4 shows an emotion map 400 onto which multiple emotions are mapped. [Figure 6] 9 shows an emotion map 900 onto which multiple emotions are mapped. [Figure 7] 4 shows an emotion table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0009] (Present embodiment) FIG. 1 shows an example of a system 5 according to the present embodiment. The system 5 includes a doll 100, a doll 101, a doll 102, and a server 300. The user 10, the user 11, and the user 12 are users of the doll 100, the doll 101, and the doll 102, respectively. The user 10, the user 11, and the user 12 are, for example, family members of the house in which the doll 100 is placed, or strangers who visit the house. The dolls 100, the doll 101, and the doll 102 can be placed at the reception desk of a home, store, or office, and can be used to respond to visiting customers. In the description of this embodiment, the dolls 101 and the doll 102 have substantially the same functions as the doll 100. Therefore, the system 5 will be described by mainly focusing on the functions of the doll 100.

[0010] The doll 100 is a robot doll that can converse with the user 10 and provide the user 10 with a video. At this time, the doll 100 cooperates with a server 300 and the like that can communicate with the doll 10 via a communication network 20 to converse with the user 10 and provide the user 10 with a video. For example, the doll 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn so as to be able to have a more appropriate conversation with the user 10. The doll 100 also records the captured video data of the user 10 in the server 300, and requests the video data from the server 300 as necessary to provide it to the user 10. In FIG. 1, the doll 100 is assumed to be a doll that the user can carry around, but it can be applied to any terminal (for example, a car-mounted terminal, a mobile terminal, etc.) equipped with an AI emotion engine or an AI chat engine as described below.

[0011] The doll 100 is equipped with an AI emotion engine (described later) that has a function of generating human emotions in a simulated manner, and determines and holds emotion values ​​that represent the types of its own emotions. For example, the doll 100 has emotion values ​​that represent the strength of each of the emotions of "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "concern," "relief," "fulfillment," "emptiness," and "normal." When the doll 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the doll 100 utters sounds at a fast speed. In this way, the doll 100 can express its own emotions through actions.

[0012] Moreover, the doll 100 may be configured to match an AI chat engine (chat engine) with an AI emotion engine (emotion engine) to determine the behavior of the doll 100 corresponding to the emotion of the user 10. Specifically, the doll 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 with respect to the behavior of the user 10, and determine the behavior of the doll 100 corresponding to the determined emotion.

[0013] More specifically, when the doll 100 recognizes the behavior of the user 10, the doll 100 automatically generates the behavior content that the doll 100 should take in response to the behavior of the user 10 using a preset chat engine. The chat engine may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The chat engine is described, for example, in JP2018-081444A and chatGPT (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) is well known, and therefore detailed description thereof will be omitted. Such a chat engine is configured using a large language model (LLM).

[0014] As described above, this embodiment combines a large-scale language model with an emotion engine, thereby making it possible to reflect the emotions of the user 10 and the doll 100 and various linguistic information in the behavior of the doll 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining a chat engine and an emotion engine.

[0015] The doll 100 also has a function of recognizing the behavior of the user 10. The doll 100 recognizes the behavior of the user 10 by analyzing the face image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function. The doll 100 determines the behavior to be performed by the doll 100 based on the recognized behavior of the user 10, etc.

[0016] The doll 100 stores rules that define actions that the doll 100 will take based on the emotions of the user 10, the emotions of the doll 100, and the actions of the user 10, and performs various actions according to the rules.

[0017] Specifically, the doll 100 has reaction rules for determining the behavior of the doll 100 based on the emotions of the user 10, the emotions of the doll 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the doll 100 as "laughing" when the behavior of the user 10 is "laughing". The reaction rules also define the behavior of the doll 100 as "apologizing" when the behavior of the user 10 is "angry". The reaction rules also define the behavior of the doll 100 as "answering" when the behavior of the user 10 is "asking a question". The reaction rules also define the behavior of the doll 100 as "calling out" when the behavior of the user 10 is "sad".

[0018] When the doll 100 recognizes the behavior of the user 10 as "angry" based on the reaction rule, the doll 100 selects the behavior of "apologizing" defined in the reaction rule as the behavior to be executed by the doll 100. For example, when the doll 100 selects the behavior of "apologizing", the doll 100 performs the motion of "apologizing" and outputs a voice representing the word "apologize".

[0019] In addition, when the emotion of the doll 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, and "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the doll 100 will change to "worried" and the action of "calling out" can be executed.

[0020] When the doll 100 recognizes based on the reaction rule that the current emotion of the doll 100 is "normal" and that the user 10 is alone and seems lonely, the doll 100 increases the emotion value of "sadness" of the doll 100. In addition, the doll 100 selects the action of "calling out" defined in the reaction rule as the action to be performed on the user 10. For example, when the doll 100 selects the action of "calling out", the doll 100 converts the words "What's wrong?", which express concern, into a worried voice and outputs it.

[0021] The doll 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the doll 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and the attributes of the user 10.

[0022] The server 300 stores the user reaction information received from the doll 100. The server 300 receives and stores user reaction information not only from the doll 100, but also from each of the dolls 101 and 102. The server 300 analyzes the user reaction information from the dolls 100, 101, and 102 and updates the reaction rules.

[0023] The doll 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The doll 100 incorporates the updated reaction rules into the reaction rules stored in the doll 100. This allows the doll 100 to incorporate the reaction rules acquired by the dolls 101 and 102 into its own reaction rules.

[0024] 2 shows a schematic functional configuration of the doll 100. The doll 100 has a sensor section 200, a sensor module section 210, a storage section 220, a user state recognition section 230, a user emotion determination section 231, an emotion determination section 232, a behavior recognition section 234, an behavior determination section 236, a memory control section 238, a behavior control section 250, a control target 252, and a communication processing section 280.

[0025] The control object 252 includes the display device, speaker, and eye LEDs mounted on the doll 100, as well as motors for driving the arms, hands, and legs. The posture and gestures of the doll 100 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the doll 100 can be expressed by controlling these motors. In addition, the facial expressions of the doll 100 can also be expressed by controlling the lighting state of the LEDs in the eyes of the doll 100. The posture, gestures, and facial expressions of the doll 100 are examples of the attitude of the doll 100.

[0026] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the doll 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like.

[0027] 2, the components other than the control target 252 and the sensor unit 200 are examples of components included in the behavior control system of the doll 100. The behavior control system of the doll 100 controls the control target 252.

[0028] The storage unit 220 includes a reaction rule 221 and history data 222. The history data 222 includes the emotion value and behavior history of the user 10 in the past. The emotion value and behavior history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, etc. Among the components of the doll 100 shown in FIG. 2, the functions of the components other than the control target 252, the sensor unit 200, and the storage unit 220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program operating on the OS.

[0029] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the user state recognition unit 230.

[0030] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice, and recognizes the voice and emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0031] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0032] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0033] The user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0034] The user emotion determination unit 231 includes an emotion recognition engine 231a, and determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0035] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.

[0036] The emotion determination unit 232 determines an emotion value indicating the emotion of the doll 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The emotion determination unit 232 includes an endocrine control unit 232a and an emotion generation engine 232b. The endocrine control unit 232a adjusts the parameters of the neural network used in the emotion generation engine 232b using the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the endocrine control unit 232a adjusts a parameter corresponding to the amount of dopamine released. Dopamine is an example of an endocrine substance. An endocrine substance means a substance that is secreted in the body and transmits a signal, such as a neurotransmitter or a hormone. However, the endocrine substance of the doll 100 itself is one of the pieces of information that affect the operation of the doll 100, and does not mean that the doll 100 actually generates an endocrine substance. In addition, since the use of endocrine secretions to determine a doll's own emotions is publicly known, as disclosed in, for example, JP 2018-81583 A, detailed explanation will be omitted.

[0037] The emotion generation engine 232b uses a neural network to determine an emotion value indicating the emotion of the doll 100 based on the information analyzed by the sensor module section 210, the state of the user 10 recognized by the user state recognition section 230, and the parameters adjusted by the endocrine control section 232a. The emotion value of the doll 100 includes emotion values ​​for each of a plurality of emotion classifications, and in this embodiment, values ​​(0 to 5) indicating the strength of each of "joy", "anger", "sorrow", and "pleasure" are assumed.

[0038] To explain with a specific example, for example, when the user state recognition unit 230 recognizes that the user 10 looks lonely, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "sadness" of the doll 100, and the emotion determination unit 232 increases the emotion value of "sadness" of the doll 100. Also, when the user state recognition unit 230 recognizes that the user 10 has smiled, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "joy" of the doll 100, and the emotion determination unit 232 increases the emotion value of "joy" of the doll 100.

[0039] The emotion determination unit 232 may determine the emotion value indicating the emotion of the doll 100 by further considering the state of the doll 100. For example, when the battery level of the doll 100 is low or when the surrounding environment of the doll 100 is pitch black, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "sadness" of the doll 100, and the emotion determination unit 232 may increase the emotion value of "sadness" of the doll 100. Furthermore, when the user 10 continues to talk to the doll 100 despite the remaining battery level being low, the endocrine control unit 232a controls to increase the parameter corresponding to the emotion value of "anger" of the doll 100, and the emotion determination unit 232 may increase the emotion value of "anger" of the doll 100.

[0040] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing", "getting angry", "asking a question", "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0041] As described above, in this embodiment, the doll 100 identifies the user 10 and acquires the contents of the user's utterance. When acquiring and using the contents of the utterance, the doll 100 obtains the necessary consent in accordance with laws and regulations from the user 10. The behavior control system of the doll 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0042] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 234 based on the current emotion value of the user 10 determined by the user emotion determining unit 231, the history data 222 of past emotion values ​​determined by the user emotion determining unit 231 before the current emotion value of the user 10 was determined, and the emotion value of the doll 100 determined by the emotion determining unit 232. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period before, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the doll 100 but also the history of the past emotion values ​​of the doll 100. The behavior determined by the behavior determining unit 236 includes gestures made by the doll 100 or speech contents of the doll 100 .

[0043] The behavior determining unit 236 according to this embodiment determines the behavior of the doll 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the doll 100, the behavior of the user 10, and the reaction rule 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior determining unit 236 determines, as a behavior corresponding to the behavior of the user 10, a behavior for changing the emotion value of the user 10 to a positive value.

[0044] The reaction rule 221 defines the behavior of the doll 100 according to a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the doll 100, and the behavior of the user 10. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the doll 100.

[0045] For example, the reaction rule 221 defines the behavior of the doll 100 for all combinations of the patterns of the emotion values ​​of the doll 100 (1296 patterns, which are the fourth power of six values ​​of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); the combination patterns of the past emotion values ​​and the current emotion values ​​of the user 10; and the behavior patterns of the user 10. That is, for each pattern of the emotion values ​​of the doll 100, the behavior of the doll 100 according to the behavior patterns of the user 10 is defined for each of a plurality of combinations of the past emotion values ​​and the current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that the behavior determining unit 236 may transition to an operation mode in which the behavior of the doll 100 is determined using the history data 222, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0046] In addition, the reaction rule 221 may prescribe at least one of a gesture and a statement as the behavior of the doll 100 for each of the patterns (1296 patterns) of the emotional value of the doll 100. Alternatively, the reaction rule 221 may prescribe at least one of a gesture and a statement as the behavior of the doll 100 for each group of patterns of the emotional value of the doll 100.

[0047] The strength of each gesture included in the behavior of the doll 100 defined in the reaction rule 221 is predefined. The strength of each utterance included in the behavior of the doll 100 defined in the reaction rule 221 is predefined.

[0048] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the doll 100 determined by the emotion determination unit 232. Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the doll 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0049] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the user state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

[0050] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the doll 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the doll 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0051] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. When an impact is detected by the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0052] Moreover, the behavior control unit 250 controls the emotion determination unit 232 to further change the emotion value of the doll 100 based on the user's reaction to the execution of the behavior after the behavior determined by the behavior determination unit 236 is executed in the execution mode determined according to the emotion of the doll 100. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the doll 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Moreover, the emotion determination unit 232 increases the emotion value of "sadness" of the doll 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.

[0053] Furthermore, the behavior control unit 250 expresses the emotion of the doll 100 based on the determined emotion value of the doll 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the doll 100, it controls the control object 252 to make the doll 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the doll 100, it controls the control object 252 to make the doll 100 droop.

[0054] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules 221.

[0055] The server 300 communicates between the dolls 100, 101, and 102 and the server 300, receives user reaction information transmitted from the doll 100, and updates the reaction rules based on the reaction rules including actions that have received positive reactions. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.

[0056] Fig. 3 shows an example of a motion flow for determining the action of the doll 100. The motion flow shown in Fig. 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the motion flow indicates the step to be executed.

[0057] First, in step S100 , the user state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210 .

[0058] In step S102, the user emotion determination unit 231 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0059] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the doll 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

[0060] In step S104 , the behavior recognition unit 234 recognizes a behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 .

[0061] In step S106, the behavior determination unit 236 determines the behavior of the doll 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the doll 100, the behavior of the user 10 recognized by the behavior recognition unit 234, and the reaction rules 221.

[0062] In step S 108 , the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determining unit 236 .

[0063] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensities for the actions determined by the action determining unit 236 and the emotion value of the doll 100 determined by the emotion determining unit 232.

[0064] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than the threshold. If the total intensity value is less than the threshold, the process ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S114.

[0065] In step S114, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the user state recognition unit 230 are stored in the history data 222.

[0066] As described above, the doll 100 determines an emotion value indicating the emotion of the doll 100 based on the user state, and determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the doll 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the doll 100 determines that the user state 10 years from now is the same as that 10 years ago, the doll 100 can present to the user 10 all kinds of peripheral information, such as the state of the user 10 10 years ago (for example, the facial expression, emotions, etc. of the user 10), and data such as sounds, images, and smells of the place.

[0067] Moreover, according to the doll 100, it is possible to make the doll 100 perform an appropriate action in response to the action of the user 10. Conventionally, the action of the user is classified, and an action including, for example, a facial expression or a posture of the robot is determined. In contrast, the doll 100 of the present embodiment determines the current emotion value of the user 10, and performs an action on the user 10 based on the past emotion value and the current emotion value. Therefore, for example, if the user 10 who was cheerful yesterday is depressed today, the doll 100 can utter such a thing as "You were cheerful yesterday, but what's wrong with you today?" The doll 100 can also utter with gestures. For example, if the user 10 who was depressed yesterday is cheerful today, the doll 100 can utter such a thing as "You were depressed yesterday, but you look cheerful today, don't you?" For example, if the user 10 who was cheerful yesterday is more cheerful today than yesterday, the doll 100 can utter such a thing as "You're more cheerful today than yesterday. Did something better happen than yesterday?" In addition, for example, for a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, the doll 100 can say something like, "You've been feeling stable lately, which is good."

[0068] Also, for example, the doll 100 can ask the user 10, "Did you finish the homework you told me about yesterday?", and when the user 10 answers, "I did it," it can utter a positive utterance such as "Great!" and make a positive gesture such as clapping or a thumbs up. Also, for example, when the user 10 utters, "The presentation you gave the day before yesterday went well," the doll 100 can utter a positive utterance such as "You did a good job!" and make the above-mentioned positive gesture. In this way, it is expected that the user 10 will feel a sense of closeness to the doll 100 by the doll 100 taking an action based on the state history of the user 10.

[0069] In the above embodiment, the doll 100 recognizes the user 10 using the face image of the user 10, but the disclosed technology is not limited to this. For example, the doll 100 may recognize the user 10 using the voice of the user 10, the email address of the user 10, the SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 possesses.

[0070] The doll 100 is an example of an electronic device equipped with a behavior control system. The behavior control system is not limited to being applied to the doll 100, but can be applied to various electronic devices. The functions of the server 300 may be implemented by one or more computers. At least a part of the functions of the server 300 may be implemented by a virtual machine. At least a part of the functions of the server 300 may be implemented in the cloud.

[0071] 4 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the doll 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0072] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0073] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0074] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0075] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0076] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0077] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0078] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0079] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0080] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0081] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0082] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0083] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0084] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0085] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0086] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order.

[0087] (Other embodiments) The doll 100 may execute a heat generation process that changes the amount of heat emitted from the doll 100 according to the user's preference, situation, and reaction, through the following steps 1 to 5-2. This heat generation process allows the doll 100 to simulate changes in body temperature based on emotions.

[0088] The doll 100 is equipped with a heat generating device. The heat generating device is a device that generates heat based on power supplied from a battery built into the doll 100. The heat generating device generates heat at a temperature that the user 10 feels is similar to human body temperature when holding the doll 100, for example, a temperature of about 36 degrees. The surface temperature of the doll 100 is about 32 to 33 degrees. The heat generating device can change the amount of heat it generates. The doll 100 of this embodiment expresses emotions through changes in its temperature.

[0089] (Step 1) The doll 100 acquires the state of the user 10, the emotional value of the user 10, the emotional value of the doll 100, and the history data 222. Specifically, the same processes as those in steps S100 to S103 are performed to acquire the state of the user 10, the emotional value of the user 10, the emotional value of the doll 100, and the history data 222.

[0090] (Step 2) The doll 100 acquires the preferences of the user 10 regarding, for example, the actions of the doll 100. Specifically, the behavior decision unit 236 decides an utterance to ask the user 10 about his / her preference regarding the behavior of the doll 100 as the behavior of the doll 100, and the behavior control unit 250 controls the control object 252 to make an utterance to ask the user 10 about his / her preference regarding the behavior of the doll 100. The behavior of the doll 100 includes behaviors such as singing and dancing. The user state recognition unit 230 recognizes the user 10's preference regarding the behavior of the doll 100 based on the information analyzed by the sensor module unit 210 (e.g., the user's answer).

[0091] (Step 3) The doll 100 determines the action to be taken toward the user 10. Specifically, the behavior determining unit 236 adds a fixed sentence such as "What is the doll's behavior recommended for the user at this time?" to the text representing the preferences of the user 10 regarding the behavior of the doll 100, the emotions of the user 10, the emotions of the doll 100, and the contents stored in the history data 222, and inputs it into the AI ​​chat engine to obtain the recommended contents regarding the behavior of the doll 100. At this time, by considering not only the preferences of the user 10 regarding the behavior of the doll 100 but also the emotions of the user 10 and the history data 222, it is possible to determine the behavior of the doll 100 suitable for the user 10. Also, by considering the emotions of the doll 100, it is possible to make the user 10 feel that the doll 100 has emotions.

[0092] (Step 4) The doll 100 performs the action determined in step 3 on the user 10, and obtains a reaction from the user 10. Specifically, the behavior determining unit 236 determines the behavior of the doll 100 to the user 10, and the behavior control unit 250 controls the control target 252 to perform a predetermined behavior on the user 10. The user state recognizing unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210, and the user emotion determining unit 231 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognizing unit 230. The behavior determining unit 236 determines whether the reaction of the user 10 is positive or not based on the state of the user 10 recognized by the user state recognizing unit 230 and the emotion value indicating the emotion of the user 10, and changes the body temperature of the doll 100 based on the result of the determination.

[0093] (Step 5-1) If the reaction of the user 10 is positive, the behavior determining unit 236 executes a process of increasing the body temperature of the doll 100. Specifically, the behavior control unit 250 controls the heat generating device, which is the control target 252, so as to increase the amount of heat emitted from the doll 100. As a result, the heat generating device emits heat of, for example, about 37 degrees. Therefore, when the user 10 is holding the doll 100, the user 10 can feel that the body temperature of the doll 100 has risen slightly. As a result, the user 10 can feel as if the doll 100 is excited based on the positive reaction of the user 10.

[0094] (Step 5-2) If the reaction of the user 10 is not positive, the behavior determining unit 236 executes a process to lower the body temperature of the doll 100. Specifically, the behavior control unit 250 controls the heat generating device, which is the control target 252, so as to reduce the amount of heat emitted from the doll 100. As a result, the heat generating device emits heat of, for example, about 35 degrees. Therefore, when the user 10 is holding the doll 100, the user 10 can feel that the body temperature of the doll 100 has slightly decreased. As a result, the user can feel as if the doll 100 has become sad based on the user's 10 negative reaction.

[0095] In this way, the doll 100 can execute a process of changing its body temperature in accordance with the user's preferences, the user's situation, and the user's reactions.

[0096] The user emotion determination unit 231 may determine the emotion of the user according to a specific mapping. Specifically, the user emotion determination unit 231 may use the emotion recognition engine 231a to determine the emotion of the user according to an emotion map (see FIG. 5), which is a specific mapping. Similarly, the emotion determination unit 232 may use the emotion generation engine 232b to determine the emotion of the doll 100 according to an emotion map (see FIG. 5), which is a specific mapping. Note that when there is no need to particularly distinguish between the emotion recognition engine 231a and the emotion generation engine 232b, they are collectively referred to as emotion engines.

[0097] FIG. 5 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0098] For example, if the emotion engine detects emotions at about 100 msec, the frequency of the reaction action (e.g., a back-channel) of the doll 100 may be set at least as frequently as the emotion engine's detection frequency (100 msec), or may be set earlier than this. The emotion engine's detection frequency may be interpreted as a sampling rate.

[0099] By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, and a natural dialogue that reads the atmosphere can be realized. The doll 100 performs a reaction (backchannel, etc.) according to the directionality and the degree (strength) of the mandala in the emotion map 400. The detection frequency (sampling rate) of the emotion recognition engine 231a is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

[0100] The direction of the emotion and its strength may be preset in reference to the emotion map 400, and the movement of the backchannel and the strength of the backchannel may be set. For example, if the doll 100 feels a sense of stability and security, the doll 100 will nod and continue to listen. If the doll 100 feels anxious, confused, or suspicious, the doll 100 may tilt its head or stop shaking its head.

[0101] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0102] If the doll 100 is praised and feels good, the filler "ah" may precede the line, and if the doll 100 is harshly criticized and feels pain, the filler "ugh!" may precede the line. Also, a physical reaction such as the doll 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 400.

[0103] In the left half of the emotion map 400, internal sensations (reactions) take precedence over situational awareness, which can give the impression of an unconscious reaction.

[0104] When the doll 100 feels a favorable feeling in its situational awareness while having an internal feeling (reaction) of a sense of satisfaction, the doll 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the doll 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, a tolerance or tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0105] Conversely, when the doll 100 feels an internal sensation (reaction) of discomfort and also feels disgust in its awareness of the situation, the doll 100 may shake its head, or, if it feels hatred, may turn the LEDs in its eyes red and glare at the other person. Such emotions are distributed around the 6 o'clock position on the emotion map 400.

[0106] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0107] When listening to someone while feeling a sense of security, which is distributed around the 3 o'clock area of ​​the emotion map 400, the doll 100 may lightly nod its head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.

[0108] The user emotion determination unit 231 may input the information analyzed by the sensor module unit 210 and the recognized state of the user 10 to a pre-trained neural network, obtain emotion values ​​indicating each emotion shown in the emotion map 400 using the emotion recognition engine 231a, and determine the emotion of the user 10. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. In addition, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6. FIG. 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "encouraging," have similar emotion values.

[0109] The emotion determination unit 232 may input the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the doll 100 to a pre-trained neural network, obtain emotion values ​​indicating each emotion shown in the emotion map 400 using the emotion generation engine 232b, and determine the emotion of the doll 100. This neural network is pre-trained based on a plurality of learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the doll 100, and emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on learning data that indicates that when it is recognized from the output of a touch sensor (not shown) that the doll 100 is being stroked by the user 10, the emotion value becomes "3" for "happiness," and learning data that indicates that when it is recognized from the output of an acceleration sensor (not shown) that the doll 100 is being hit by the user 10, the emotion value becomes "3" for "anger." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0110] The behavior determination unit 236 may generate the behavior content of the doll by adding a fixed sentence for asking about the behavior content of the doll corresponding to the user's behavior to the text representing the user's behavior, the user's emotions, and the doll's emotions, and inputting the added text into the dialogue function.

[0111] For example, the behavior determining unit 236 obtains text representing the state of the doll 100 from the emotion of the doll 100 determined by the emotion determining unit 232, using an emotion table as shown in Fig. 7. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the doll 100 is stored for each index number.

[0112] When the emotion of the doll 100 determined by the emotion determining unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotion of the doll 100 corresponds to multiple index numbers, multiple texts expressing the state of the doll 100 are obtained.

[0113] Also, for the emotions of the user 10, an emotion table that is the same as or similar to the emotion table shown in FIG. 7 is prepared. Here, if the user's behavior is "Ask how are you", the emotion of the doll 100 is index number "2", and the emotion of the user 10 is index number "3", then the following is input to the chat engine: "The doll is in a very happy state. The user is in a normal happy state. The user asks how are you. How would you respond as the doll?" and the action content of the doll is obtained. The action decision unit 236 decides the action of the doll from the action content.

[0114] In this way, the doll 100 can change its behavior according to the index number which corresponds to the doll's emotions, so that the user gets the impression that the doll has a heart, and is encouraged to take actions such as talking to the doll.

[0115] The behavior determining unit 236 may generate the behavior of the doll by adding not only the text representing the user's behavior, the user's emotion, and the emotion of the doll, but also the text representing the content of the history data 222, adding a fixed sentence for asking the user about the behavior of the doll corresponding to the user's behavior, and inputting the result into the dialogue function. This allows the doll 100 to change its behavior according to the history data representing the emotion and behavior of the user, so that the user has the impression that the doll has a personality, and is encouraged to take actions such as talking to the doll. The history data may further include the emotion and behavior of the doll. [Explanation of symbols]

[0116] 5 system, 10, 11, 12 user, 20 communication network, 100, 101, 102 doll, 200 sensor unit, 201 microphone, 202 depth sensor, 203 camera, 204 distance sensor, 210 sensor module unit, 211 voice emotion recognition unit, 212 speech understanding unit, 213 facial expression recognition unit, 214 face recognition unit, 220 storage unit, 221 reaction rule, 222 history data, 230 user state recognition unit, 232 emotion determination unit, 234 behavior recognition unit, 236 behavior determination unit, 238 memory control unit, 250 behavior control unit, 252 control target, 280 communication processing unit, 300 server, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 I / O controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 I / O chip

Claims

1. A user state recognition unit that recognizes a user state including a user's behavior; an emotion determining unit for determining an emotion of the user or an emotion of the doll; and a behavior determining unit that determines the behavior of the doll corresponding to the user's behavior and the user's emotion or the doll's emotion based on an interaction function that allows the user and the doll to interact with each other. Behavioral control system.

2. The action determination unit performs an action of changing the heat emitted from the doll in accordance with the emotion of the user or the emotion of the doll as the action content of the doll. The behavior control system according to claim 1 .

Citation Information

Patent Citations

  • Determination of analite in medium containing particle

    JP1985053847A

Cited By

  • Ai emotion recognition-based thermal control plush doll and interactive charater system using it

    KR103013846B1