Action control system

Abstract

An action control system according to the present invention includes: a user state recognition unit that recognizes a user state including an action of a user; an emotion determination unit that determines an emotion of the user or an emotion of the robot; an action specifying unit that specifies an action of the robot corresponding to the user state and the emotion of the user or the emotion of the robot on the basis of a text generation model having a dialogue function for causing the user to dialogue with the robot; and a storage control unit that stores data including an action of the user in history data, the action determination unit further selects a scene satisfying a predetermined selection criterion on the basis of the history data, inputs a text indicating the selected scene into an image generation model, generates an image indicating the scene, and outputs the image to the image generation model. And outputting the image.

Description

TECHNICAL FIELD

[0001] The present application relates to an action control system BACKGROUND

[0002] A technology for determining an appropriate action of a robot with respect to a state of a user is disclosed in Japanese Patent No. 6053847. In the related art of Patent Document 1, a reaction of the user when the robot performs a specific action is recognized, and in a case where an action of the robot with respect to the recognized reaction of the user cannot be determined, information related to an action appropriate for a recognized state of the user is received from a server, whereby the action of the robot is updated. SUMMARY

[0003] Problems to be Solved by the Invention However, in the related art, there is room for improvement in causing the robot to perform an appropriate action with respect to an action of the user.

[0004] Means for Solving the Problems According to a first aspect of the present disclosure, an action control system is provided. The action control system includes a user state recognition unit that recognizes a user state including an action of a user, an emotion determination unit that determines an emotion of the user or an emotion of a robot, an action determination unit that determines an action of the robot corresponding to the user state and the emotion of the user or the emotion of the robot, based on an article generation model having a conversation function with which the user and the robot have a conversation, and a storage control unit that stores data including the action of the user in history data. The action determination unit further selects a scenario that satisfies a predetermined selection criterion based on the history data, inputs a text indicating the selected scenario to an image generation model, generates an image indicating the scenario, and outputs the image.

[0005] In a second aspect, the storage control unit stores, in the history data, an emotion determined by the emotion determination unit and data including the action of the user. The action determination unit selects a scenario that satisfies the selection criterion related to the emotion based on the history data.

[0006] In a third aspect, the selection criterion related to the emotion includes a case where the emotion of the user and the emotion of the robot are the strongest, or a case where a specific emotion among a plurality of emotions is the strongest with respect to the emotion of the user or the emotion of the robot.

[0007] In a fourth aspect, the selection criterion includes a case where a smiling face of the user is the most.

[0008] In a fifth aspect, the action determination section receives at least one of a title and an article of the selected scene designated by the user, and outputs a combination of an image representing the scene and the received at least one of the title and the article.

[0009] In a sixth aspect, the action determination section receives an image of the user, inputs text representing the user of the image and text representing the selected scene to the image generation model, and generates an image representing the scene.

[0010] In a seventh aspect, the robot is mounted on a cloth doll, or connected to a control target device mounted on the cloth doll by wireless or wired.

[0011] In an eighth aspect, the control target device is a speaker mounted on a mouth of the cloth doll, a camera mounted on eyes constituting a face of the cloth doll, or a microphone mounted on an ear.

[0012] In a ninth aspect, the action determination section generates and outputs an image based on a feeling of the robot in a case where the robot is placed.

[0013] In a tenth aspect, the feeling of the robot is a feeling randomly determined or a personality given to the robot in advance, the action determination section prepares an image of the robot in advance for each feeling of the robot, and outputs the image corresponding to the feeling of the robot.

[0014] In an eleventh aspect, further comprising a storage section that stores a sound around the robot and a captured image as an event of a day, the action determination section selects a scene satisfying a predetermined selection criterion from the event of a day stored in the storage section in a case where the robot is placed, inputs text representing a feeling of the robot and the event corresponding to the selected scene to the image generation model, generates an image representing the scene, and outputs the image.

[0015] In a twelfth aspect, comprising: an extraction section that extracts a predetermined specific word from a speech of a user; a feeling determination section that determines a feeling of the user from at least one of the specific word extracted by the extraction section and a conversation content after the specific word; and an action determination section that determines an action of a robot based on the feeling determined by the feeling determination section.

[0016] In a thirteenth aspect, the specific word is determined using a learning completion model learned based on a conversation content of the user.

[0017] In a fourteenth aspect, the action determination section selects a scenario that satisfies a predetermined selection criterion based on the emotion determined by the emotion determination section and historical data including the user's actions, inputs text representing the selected scenario to an image generation model, generates an image representing the scenario, combines the generated image and the article representing the content of the conversation after the specific word, and outputs the combination.

[0018] In a fifteenth aspect, the emotion determination section determines an emotion value representing the user's emotion, and the action determination section selects a scenario in which the user's emotion value determined by the emotion determination section is above a predetermined threshold from the historical data.

[0019] In a sixteenth aspect, the action determination section excludes predetermined content and combines and outputs the article and an image representing the scenario.

[0020] In a seventeenth aspect, the action determination section selects a scenario in which the specific word is extracted by the extraction section.

[0021] In an eighteenth aspect, there is included a user state recognition section that recognizes a user state including the user's actions, an emotion determination section that determines the user's emotion or the robot's emotion, and an action determination section that determines an action of the robot corresponding to the user state and the user's emotion or the robot's emotion based on an article generation model having a conversation function for the user and the robot to converse, selects a scenario that satisfies a predetermined selection criterion based on historical data including the user's actions, inputs text representing the selected scenario to an image generation model, generates an image representing the scenario, extracts information of the user including the user's hobby based on the text, and determines content proposed to the user based on the extracted information of the user.

[0022] In a nineteenth aspect, the action determination section collects association information associated with the extracted information of the user and determines an action proposed to the user including the collected association information.

[0023] In a twentieth aspect, there is further included an action control section that controls an utterance from a speaker provided to the robot based on the action determined by the action determination section, thereby making a proposal of an action to the user.

[0024] In a twenty-first aspect, the action determination section stores the extracted information of the user in the historical data, and the action control section makes a proposal of an action to the user at a time point associated with the information of the user stored in the historical data.

[0025] In a twenty-second aspect, the action determination section stores the extracted information of the user in the history data, and performs matching with other users using the information of the user stored in the history data.

[0026] In a twenty-third aspect, includes: a user state recognition section that recognizes a user state including an action of a user; an emotion determination section that determines an emotion of a user or an emotion of a robot; and an action determination section that determines an image to be generated as an action of the robot corresponding to the user state and the emotion of the user or the emotion of the robot, based on an article generation model having a conversation function for the user to converse with the robot, the action determination section selecting a scenario that satisfies a predetermined selection criterion based on history data including past emotions of the user and actions of the user, inputting a text representing the selected scenario to an image generation model, generating an image representing the scenario, and outputting the image, and analyzing a lifestyle of the user based on the text, and outputting an analysis result.

[0027] In a twenty-fourth aspect, the storage control section further scores and outputs a pre-set ideal lifestyle as the analysis result of the lifestyle.

[0028] In a twenty-fifth aspect, further includes an action control section that controls the robot to speak the analysis result of the lifestyle analyzed by the action determination section.

[0029] In a twenty-sixth aspect, the action determination section outputs the analysis result to a mobile terminal of a pre-registered user.

[0030] In a twenty-seventh aspect, the action determination section further determines and outputs a content proposed to the user based on the analysis result. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 An example of a system 5 according to the present embodiment is schematically shown.

[0032] Figure 2 A functional structure of a robot 100 is schematically shown.

[0033] Figure 3 An example of an action flow of the robot 100 is schematically shown.

[0034] Figure 4 An example of a hardware structure of a computer 1200 is schematically shown.

[0035] Figure 5 An emotion map 400 that maps a plurality of emotions is shown.

[0036] Figure 6An emotion map 900 representing mapping a plurality of emotions.

[0037] Figure 7 An appearance diagram of a cloth doll according to another embodiment.

[0038] Figure 8 An example of a painting diary according to another embodiment.

[0039] Figure 9 A flowchart showing an example of a processing flow in a case where the painting diary P is created based only on the emotion of the robot 100.

[0040] Figure 10 A flowchart showing an example of a processing flow in a case where, in a case where the robot 100 is not activated or is placed, when the painting diary P is generated by only the emotion of the robot 100, a scene in which an event of one day is selected to create the painting diary P.

[0041] Figure 11 A flowchart showing an example of a process of determining the emotion value of the user.

[0042] Figure 12 A flowchart showing an example of a process when creating the painting diary.

[0043] Figure 13 A flowchart showing a processing flow in a case where the painting diary P is created in a case where the content that is not intended to be excluded as the painting diary.

[0044] Figure 14 A flowchart showing an example of a specific processing flow in a case where the robot 100 updates the information including the hobby of the user 10 as the history data 222 by analyzing the hint words generated when creating the painting diary P.

[0045] Figure 15 A flowchart showing an example of a specific processing flow in a case where the robot 100 makes a proposal of an action to the user 10.

[0046] Figure 16 A flowchart showing an example of a specific processing flow in a case where the robot 100 performs matching with other users using the information of the user 10 including the hobby by analyzing the hint words for creating the painting diary P.

[0047] Figure 17 A flowchart showing an example of a specific processing flow in a case where the robot 100 analyzes the lifestyle of the user 10 by analyzing the hint words generated when creating the painting diary P.

[0048] Figure 18 A diagram showing an example of displaying the ideal lifestyle and the analysis result by a pie chart. DETAILED DESCRIPTION

[0049] Hereinafter, the present disclosure will be described through embodiments of the invention, but the following embodiments do not limit the invention involved in the claims. In addition, the combination of features described in the embodiments is not necessarily all required for the solution means of the invention.

[0050] Figure 1 An example of a system 5 involved in the present embodiment is schematically shown. The system 5 is provided with a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In addition, in the description of the present embodiment, the user 10a, the user 10b, the user 10c, and the user 10d are sometimes collectively referred to as the user 10. In addition, the user 11a, the user 11b, and the user 11c are sometimes collectively referred to as the user 11. In addition, the user 12a and the user 12b are sometimes collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the functions of the robot 100 are mainly cited to describe the system 5.

[0051] The robot 100 has a conversation with the user 10, or provides an image to the user 10. At this time, the robot 100 cooperates with the server 300 or the like capable of communicating via the communication network 20, and performs the conversation with the user 10, the provision of the image to the user 10, or the like. For example, the robot 100 not only learns an appropriate conversation by itself, but also cooperates with the server 300 to learn in a manner to have a more appropriate conversation with the user 10. In addition, the robot 100 causes the server 300 to record image data or the like of the user 10 taken, and requests the image data or the like from the server 300 as needed, and provides it to the user 10.

[0052] In addition, the robot 100 has an emotion value indicating a kind of emotion of itself. For example, the robot 100 has an emotion value indicating the intensity of emotion of each of "joy", "anger", "sorrow", "happiness", "pleasure", "unhappiness", "comfort", "discomfort", "sadness", "excitement", "worry", "relief", "fulfillment", "emptiness", and "normal". The robot 100, for example, emits a sound at a faster speed when having a conversation with the user 10 in a state where the emotion value of excitement is large. In this way, the robot 100 can express its own emotion by action.

[0053] In addition, the robot 100 can be configured to determine the action of the robot 100 corresponding to the emotion of the user 10 by matching an article generation model using AI (Artificial Intelligence) with the emotion engine. Specifically, the robot 100 can be configured to recognize the action of the user 10, determine the emotion of the user 10 with respect to the action of the user, and determine the action of the robot 100 corresponding to the determined emotion.

[0054] More specifically, the robot 100 automatically generates the action content that the robot 100 should take with respect to the action of the user 10 using a pre-set article generation model in a case where the action of the user 10 is recognized. The article generation model can be interpreted as an algorithm and an operation for automatic dialogue processing based on text. The article generation model is known, for example, as disclosed in Japanese Patent Application Publication No. 2018-081444, chatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), and thus a detailed description thereof is omitted. Such an article generation model is constituted by a large-scale language model (LLM: Large Language Model).

[0055] In the above, the present embodiment can reflect the emotions of the user 10 and the robot 100 and various language information into the action of the robot 100 by combining the large-scale language model and the emotion engine. That is, according to the present embodiment, a synergistic effect can be obtained by combining the article generation model and the emotion engine.

[0056] In addition, the robot 100 has a function of recognizing the action of the user 10. The robot 100 recognizes the action of the user 10 by analyzing the facial image of the user 10 acquired by the camera function and the voice of the user 10 acquired by the microphone function. The robot 100 determines the action to be performed by the robot 100 on the basis of the recognized action of the user 10 or the like.

[0057] The robot 100 stores a rule for determining the action to be performed by the robot 100 on the basis of the emotion of the user 10, the emotion of the robot 100, and the action of the user 10, and performs various actions in accordance with the rule.

[0058] Specifically, the robot 100 has a reaction rule for determining an action of the robot 100 based on an emotion of the user 10, an emotion of the robot 100, and an action of the user 10. In the reaction rule, for example, in a case where the action of the user 10 is "laugh", the action of "laugh" is determined as the action of the robot 100. In addition, in the reaction rule, in a case where the action of the user 10 is "anger", the action of "apology" is determined as the action of the robot 100. In addition, in the reaction rule, in a case where the action of the user 10 is "question", the action of "answer" is determined as the action of the robot 100. In the reaction rule, in a case where the action of the user 10 is "sadness", the action of "approach" is determined as the action of the robot 100.

[0059] The robot 100 selects the action of "apology" determined by the reaction rule as the action to be performed by the robot 100 in a case where the action of the user 10 is identified as "anger" based on the reaction rule. For example, the robot 100, in a case where the action of "apology" is selected, performs an "apology" action while outputting a voice indicating an "apology" word.

[0060] In addition, it is stipulated that, in a case where the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, and "happiness" = 0) and the state of the user 10 satisfies the condition of "being alone and looking lonely", the change content of the emotion of the robot 100 to the emotion of "worry" and the action of "approach" can be performed.

[0061] The robot 100, based on the reaction rule, increases the emotion value of "sadness" of the robot 100 in a case where it is identified that the current emotion of the robot 100 is "normal" and the user 10 is in a state of being alone and looking lonely. In addition, the robot 100 selects the action of "approach" determined by the reaction rule as the action to be performed on the user 10. For example, the robot 100, in a case where the action of "approach" is selected, converts a word of "What's wrong?" indicating worry into a tone of worry and outputs it.

[0062] In addition, the robot 100, through the action, transmits user reaction information indicating a positive reaction from the user 10 to the server 300. In the user reaction information, for example, the user action of "anger", the action of the robot 100 of "apology", the case where the reaction of the user 10 is positive, and the attribute of the user 10 are included.

[0063] The server 300 stores the user reaction information received from the robot 100. Furthermore, the server 300 receives and stores the user reaction information from not only the robot 100 but also the robot 101 and the robot 102, respectively. Then, the server 300 analyzes the user reaction information from the robot 100, the robot 101, and the robot 102, and updates the reaction rule.

[0064] The robot 100 receives the updated reaction rule from the server 300 by inquiring the server 300 for the updated reaction rule. The robot 100 incorporates the updated reaction rule into the reaction rule stored in the robot 100. Thus, the robot 100 can incorporate the reaction rule obtained by the robot 101, the robot 102, and the like into its own reaction rule.

[0065] Figure 2 The functional structure of the robot 100 is schematically shown. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, an action recognition unit 234, an action determination unit 236, a storage control unit 238, an action control unit 250, a control object 252, and a communication processing unit 280.

[0066] The control object 252 includes a display device, a speaker, and an LED of an eye portion, and a motor or the like that drives an arm, a hand, a foot, and the like. The posture and the motion of the robot 100 are controlled by controlling the motors of the arm, the hand, the foot, and the like. Part of the emotion of the robot 100 can be expressed by controlling these motors. In addition, by controlling the light emission state of the LED of the eye portion of the robot 100, the expression of the robot 100 can also be expressed. Furthermore, the posture, the motion, and the expression of the robot 100 are examples of the attitude of the robot 100.

[0067] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects a sound and outputs sound data. Furthermore, the microphone 201 is provided at the head portion of the robot 100, and can have a function of performing binaural recording. The 3D depth sensor 202 continuously irradiates an infrared light pattern, and analyzes the infrared light pattern from an infrared image continuously captured by an infrared camera, thereby detecting the outline of an object. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image by visible light, and generates image information of the visible light. The distance sensor 204 detects the distance to an object by, for example, irradiating laser light, ultrasonic waves, or the like. Furthermore, the sensor unit 200 can include a clock, a gyro sensor, a touch sensor, a sensor for motor feedback, and the like, in addition to these.

[0068] Furthermore, Figure 2The elements of the robot 100 other than the control target 252 and the sensor section 200 among the elements of the robot 100 shown are examples of elements of an action control system that the robot 100 has. The action control system of the robot 100 takes the control target 252 as a target of control.

[0069] The storage section 220 contains reaction rules 221 and history data 222. The history data 222 includes a history of past emotional values and actions of the user 10. The history of emotional values and actions is recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage section 220 is installed by a storage medium such as a memory. A person DB that stores a face image of the user 10, attribute information of the user 10, and the like can also be contained. Furthermore, Figure 2 The functions of the elements of the robot 100 other than the control target 252, the sensor section 200, and the storage section 220 among the elements of the robot 100 shown can be realized by the CPU acting based on a program. For example, the functions of these elements can be installed as actions of the CPU by a basic software (OS) and a program that acts on the OS.

[0070] The sensor module section 210 includes a voice emotion recognition section 211, a speech understanding section 212, an expression recognition section 213, and a face recognition section 214. The sensor module section 210 is input with information detected by the sensor section 200. The sensor module section 210 analyzes the information detected by the sensor section 200 and outputs the analysis result to the user state recognition section 230.

[0071] The voice emotion recognition section 211 of the sensor module section 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition section 211 extracts a feature amount such as a frequency component of the voice and recognizes the emotion of the user 10 based on the extracted feature amount. The speech understanding section 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information indicating the speech content of the user 10.

[0072] The expression recognition section 213 recognizes the expression of the user 10 and the emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the expression recognition section 213 recognizes the expression and the emotion of the user 10 based on the shape, positional relationship, and the like of the eyes and the mouth.

[0073] The face recognition section 214 recognizes the face of the user 10. The face recognition section 214 recognizes the user 10 by matching a face image stored in a person DB (omitted from the drawing) with the face image of the user 10 captured by the 2D camera 203.

[0074] The user state recognition unit 230 recognizes the state of the user 10 on the basis of the information analyzed by the sensor module unit 210. For example, the processing relating to the perception is mainly performed using the analysis result of the sensor module unit 210. For example, the perception information such as "father is alone", "the probability that the father is not a smiling face is 90%", and the like is generated. The processing of understanding the meaning of the generated perception information is performed. For example, the meaning information such as "dad is alone and looks lonely" and the like is generated.

[0075] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 on the basis of the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230 are input to a neural network learned in advance, and an emotion value indicating the emotion of the user 10 is acquired.

[0076] Here, the emotion value indicating the emotion of the user 10 refers to a value indicating the positive or negative of the emotion of the user, and for example, if the emotion of the user is a bright emotion such as "joy", "happiness", "pleasure", "comfort", "excitement", "relief", and "fulfillment" that accompanies a feeling of pleasure and peace, a positive value is indicated, and the more bright the emotion, the larger the value becomes. If the emotion of the user is an unpleasant emotion such as "anger", "sorrow", "unhappiness", "unease", "sadness", "worry", and "emptiness", a negative value is indicated, and the more unpleasant the emotion, the larger the absolute value of the negative value becomes. In the case where the emotion of the user is not any of the above emotions ("normal"), a value of 0 is indicated.

[0077] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 on the basis of the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0078] The emotion value of the robot 100 includes an emotion value for each of a plurality of emotion categories, and for example, is a value indicating the intensity of each of "joy", "anger", "sorrow", and "happiness" (0 to 5).

[0079] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100 determined in correspondence with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the user state recognition unit 230.

[0080] For example, the emotion determination section 232 increases the emotion value of "sadness" of the robot 100 in a case where the user state recognition section 230 recognizes that the user 10 appears to be lonely. In addition, the emotion determination section 232 increases the emotion value of "joy" of the robot 100 in a case where the user state recognition section 230 recognizes that the user 10 becomes a smiling face.

[0081] Further, the emotion determination section 232 can further consider the state of the robot 100 to determine the emotion value indicating the emotion of the robot 100. For example, in a case where the amount of charge of the robot 100 is small, in a case where the surrounding environment of the robot 100 is pitch dark, or the like, the emotion value of "sadness" of the robot 100 can be increased. Furthermore, in a case where the user 10 continues to talk despite the small amount of charge, the emotion value of "anger" can be increased.

[0082] The action recognition section 234 recognizes the action of the user 10 on the basis of the information analyzed by the sensor module section 210 and the state of the user 10 recognized by the user state recognition section 230. For example, the information analyzed by the sensor module section 210 and the state of the user 10 recognized are input to a neural network learned in advance, the probabilities of a plurality of action categories (for example, "laugh", "anger", "question", "sadness") each determined in advance are acquired, and the action category with the highest probability is recognized as the action of the user 10.

[0083] As described above, in the present embodiment, the robot 100 acquires the utterance content of the user 10 on the basis of the user 10, but in the acquisition and use of the utterance content, and the like, the action control system of the robot 100 according to the present embodiment considers the protection of the personal information and the privacy of the user 10 in addition to the necessary consent from the user 10 in accordance with the law.

[0084] The action determination section 236 determines an action corresponding to the action of the user 10 recognized by the action recognition section 234, on the basis of the current emotional value of the user 10 determined by the emotion determination section 232, the history data 222 of the past emotional values determined by the emotion determination section 232 before the current emotional value of the user 10 is determined, and the emotional value of the robot 100. In the present embodiment, a case where the action determination section 236 uses the most recent one of the emotional values included in the history data 222 as the past emotional value of the user 10 is described, but the technology disclosed is not limited to this manner. For example, the action determination section 236 can use a plurality of recent emotional values as the past emotional value of the user 10, or can use an emotional value before a unit period of one day ago or the like. In addition, the action determination section 236 can determine an action corresponding to the action of the user 10, taking into account not only the current emotional value of the robot 100 but also the history of the past emotional values of the robot 100. The action determined by the action determination section 236 includes a gesture performed by the robot 100 or the utterance content of the robot 100.

[0085] As an action corresponding to the action of the user 10, the action determination section 236 according to the present embodiment determines an action of the robot 100 on the basis of the combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, the action of the user 10, and the reaction rule 221. For example, the action determination section 236 determines, as an action corresponding to the action of the user 10, an action for making the emotional value of the user 10 change positively, in a case where the past emotional value of the user 10 is a positive value and the current emotional value is a negative value.

[0086] An action of the robot 100 corresponding to the combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the action of the user 10 is determined in the reaction rule 221. For example, in a case where the past emotional value of the user 10 is a positive value, the current emotional value is a negative value, and the action of the user 10 is sadness, as an action of the robot 100, a combination of a gesture and utterance content at the time of a greeting that performs a combined gesture to encourage the user 10 is prescribed.

[0087] For example, in the reaction rule 221, the pattern of the emotional value of the robot 100 (4th power of 6 values of the value "0" to "5" of "joy", "anger", "sadness", "happiness", that is, 1296 patterns), the pattern of the combination of the past emotional value and the current emotional value of the user 10, and the entire combination of the action pattern of the user 10 are determined to determine the action of the robot 100. That is, for each pattern of the emotional value of the robot 100, the combination of the past emotional value and the current emotional value of the user 10 such as a negative value and a negative value, a negative value and a positive value, a positive value and a negative value, a positive value and a positive value, a negative value and ordinary, and ordinary and ordinary, and so on, the action of the robot 100 corresponding to the action pattern of the user 10 is determined for each of a plurality of combinations. In addition, the action determination section 236 can also shift to determine the action of the robot 100 using the history data 222 in the case where the user 10 has uttered a speech of the intention to continue the conversation from the past topic such as "I want to talk about the topic I talked about before".

[0088] In addition, in the reaction rule 221, at least one of the gesture and the speech content can be determined as the action of the robot 100 for each of the patterns of the emotional value of the robot 100 (1296 patterns). Alternatively, in the reaction rule 221, at least one of the gesture and the speech content can be determined as the action of the robot 100 for each of the patterns of the emotional value of the robot 100.

[0089] The intensity of each gesture included in the action of the robot 100 determined in the reaction rule 221 is determined in advance. The intensity of each speech content included in the action of the robot 100 determined in the reaction rule 221 is determined in advance.

[0090] The storage control section 238 determines whether or not to store the data including the action of the user 10 in the history data 222 on the basis of the intensity of the action determined in advance for the action determined by the action determination section 236 and the emotional value of the robot 100 determined by the emotion determination section 232.

[0091] Specifically, in the case where the integrated value of the intensity, that is, the sum of the sum of the emotional value for each of a plurality of emotional classifications of the robot 100, the intensity determined in advance for the gesture included in the action determined by the action determination section 236, and the intensity determined in advance for the speech content included in the action determined by the action determination section 236, is equal to or greater than a threshold value, it is determined to store the data including the action of the user 10 in the history data 222.

[0092] The storage control section 238 stores, in the case where it is determined to store data including the action of the user 10 in the history data 222, the action determined by the action determination section 236, information (for example, all of the surrounding information such as the sound, the image, the odor, and the like of the local area) analyzed by the sensor module section 210 from the current time point until a certain period before, and the state (for example, the expression, the emotion, and the like of the user 10) of the user 10 recognized by the user state recognition section 230 in the history data 222.

[0093] The action control section 250 controls the control target 252 on the basis of the action determined by the action determination section 236. For example, the action control section 250 outputs sound from a speaker included in the control target 252 in the case where the action determination section 236 determines an action including speech. At this time, the action control section 250 can also determine the speed of the voice output on the basis of the emotion value of the robot 100. For example, the greater the emotion value of the robot 100, the faster the speed of the voice output determined by the action control section 250. In this way, the action control section 250 determines the execution manner of the action determined by the action determination section 236 on the basis of the emotion value determined by the emotion determination section 232.

[0094] The action control section 250 can also recognize a change in the emotion of the user 10 with respect to the execution of the action determined by the action determination section 236. For example, the change in the emotion can be recognized on the basis of the voice, the expression of the user 10. In addition, the change in the emotion of the user 10 can also be recognized on the basis of the case where an impact is detected by the touch sensor included in the sensor section 200. It can also be recognized that the emotion of the user 10 becomes worse in the case where an impact is detected by the touch sensor included in the sensor section 200, or that the emotion of the user 10 becomes better in the case where it is judged from the detection result of the touch sensor included in the sensor section 200 that the reaction of the user 10 is laughter or joy, and the like. Information indicating the reaction of the user 10 is output to the communication processing section 280.

[0095] In addition, the action control section 250, after the execution of the action determined by the action determination section 236 in the execution manner determined on the basis of the emotion of the robot 100, the emotion determination section 232 causes the emotion value of the robot 100 to further change on the basis of the reaction of the user to the execution of the action. Specifically, the emotion determination section 232 increases the emotion value of "joy" of the robot 100 in the case where the reaction of the user to the execution of the action determined by the action determination section 236 in the execution manner determined by the action control section 250 is not bad, and the emotion determination section 232 increases the emotion value of "sorrow" of the robot 100 in the case where the reaction of the user to the execution of the action determined by the action determination section 236 in the execution manner determined by the action control section 250 is bad.

[0096] Further, the action control section 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, the action control section 250 controls the control object 252 so that the robot 100 performs a happy action in a case where the emotion value of "joy" of the robot 100 is increased. In addition, the action control section 250 controls the control object 252 so that the posture of the robot 100 becomes a drooping posture in a case where the emotion value of "sorrow" of the robot 100 is increased.

[0097] The communication processing section 280 is responsible for communication with the server 300. As described above, the communication processing section 280 transmits the user reaction information to the server 300. In addition, the communication processing section 280 receives the updated reaction rule from the server 300. When the communication processing section 280 receives the updated reaction rule from the server 300, the reaction rule 221 is updated.

[0098] The server 300 performs communication between the robot 100, the robot 101, and the robot 102 and the server 300, receives the user reaction information transmitted from the robot 100, and updates the reaction rule based on the reaction rule including the action that results in a positive reaction.

[0099] Figure 3 An example of an action flow related to determination of an action in the robot 100 is schematically shown. The action flow is repeatedly executed Figure 3 The action flow is shown. At this time, it is assumed that information analyzed by the sensor module section 210 is input. In addition, "S" in the action flow indicates a step that is executed.

[0100] First, in step S100, the user state recognition section 230 recognizes the state of the user 10 based on information analyzed by the sensor module section 210.

[0101] In step S102, the emotion determination section 232 determines an emotion value indicating the emotion of the user 10 based on information analyzed by the sensor module section 210 and the state of the user 10 recognized by the user state recognition section 230.

[0102] In step S103, the emotion determination section 232 determines an emotion value indicating the emotion of the robot 100 based on information analyzed by the sensor module section 210 and the state of the user 10 recognized by the user state recognition section 230. The emotion determination section 232 appends the determined emotion value of the user 10 to the history data 222.

[0103] In step S104, the action recognition section 234 recognizes the action classification of the user 10 based on information analyzed by the sensor module section 210 and the state of the user 10 recognized by the user state recognition section 230.

[0104] In step S106, the action determination section 236 determines the action of the robot 100 based on the combination of the current emotional value of the user 10 determined in step S102 and the past emotional values contained in the history data 222, the emotional value of the robot 100, the action of the user 10 recognized by the action recognition section 234, and the reaction rule 221.

[0105] In step S108, the action control section 250 controls the control object 252 based on the action determined by the action determination section 236.

[0106] In step S110, the storage control section 238 calculates a comprehensive value of the intensity based on the intensity of the action determined in advance for the action determined by the action determination section 236 and the emotional value of the robot 100 determined by the emotion determination section 232.

[0107] In step S112, the storage control section 238 determines whether the comprehensive value of the intensity is equal to or greater than a threshold value. In the case where the comprehensive value of the intensity is less than the threshold value, the data containing the action of the user 10 is not stored in the history data 222, and the process ends. On the other hand, in the case where the comprehensive value of the intensity is equal to or greater than the threshold value, the process proceeds to step S114.

[0108] In step S114, the action determined by the action determination section 236, the information analyzed by the sensor module section 210 from the current time point to a certain period before, and the state of the user 10 recognized by the user state recognition section 230 are stored in the history data 222.

[0109] As described above, with the robot 100, the emotional value indicating the emotion of the robot 100 is determined based on the user state, and it is determined whether to store the data containing the action of the user 10 in the history data 222 based on the emotional value of the robot 100. Thereby, it is possible to suppress the capacity of the history data 222 storing the data containing the action of the user 10. Also, for example, in the case where the robot 100 judges that the user state is the same state as that ten years ago after ten years, by reading in the history data 222 ten years ago, the robot 100 can present the user 10 with all the surrounding information such as the state of the user 10 (e.g., the expression of the user 10, the emotion, etc.) at that time ten years ago, even the data of the sound, the image, the odor, etc. of the place.

[0110] In addition, by the robot 100, it is possible to cause the robot 100 to perform an appropriate action with respect to the action of the user 10. In the past, an action including an expression and an appearance of the robot was determined by classifying the action of the user. In contrast, the robot 100 determines the current emotional value of the user 10, and performs an action with respect to the user 10 on the basis of the past emotional value and the current emotional value. Thus, for example, in a case where the user 10 who was in high spirits yesterday is in low spirits today, the robot 100 can utter a speech of "You were in high spirits yesterday, but what happened today?". In addition, the robot 100 can utter the speech with a gesture interposed. In addition, for example, in a case where the user 10 who was in low spirits yesterday is in high spirits today, the robot 100 can utter a speech of "You were in low spirits yesterday, but it seems that you are in high spirits today!". In addition, for example, in a case where the user 10 who was in high spirits yesterday is in higher spirits today than yesterday, the robot 100 can utter a speech of "You are in higher spirits today than yesterday, did something good happen compared to yesterday?". In addition, for example, the robot 100 can utter a speech of "You have been in stable mood recently, and you feel good" with respect to the user 10 whose emotional value is 0 or more and whose variation in emotional value is within a certain range.

[0111] In addition, for example, the robot 100 asks the user 10 "Did you finish the homework you said yesterday?", and in a case where an answer of "Yes" is obtained from the user 10, can utter an affirmative speech such as "Great!", and perform an affirmative gesture such as clapping hands or sticking out a thumb. In addition, for example, the robot 100 can utter an affirmative speech such as "You did a great job!" and perform the above-described affirmative gesture when the user 10 utters a speech of "The presentation you said the other day went well". In this way, by the robot 100 performing an action on the basis of the history of the state of the user 10, it is possible to expect that the user 10 feels close to the robot 100.

[0112] In the above-described embodiment, a case where the robot 100 recognizes the user 10 using the facial image of the user 10 is described, but the disclosed technology is not limited to this. For example, the robot 100 can recognize the user 10 using a voice uttered by the user 10, a mail address of the user 10, an ID of an SNS of the user 10, or an ID card on which a wireless IC tag held by the user 10 is built in.

[0113] In addition, the robot 100 is an example of an electronic device provided with an action control system. The application target of the action control system is not limited to the robot 100, and the action control system can be applied to various electronic devices. In addition, the functions of the server 300 can be installed by one or more computers. At least a part of the functions of the server 300 can be implemented by a virtual machine. In addition, at least a part of the functions of the server 300 can be installed on a cloud.

[0114] Figure 4 An example of a hardware structure of the computer 1200 functioning as the robot 100 and the server 300 is shown schematically. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus related to the present embodiment, or can cause the computer 1200 to execute operations associated with the apparatus related to the present embodiment or the one or more "parts", and / or can cause the computer 1200 to execute the process related to the present embodiment or a stage of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with several or all of the flowcharts and the blocks of the block diagrams described in the present specification.

[0115] The computer 1200 according to the present embodiment includes the CPU 1212, the RAM 1214, and the graphics controller 1216, which are connected to each other by the host controller 1210. The computer 1200 further includes the communication interface 1222, the storage device 1224, the DVD drive 1226, and an input / output unit such as an IC card drive, which are connected to the host controller 1210 via the input / output controller 1220. The DVD drive 1226 can be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 can be a hard disk drive, a solid state drive, or the like. The computer 1200 further includes the ROM 1230 and a conventional input / output unit such as a keyboard, which are connected to the input / output controller 1220 via the input / output chip 1240.

[0116] The CPU 1212 acts in accordance with a program stored in the ROM 1230 and the RAM 1214, thereby controlling the units. The graphics controller 1216 acquires image data generated by the CPU 1212 in a frame buffer or the like provided in the RAM 1214 or in itself, and causes the image data to be displayed on the display device 1218.

[0117] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like, and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card, and / or writes programs and data to the IC card.

[0118] The ROM 1230 stores therein a boot program or the like to be executed by the computer 1200 at activation and / or a program depending on the hardware of the computer 1200. The input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0119] A program is provided by a computer-readable storage medium such as the DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are examples of the computer-readable storage medium, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, bringing cooperation between the program and the various types of hardware resources described above. An apparatus or a method can be constituted by implementing the operation or processing of information according to the use of the computer 1200.

[0120] For example, in the case where communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded into the RAM 1214, instruct the communication interface 1222 to the communication processing based on the processing described in the communication program. The communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a recording medium such as the DVD-ROM 1227 or an IC card, under the control of the CPU 1212, transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0121] In addition, the CPU 1212 can read all or a necessary part of a file or a database stored in the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an external recording medium such as an IC card, to the RAM 1214, and execute various types of processing on the data on the RAM 1214. Subsequently, the CPU 1212 can write the processed data back to the external recording medium.

[0122] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium, and information processing is accepted. The CPU 1212 can perform various types of processing described throughout the present disclosure on data read from the RAM 1214, including various types of operations, information processing, conditional judgments, conditional branches, unconditional branches, search / replacement of information, and the like, specified by an instruction sequence of a program, and write the results back to the RAM 1214. In addition, the CPU 1212 can search for information in files, databases, and the like within the recording medium. For example, in a case where a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 can search for an entry that coincides with a condition in which the attribute value of the first attribute is specified, from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby acquire the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0123] The program or software module described above can be stored in a computer-readable storage medium on or near the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer-readable storage medium, and thereby the program can be provided to the computer 1200 via the network.

[0124] The blocks in the flowcharts and block diagrams in the present embodiment can represent stages of a process of performing operations or "parts" of an apparatus that have a role of performing operations. The specific stages and "parts" can be implemented by a dedicated circuit, a programmable circuit provided together with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided together with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit can include digital and / or analog hardware circuitry, and can include integrated circuits (ICs) and / or discrete circuits. The programmable circuit can include reconfigurable hardware circuitry such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including logical product, logical sum, exclusive OR, and not, and other logical operations, flip-flops, registers, and memory elements.

[0125] The computer readable storage medium can include any tangible device that can store instructions that are executed by an appropriate device to cause the computer readable storage medium having stored therein the instructions to be a product that comprises the instructions executable by a machine to produce an apparatus that implements, for example, the procedures of the flow charts or block diagrams. Examples of computer readable storage media include an electronic storage media, a magnetic storage media, an optical storage media, an electromagnetic storage media, a semiconductor storage media, and the like. More specific examples of the computer readable storage media include a floppy disk, a magnetic hard disk, a magnetic tape, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray Disc (registered trademark), a memory stick, an integrated circuit card, and the like.

[0126] The computer readable instructions can include any one of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, such as Smalltalk, JAVA (registered trademark), C++, and the like, and a previous procedural programming language including a "C" programming language or the like.

[0127] The computer readable instructions can be provided to a processor or programmable circuit of a general purpose computer, a special purpose computer, or other programmable data processing apparatus to cause the processor or programmable circuit of the general purpose computer, the special purpose computer, or other programmable data processing apparatus to perform a function of the flow chart or block diagram by local or via a wide area network (WAN) such as a local area network (LAN), the Internet, and the like. Examples of the processor include a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, and the like.

[0128] The above describes the present disclosure using embodiments, but the technical scope of the present disclosure is not limited to the range described in the above embodiments. It is obvious to those skilled in the art that various changes or modifications can be made to the above embodiments. The embodiments to which such changes or modifications are applied are included in the technical scope of the present disclosure according to the recitations of the claims.

[0129] It should be noted that, unless explicitly stated as "before" or "in advance," and unless the output of a previous process is used in a later process, the execution order of the actions, processes, steps, and stages in the apparatus, system, program, and method shown in the claims, specification, and drawings can be implemented in any order. Even if terms such as "firstly" or "next" are used for convenience in describing the flow of actions in the claims, specification, and drawings, it does not mean that they must be implemented in that order.

[0130] (Other implementation methods) As another implementation, the robot 100 described above can also be applied to robots mounted on electrical appliances, computers, automobiles, and motorcycles. Additionally, the robot 100 can be mounted on a cloth doll, or it can be applied to a control device connected wirelessly or wired to a control object device (speaker, camera, microphone) mounted on the cloth doll. Specifically, other implementations are configured as follows. For example, the robot 100 can also be applied to a cohabitant who spends their daily life with user 10 and creates a drawing diary P based on events involving dialogue with user 10 (specifically, Figure 7 The cloth doll shown is 100N. In this embodiment, it is an animal-type cloth doll, but the cloth doll is not limited to animal type and can also be other characters.

[0131] like Figure 7 As shown, in this embodiment (other embodiments), the cloth doll 100N is a bear-shaped doll covered with a soft-looking cloth. Furthermore, although not shown, a microphone 201 of a sensor unit 200 is disposed as an input / output device in the space formed inside, corresponding to the ear 54 (see reference). Figure 2 A 2D camera 203 with a sensor unit 200 is configured in the part corresponding to the eye 56 (see reference). Figure 2 ), and in the part corresponding to mouth 58, there is a control object 252 (see reference). Figure 2 The microphone 201 and the speaker 60 are not necessarily separate units; they can also be an integrated unit. In the case of an integrated unit, it can be positioned where the doll 100N can naturally hear the speech, such as at the nose position.

[0132] In addition, the robot 100 stores, for example, events including conversations with the user 10 who is a child. Specifically, events of a day acquired by the microphones 201, the 2D camera 203 are stored in the storage section 220. Further, as events, the history data 222 in which past emotional values and actions of the user 10 are stored can also be used as event data. Then, from the events stored at the time point of the end of the day or the like, a predetermined scene is selected, and the selected scene is texturized. Here, the scene that is texturized is a scene in which the emotion of the user determined by the emotion determination section 232 or the emotion of the robot satisfies a predetermined selection criterion. The selection criterion includes a case where the emotion of the user 10 and the emotion of the robot 100 are the strongest, or a case where a specific emotion, for example, "joy", among a plurality of emotions is the strongest with respect to the emotion of the user 10 or the emotion of the robot 100, or a case where the number of smiles of the user is the largest. For example, based on the history data 222, a scene in which the sum of the emotional value of the user 10 and the emotional value of the robot 100 is the strongest is selected, the action of the user 10 stored in the history data 222 is acquired in the selected scene, and a text indicating the acquired action of the user 10 is input to the image generation model. Then, for example, in a case where the scene is a scene of going to the zoo, it is texturized as "Teddy bear and user go to the zoo" or the like. In addition, the action determination section 236 receives an image indicating the user 10 in advance, inputs a text indicating the user of the image and a text indicating the selected scene to the image generation model, and generates an image indicating a picture. The text indicating the user is image data obtained from the photographed event, and in particular, is obtained by texturizing the appearance of the user, such as clothes, a hairstyle, and the like, of the user, and makes the character appearing in the drawing diary P closer to the user. For example, in a case where the clothes worn by the user are a blue T-shirt, it is texturized as "wearing a blue T-shirt" or the like.

[0133] In addition, the robot 100 creates a drawing diary P with respect to the texturized scene by a pre-set image generation AI. The image generation AI is known as disclosed in DALL·E2 (Internet search <URL: https: / / openai.com / product / dall-e-2>), for example, and thus a detailed description thereof is omitted. Here, the image generation AI is an example of the image generation model. In addition, the image generation AI can be stored in another device connected to the robot 100 through a network in addition to the case where it is stored in the storage section 220.

[0134] Specifically, Robot 100 inputs text representing the user and text representing the selected scene into the image generation model to generate an image representing the scene. For example, inputting phrases like "a drawing diary depicting a teddy bear at the zoo and the user's crayon drawing" into the image generation AI produces drawing P1 of the drawing diary P as an image representing the scene. Additionally, inputting phrases like "taking the event with the teddy bear at the zoo as a diary entry" into the article generation model produces the title P2, article P3, etc., of the drawing diary P. Figure 8 The image shows an example of a drawing diary P created by an image generation AI and a text generation model. For example... Figure 8 As shown, the image generation AI creates a drawing P1, and the article generation model creates a title P2 and an article P3. Then, robot 100 creates a drawing diary P that combines the drawing P1, title P2, and article P3. Furthermore, the drawing P1 created by the image generation AI and the title P2 and article P3 created by the article generation model can be generated in multiple patterns, and these can be specified by the user. Additionally, the title P2 and article P3 are not limited to content created from articles; they can also be input by the user 10.

[0135] Then, the robot 100 outputs the created drawing diary P, and sends it, for example, via email to a predetermined person including user 10. Preferably, the predetermined person is user 10's parents, grandparents living far away, etc.

[0136] Furthermore, in this embodiment (other embodiments), a bear plush toy 100N is exemplified, but it can also be other animals, or even humanoid dolls. Additionally, the outer skin can be replaced. Furthermore, the material of the outer skin is not limited to cloth; it can also be other materials such as soft vinyl, but a soft material is preferred.

[0137] Alternatively, a display can be installed on the surface of the cloth doll 100N, and a control object 252 can be added to provide information to the user 10 visually. For example, a display can be provided on the abdomen to display the drawing diary P. Alternatively, the eyes 56 can be used as a projector to project the drawing diary P onto a wall. Alternatively, the drawing diary P can be associated with a calendar function, and when a date is selected through the calendar function, the drawing diary P for that date can be displayed.

[0138] Robot 100 (including Robot 100 with the appearance of a cloth doll 100N). Perform the process of creating a drawing diary P by following steps 1 to 4.

[0139] (Step 1) The robot 100 acquires the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, the history data 222, and the data of the event. Specifically, the same processing as in steps S100 to S103 described above is performed, and the state of the user 10, the emotional value of the user 10, the emotional value of the robot 100, the history data 222, and the data of the event are acquired.

[0140] (Step 2) The robot 100 selects a scene to be set as the picture diary P.

[0141] Specifically, the action determination section 236 reviews the events of the day at the end of the day, and selects a scene to be set as the picture diary P on the basis of the history data 222.

[0142] (Step 3) The robot 100 generates the picture diary P.

[0143] Specifically, the action determination section 236 texturizes the scene selected in step 2. For example, the text "Teddy bear and user at zoo" is obtained. Then, a fixed sentence such as "Picture diary drawn with crayon" is appended to the text, and input to the image generation AI, and a picture P1 of the picture diary P is created. In addition, the action determination section 236 receives a designation of the title and the article of the picture diary from the user 10, and acquires the title P2 and the article P3 of the picture diary P. The action determination section 236 generates the picture diary P by combining the picture P1 of the picture diary P, the title P2 of the picture diary P, and the article P3 of the picture diary P.

[0144] (Step 4) The robot 100 outputs the generated picture diary P to a person determined in advance.

[0145] In this way, the robot 100 can perform the processing of creating the picture diary P on the basis of the history data 222.

[0146] Further, the robot 100 can also analyze the prompt word input to the image generation model used when creating the picture diary P, extract the information of the user 10 including the hobby of the user 10, and update the history data 222. For example, when creating the picture diary P, the text representing the action of the user 10 stored in the history data 222 is generated as a prompt word input to the image generation model, and therefore the text is analyzed to extract the information of the user 10 and update the history data 222. Then, the content proposed to the user 10 can also be determined and proposed on the basis of the updated information of the user 10. Furthermore, the information required for the proposal can also be acquired and provided to the user 10.

[0147] For example, in a case where the robot 100 sets a case where the user 10 ate strawberries with a smile as a drawing diary P, the robot 100 stores information such as "like strawberries" of the user in the history data. Then, when it becomes a season of strawberry picking, the robot 100 proposes to the user 10 "whether to go to strawberry picking?". Further, the robot 100 acquires information of a web page, and conveys to the user relevant information related to "strawberries" such as a date, a fee, and the like of "strawberry picking".

[0148] Here, a specific process in a case where the robot 100 analyzes a cue generated at the time of making the drawing diary P, and updates information including the hobby of the user 10 to the history data 222 will be described. Figure 14 is a flowchart indicating an example of a specific process flow in a case where the robot 100 analyzes a cue generated at the time of making the drawing diary P, and updates information including the hobby of the user 10 to the history data 222. Further, Figure 14 The process of

[0149] In step S700, the action determination section 236 acquires a text indicating an action of the user 10 used in the making of the drawing diary P.

[0150] In step S702, the action determination section 236 analyzes the acquired text. For example, with respect to a scene selected as the drawing diary P, since a scene in which the emotional value of the user 10 and the robot 100 is strong is selected, the content of the hobby, the preference, and the like of the user is extracted from the words included in the text used at the time of making the drawing diary P. For example, a word related to the hobby is set in advance, and a word corresponding to the hobby is extracted from the text as information including the hobby.

[0151] In step S704, the action determination section 236 determines whether or not there is information including the hobby of the user 10. This determination determines, for example, whether or not a word related to the hobby is extracted as a result of analyzing the text. In a case where this determination is negative, the process is ended, and in a case where it is affirmative, the processing is transferred to step S706.

[0152] In step S706, the action determination section 236 updates the information of the user 10. For example, the history data 222 is updated with the information related to the hobby of the user 10 extracted. Thereby, the information of the user 10 such as the hobby, the interest preference, the action, the lifestyle, and the like is updated, and thus it is possible to make a proposal of an action in accordance with the hobby, the interest, the trend of the user by the action control section 250.

[0153] Next, a specific process in a case where the robot 100 makes a proposal of an action to the user 10 will be described. Figure 15is a flowchart of an example of a specific processing flow in a case where the robot 100 makes a proposal of an action to the user 10. Figure 15 The processing of the history data 222 starts at a predetermined time point such as the end of the day.

[0154] In step S800, the action determination section 236 acquires the history data 222. For example, the updated information of the user 10 (information related to the hobby of the user 10) is acquired.

[0155] In step S802, the action determination section 236 analyzes the history data 222. For example, a word corresponding to the hobby is extracted from the updated information of the user 10, and information related to the hobby is collected. Specifically, the user 10 of the history data 222 collects associated information (for example, season, etc.) associated with "strawberry" from a website based on the information of the user 10 such as "likes strawberry".

[0156] In step S804, the action determination section 236 determines whether it is a proposal time point. The determination is based on the collection result of the associated information, and determines whether it is a time point to make a proposal on the information of the user 10. For example, in a case where the user 10 has the information of the user 10 such as "likes strawberry" in the history data 222, it is determined whether it is the season of "strawberry picking" based on the associated information collected by the analysis of the history data. The processing is directly ended in a case where the determination is negative, and is transferred to step S804 in a case where the determination is positive.

[0157] In step S806, the action determination section 236 makes a proposal to the user 10 based on the collected associated information. For example, the action determination section 236 inputs "make a proposal related to strawberry" or the like to the article generation model, and creates a document of making a proposal related to "strawberry". Thus, the robot 100 makes a proposal of "want to go to pick strawberries?" or the like by the utterance from the speaker controlled by the action control section 250.

[0158] In step S808, the action determination section 236 collects associated information required for the proposed action. For example, the information of the web page is acquired, and the associated information of the strawberry picking date, the cost, the nearby place, and the like is collected.

[0159] In step S810, the action determination section 236 prompts the user 10 of the collected associated information. For example, the action control section 250 controls the robot 100 based on the action determined by the action determination section 236, and thus the robot 100 utters "strawberry picking is available at a nearby strawberry farm from April with a cost of ○○ yen" or the like. Alternatively, the action determination section 236 can transmit the collected information to the mobile phone of the user 10 registered in advance via the communication processing section 280.

[0160] Thus, by providing the web page information, the user 10 can be provided with information having high real-time performance.

[0161] In addition, the user's hobby is grasped from the cue words made for the picture diary P, but since the cue words are made by the robot 100, the hobby or specialty that the user does not know is known.

[0162] Further, the user 10's information including the hobby updated by analyzing the cue words made for the picture diary P can also be used to match with other users. For example, the SNS (Social Networking Service) is cooperated to match with each other having similar hobbies. In addition, in addition to the hobby, based on the history data 222, people having similar values such as interest preference, action, lifestyle, and the like can be matched with each other. For example, by analyzing the cue words, people who are aggressive are matched with each other, or people who like strawberries are matched with each other.

[0163] The scene that is texted as the cue word is a scene that elicits positive emotions of the user 10, and thus is a scene that extracts the values of the user 10. Therefore, the values of the user can be extracted by analyzing the cue words made for the picture diary P.

[0164] For example, the cue words made for the picture diary P can be used for dating or recruitment, and the like. In addition, an advertisement that matches the values of the user 10 can also be displayed.

[0165] Here, a specific process when the robot 100 uses the user 10's information including the hobby updated by analyzing the cue words made for the picture diary P to match with other users is described. Figure 16 is a flowchart showing an example of a flow of a specific process when the robot 100 uses the user 10's information including the hobby to match with other users by analyzing the cue words made for the picture diary P. Further, Figure 16 The process of can be started at the time of making the picture diary P, after making the picture diary P, and the like, for example. Alternatively, can be started at a time point set in advance by the user 10, or can be started at another time point.

[0166] In step S900, the action determination section 236 acquires the history data 222. That is, the user 10's information saved in the updated history data is acquired by analyzing the cue words made for the picture diary P.

[0167] In step S902, the action determination section 236 extracts the values of the user 10 from the information of the user 10. For example, at least one of the values of the user 10 in the hobby, the interest preference, the action, and the lifestyle is extracted from the information of the user 10 stored in the history data 222.

[0168] In step S904, the action determination section 236 determines whether a plurality of kinds of values are extracted. In the case where the determination is affirmative, the processing proceeds to step S906, and in the case where the determination is negative, the processing proceeds to step S908.

[0169] In step S906, the action determination section 236 focuses on one value. That is, one value is focused on from among the plurality of kinds of values of the user 10 extracted.

[0170] In step S908, the action determination section 236 searches for the SNS user of the value corresponding to the value of the user 10. For example, the action determination section 236 accesses a website via the communication processing section 280, and searches for the user of the value corresponding to the value of the user 10 from among the users of the SNS registered in advance.

[0171] In step S910, the action determination section 236 determines whether the search is completed for all the values of the user 10 extracted. In the case where the determination is negative, the processing proceeds to step S912, another value is focused on, and the processing returns to step S908, and the above-described processing is repeated. On the other hand, in the case where the determination is affirmative, the processing proceeds to step S914.

[0172] In step S914, the action determination section 236 outputs the search result and ends the series of processing. For example, the action determination section 236 can output the search result to the mobile phone or the like of the user 10 registered in advance via the communication processing section 280. Alternatively, the robot 100 can be controlled via the action control section 250, and the search result can be output by utterance.

[0173] In this way, the values of the user 10 can be extracted by analyzing the cue words for the painting diary P. Also, since the SNS user of the value corresponding to the value of the user 10 extracted is searched for, it is possible to match active people to each other, to match people who like strawberries to each other, and the like.

[0174] In addition, the robot 100 can analyze the prompt words input to the image generation model when creating the drawing diary P, and analyze the lifestyle of the user 10 such as the frequency of events, diet, and tidying of the room. For example, when creating the drawing diary P, as the prompt words input to the image generation model, the text indicating the action of the user 10 stored in the history data 222 is used, and thus the lifestyle of the user 10 is analyzed by analyzing the text indicating the action of the user 10. The analysis result can also be scored and prompted to the user 10. By scoring and prompting the analysis result of the lifestyle, it is possible to easily understand the part that is insufficient in order to aim for an ideal lifestyle. In addition, the ideal lifestyle is set in advance by the user 10.

[0175] For example, by the analysis of the prompt words, it is known that the user 10 often goes camping and cannot tidy the room. Therefore, the activity item such as hobby has 90 points, and the tidying item such as life has 30 points. The user 10 wants to clean the room, but 3 months have passed in the process of procrastination. The user 10 notices that the cleaning of the room is a pressing matter by being urged to tidy up by the robot 100.

[0176] In this way, by analyzing the prompt words, it is possible to inform the user 10 of the improvement point of the daily life. In addition, by scoring the analysis result, it is possible to notice the point that the user 10 has not noticed by an objective numerical value.

[0177] In the analysis of the lifestyle, for example, the prompt words can be acquired and the words and the like included in the prompt can be analyzed every time the prompt words for the drawing diary P are generated, and thus the content of the prompt words is classified into a predetermined lifestyle item. The lifestyle item can be set in advance by the user 10, for example, and can be classified into a general lifestyle item such as work, life, and hobby, and can be more finely classified into a lifestyle item. In addition, the lifestyle item can be prepared in advance, or can be selected by the user 10.

[0178] In addition, as an example of a method of scoring, for example, a proportion corresponding to an ideal lifestyle set in advance by the user 10 can be calculated with respect to the classification result of the lifestyle item, and thus scoring is performed for each item.

[0179] Here, a specific process in a case where the lifestyle of the user 10 is analyzed by analyzing the prompt words generated when creating the drawing diary P will be described. Figure 17 is a flowchart indicating an example of a specific process flow in a case where the robot 100 analyzes the lifestyle of the user 10 by analyzing the prompt words generated when creating the drawing diary P. In addition, Figure 17 The process of is started, for example, in a case where the robot 100 generates the prompt words when creating the drawing diary P.

[0180] In step S1000, the action determination section 236 acquires the cue words for the picture diary P. That is, the cue words generated for the production of the picture diary P are acquired. Specifically, the text indicating the action of the user 10 is acquired.

[0181] In step S1002, the action determination section 236 analyzes the acquired cue words. For example, the words and the like included in the acquired text indicating the action of the user 10 are analyzed.

[0182] In step S1004, the action determination section 236 classifies into the predetermined lifestyle items. For example, the content of the cue words is classified into the predetermined lifestyle items in accordance with the words and the like included in the cue words.

[0183] In step S1006, the action determination section 236 stores the classification result of the lifestyle items in the history data 222.

[0184] In step S1008, the action determination section 236 determines whether it is the notification time point of the analysis result of the lifestyle. The determination can determine whether it is the preset date and time, or can determine whether the predetermined period has passed from the last notification, for example. The series of processes are ended in the case where the determination is negative, and proceeds to step S1010 in the case where the determination is affirmative.

[0185] In step S1010, the action determination section 236 scores based on the classification result of the lifestyle and the preset ideal lifestyle. For example, each item is scored by calculating the proportion corresponding to the ideal lifestyle preset by the user 10 with respect to the classification result stored in the history data 222.

[0186] In step S1012, the action determination section 236 outputs the analysis result of the lifestyle. For example, the action determination section 236 outputs the analysis result of the lifestyle to the action control section 250, and thereby the action control section 250 controls the robot 100 so that the robot 100 speaks the analysis result of the lifestyle. Alternatively, the action determination section 236 can transmit the analysis result of the lifestyle to the mobile phone of the user 10 registered in advance via the communication processing section 280. In the case where the analysis result of the lifestyle is transmitted to the mobile phone of the user 10, for example, as shown in FIG. 14, the ideal lifestyle and the analysis result can be transmitted and displayed on the mobile terminal of the user 10 by a pie chart and the like. Alternatively, the scores of each of the lifestyle items can be transmitted and displayed on the mobile terminal of the user 10. Figure 18

[0187] ​Further, in step S1012, the action determination section 236 can further determine and output the content proposed to the user 10 based on the analysis result when outputting the analysis result of the lifestyle. As the proposed content, a point lacking with respect to the ideal lifestyle can be extracted from the analysis result. For example, according to the analysis result of the lifestyle, in the case of being insufficient in tidying up, a proposal to urge tidying up is determined since the tidying up is insufficient. Then, the action control section 250 controls the robot 100 to control the utterance to make the proposal to urge tidying up.

[0188] Further, the emotion determination section 232 can determine the emotion of the user in accordance with a specific mapping. Specifically, the emotion determination section 232 can determine the emotion of the user in accordance with an emotion mapping (refer to Fig. 4) as a specific mapping. Figure 5

[0189] Figure 5 is a diagram showing an emotion mapping 400 that maps a plurality of emotions. In the emotion mapping 400, emotions are arranged radially from the center in concentric circles. The closer to the center of the concentric circles, the more emotions in the original state are arranged. Emotions representing states, actions generated by the mood are arranged on the outer side of the concentric circles. Emotion is a concept that also includes mood, mental state. Emotions generated by reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions induced in the situation judgment are arranged on the right side of the concentric circles. On the upper and lower directions of the concentric circles, emotions generated from reactions occurring in the brain and induced in the situation judgment are arranged. In addition, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "unpleasantness" is arranged on the lower side. In this way, in the emotion mapping 400, a plurality of emotions are mapped based on the configuration of the emotions generated, and emotions that are likely to be generated at the same time are mapped in the vicinity.

[0190] (1) For example, in a case where the emotion engine as the emotion determination section 232 of the robot 100 detects emotions at around 100 msec, the determination of the reaction action (for example, echo) of the robot 100 can be set to a time point at least as frequently as the detection frequency (100 msec) of the emotion engine, or can be set to a time point earlier than that. The detection frequency of the emotion engine can be interpreted as a sampling rate.

[0191] By detecting emotions at around 100 msec, the reaction action (for example, echo) is performed in real time, and a conversation that is not unnatural echo but natural observation of emotions can be realized. According to the directionality and the degree (intensity) of the mandala of the emotion mapping 400 of the robot 100, the reaction action (echo, etc.) is performed. In addition, the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and can be changed according to the situation (the case where the movement is performed, etc.), the age of the user, and the like.

[0192] ​(2) The directionality of the emotion and the intensity of the degree thereof can be set in advance in correspondence with the emotion map 400, and the accompanying action and the strength of the accompanying action can be set. For example, in a case where the robot 100 feels a sense of stability, security, and the like, the robot 100 nods and continues to speak. In a case where the robot 100 feels a sense of unease, confusion, strangeness, and the like, the robot 100 can tilt the head or can stop shaking the head.

[0193] These emotions are distributed in the direction of the 3 o'clock of the emotion map 400, and are usually moved around the vicinity of security and unease. In the right half of the emotion map 400, the situation recognition is more dominant than the internal feeling, and thus a calm impression is given.

[0194] (3) In a case where the robot 100 is praised and feels pleasure, a filler word such as "ah-" can be added before the dialogue, and in a case where the robot 100 is scolded and feels pain, a filler word such as "woof!" can be added before the dialogue. In addition, the robot 100 can also include a body reaction such as a crouching action while saying "woof!". These emotions are distributed in the vicinity of the 9 o'clock of the emotion map 400.

[0195] (4) In the left half of the emotion map 400, the internal feeling (reaction) is more dominant than the situation recognition. Thus, an unexpected impression can be given.

[0196] In a case where the robot 100 feels a sense of conviction as the internal feeling (reaction) and also feels a sense of favor in the situation recognition, the robot 100 can deeply nod while observing the other party, and can also say "um". In this way, the robot 100 can generate an action of giving a balanced favor to the other party, that is, an action of allowing and tolerating the other party. Such an emotion is distributed in the vicinity of the 12 o'clock of the emotion map 400.

[0197] On the contrary, in a case where the robot 100 feels a sense of discomfort as the internal feeling (reaction) and also feels a sense of aversion in the situation recognition, the robot 100 can swing the head horizontally, and if the degree reaches a sense of loathing, the robot 100 can make the LED of the eyes red and stare at the other party. Such an emotion is distributed in the vicinity of the 6 o'clock of the emotion map 400.

[0198] (5) The inside of the emotion map 400 indicates the heart, and the outside of the emotion map 400 indicates the action, and thus the emotion becomes more visible (is expressed as an action) as the outside of the emotion map 400 is approached.

[0199] (6) In a case where one feels comfort, which is distributed around 3 o'clock of the emotion map 400, while listening to a person speaking, the robot 100 slightly shakes the head vertically to the extent of "um-um", but when it becomes the one of love around 12 o'clock, it can perform a strong nod such as shaking the head vertically deeply.

[0200] The emotion determination section 232 inputs the information analyzed by the sensor module section 210 and the state of the user 10 recognized to the neural network learned in advance, acquires the emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is a neural network learned in advance based on a plurality of learning data which are combinations of the information analyzed by the sensor module section 210 and the state of the user 10 recognized and the emotion values indicating each emotion shown in the emotion map 400. Further, this neural network is learned in such a way that the emotions arranged in the vicinity have values close to each other as in the emotion map 900 shown in FIG. 9. Figure 6 Figure 6 In the emotion map 400 shown in FIG. 4, an example in which a plurality of emotions such as "comfort", "calm", and "solid" become close emotion values is shown.

[0201] Further, the emotion determination section 232 can determine the emotion of the robot 100 in accordance with a specific map. Specifically, the emotion determination section 232 inputs the information analyzed by the sensor module section 210, the state of the user 10 recognized by the user state recognition section 230, and the state of the robot 100 to the neural network learned in advance, acquires the emotion values indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is a neural network learned in advance based on a plurality of learning data which are combinations of the information analyzed by the sensor module section 210, the state of the user 10 recognized, and the state of the robot 100 and the emotion values indicating each emotion shown in the emotion map 400. For example, in a case where it is recognized from the output of the touch sensor (omitted from illustration) that the robot 100 is stroked by the user 10, the neural network is learned based on learning data indicating an emotion value "3" of becoming "happy", and in a case where it is recognized from the output of the acceleration sensor (omitted from illustration) that the robot 100 is slapped by the user 10, the neural network is learned based on learning data indicating an emotion value "3" of becoming "angry". Further, this neural network is learned in such a way that the emotions arranged in the vicinity have values close to each other as in the emotion map 900 shown in FIG. 9. Figure 6

[0202] The action determination section 236 generates the action content of the robot by appending a fixed sentence for asking the action content of the robot corresponding to the action of the user to the text indicating the action of the user, the emotion of the user, and the emotion of the robot, and inputting to an article generation model having a dialogue function. ​​

[0203] For example, the action determination section 236 acquires a text indicating the state of the robot 100 using an emotion table shown in Table 1 in accordance with the emotion of the robot 100 determined by the emotion determination section 232. Here, in the emotion table, each emotion value is given an index number by each category of emotion, and a text indicating the state of the robot 100 is stored for each index number.

[0204] In a case where the emotion of the robot 100 determined by the emotion determination section 232 corresponds to the index number "2", a text such as "a very happy state" is obtained. Further, in a case where the emotion of the robot 100 corresponds to a plurality of index numbers, a plurality of texts indicating the state of the robot 100 are obtained.

[0205] In addition, for the emotion of the user 10, an emotion table shown in Table 2 is also prepared in advance.

[0206] Here, in a case where the action of the user is to say "make a painting diary", the emotion of the robot 100 is the index number "2", and the emotion of the user 10 is the index number "3", the article generation model is input with "the robot is in a very happy state. The user is generally in a happy state. The user said'make a painting diary'. How should the robot respond?", and the action content of the robot is acquired. The action determination section 236 determines the action of the robot in accordance with the action content.

[0207] [Table 1]

[0208] [Table 2]

[0209] In this way, the robot 100 can change the action of the robot in accordance with the index number corresponding to the emotion of the robot, and thus the user has an impression that the robot has a mind, and the action such as approaching the robot is promoted.

[0210] In addition, the action determination section 236 can also add a text indicating the content of the history data 222 in addition to the texts indicating the action of the user, the emotion of the user, and the emotion of the robot, and on this basis, add a fixed sentence for asking the action content of the robot corresponding to the action of the user to the article generation model having a dialogue function, and thereby generate the action content of the robot. By this, the robot 100 can change the action of the robot in accordance with the history data indicating the emotion and the action of the user, and thus the user has an impression that the robot has an individuality, and the action such as approaching the robot is promoted. In addition, the emotion and the action of the robot can also be included in the history data.

[0211] However, the robot 100 is sometimes not activated or placed. In this case, since there is no conversation with the user 10, the user 10 sometimes does not have a strong emotion, and thus the drawing diary P can also be made only by the emotion of the robot 100. The emotion of the robot 100 in this case can be determined randomly or can follow a personality setting in a case where the robot is given a personality (for example, gentle, angry, and the like).

[0212] In addition, the emotion of the robot 100 can also be determined based on a conversation with the user 10 for a predetermined period (for example, the past week). For example, if there is a history in which the robot 100 quarrels with the user 10, the robot 100 can be made to have a sad emotion, and in a case where a happy topic is heard from the user 10, the robot 100 can be made to have a happy emotion.

[0213] For example, in a case where the emotion of the robot 100 is determined randomly, an image of the robot 100 and text corresponding to the image can also be prepared in advance for each emotion of the robot 100, and the drawing diary P can be made by the image and the text corresponding to the emotion of the robot 100 determined randomly.

[0214] Figure 9 is an example of a flowchart of a flow of processing when the drawing diary P is made only by the emotion of the robot 100. Furthermore, Figure 9 The processing of is started at a predetermined time or the like at the end of the day on a day when the user 10 does not have a conversation with the robot 100, for example.

[0215] In step S200, the emotion determination section 232 determines the emotion of the robot 100. The emotion of the robot can be determined randomly or can follow a personality setting in a case where the robot is given a personality (for example, gentle, angry, and the like), for example.

[0216] In step S202, the action determination section 236 makes the drawing diary P based on the emotion of the robot 100. For example, an image of the robot 100 and text corresponding to the image can be prepared in advance for each emotion of the robot 100, and the drawing diary P can be made by the image and the text corresponding to the determined emotion of the robot 100.

[0217] In step S204, the action determination section 236 outputs the made drawing diary P. For example, the action determination section 236 transmits it to a predetermined person including the user 10 by mail or the like via the communication processing section 280.

[0218] Furthermore, when the robot 100 is not activated or placed, and the drawing diary P is generated solely based on the robot 100's emotions, the scene for creating the drawing diary P can be selected using the events of the day (sounds and images around the robot 100) acquired by the microphone 201 and the 2D camera 203 stored in the storage unit 220. For example, a scene that meets predetermined selection criteria can be selected from the events of the day stored in the storage unit 220, and the text representing the robot's emotions and the events corresponding to the selected scene can be input into an image generation model to generate an image representing the scene and output the image. Additionally, the text representing the robot 100's emotions and the events can be prepared in advance from various texts, and the text corresponding to the robot 100's emotions can be selected and generated using an article generation model.

[0219] For example, if the 2D camera 203 of robot 100 is facing outwards from the house, a drawing diary is created along with the weather. As an example, based on a predetermined selection criterion, a rain scene is selected from the events of a day stored in storage unit 220 by recognizing weather-related sounds such as "It's raining!", and an image is generated by inputting text such as "Drawing diary of a teddy bear sadly playing in the rain with an umbrella" into an image generation model.

[0220] Additionally, if the robot 100's 2D camera 203 is facing the home, it can also create a drawing diary along with the appearance of the family. As an example, as a pre-determined selection criterion, a dinner scene can be identified through image recognition, the dinner scene can be selected, and inputs such as "drawing diary depicting a teddy bear happily watching the user's family eat dinner" into the image generation model to generate an image and create a drawing diary P.

[0221] Therefore, User 10 does not need to accompany Robot 100 all the time. In addition, it can objectively record User 10's daily life and the scenery around them.

[0222] Next, the specific handling of selecting the scene for creating the drawing diary P using one day's events will be explained when the drawing diary P is generated solely through the emotions of the robot 100 without the robot 100 being activated or placed. Figure 10 This is a flowchart illustrating an example of the process for selecting a scenario for creating the drawing diary P using events of one day, when the robot 100 is neither activated nor placed, and the drawing diary P is generated solely based on the emotions of the robot 100. Furthermore, Figure 10 Processing can begin, for example, at a predetermined time on a day when user 10 has no conversation with robot 100.

[0223] In step S300, the emotion determination section 232 determines the emotion of the robot 100. The emotion of the robot 100 can be determined randomly, for example, or can follow a personality setting given to the robot (e.g., gentle, angry, etc.).

[0224] In step S302, the action determination section 236 acquires the events of the day stored in the storage section 220 acquired by the microphones 201, 2D camera 203.

[0225] In step S304, the action determination section 236 selects a scene from the events of the day stored in the storage section 220. For example, the scene of rain is selected from the sound "It's raining!" recognized from the events of the day stored in the storage section 220.

[0226] In step S306, the action determination section 236 creates the picture diary P based on the emotion of the robot 100 and the selected scene. For example, the picture diary P is created by inputting the text "picture diary depicting a teddy bear playing sadly with an umbrella in the rain" or the like to the image generation model and the article generation model.

[0227] In step S308, the action determination section 236 outputs the created picture diary P. For example, the action determination section 236 transmits it to a predetermined person including the user 10 by mail or the like via the communication processing section 280.

[0228] In this way, even in a case where the robot 100 is placed, the picture diary P is created only by the emotion of the robot 100, so the user 10 does not need to always accompany the robot 100, and the daily life and the daily scenery of the user 10 can be objectively recorded.

[0229] Further, in a case where the robot 100 is not activated or is placed, the emotion of the robot 100 can also change according to the time not talking with the user 10. For example, it can also be that the more the time not talking with the user 10, the greater the negative emotion value. Also, in a case where the absolute value of the negative emotion value of the robot 100 becomes equal to or greater than a predetermined threshold value, the picture diary P can be created and output to the user 10.

[0230] In addition, the emotion value of the user gradually decreases from the moment the event occurs. Therefore, in a case where the user reports the event to the robot 100 at the end of the day, it can not be determined that the emotion value of the user is high. If it is not determined that the emotion value of the user is high, as explained in the other embodiment described above, in a case where the robot 100 creates the picture diary P, sometimes the scene for the picture diary P is not selected, and the picture diary P is not created.

[0231] Accordingly, the robot 100 can also extract a predetermined specific word such as a predetermined keyword from the utterance of the user when determining the emotion of the user, determine the emotion value of the user from at least one of the extracted specific word and the conversation content after the specific word, and determine the action of the robot 100.

[0232] For example, the wake-up word of the robot 100 is applied as the specific word. Also, the utterance understanding section 212 can function as an extraction section to extract the wake-up word from the utterance of the user, and the emotion determination section 232 can determine the emotion of the user using at least one of the extracted wake-up word and the conversation content after the wake-up word analyzed by the utterance understanding section 212. As the wake-up word, for example, the name of the robot or the like can be applied, or another word can be applied.

[0233] As a specific processing example, for example, instead of S102 of Figure 3 , the processing of Figure 11 may be performed to determine the emotion value of the user. Figure 11 is a flowchart indicating an example of the processing of determining the emotion value of the user.

[0234] That is, in step S400, the utterance understanding section 212 analyzes the voice of the user 10 detected by the microphone 201 based on the history data 222, and outputs the text information indicating the utterance content of the user 10.

[0235] In step S402, the emotion determination section 232 determines whether or not there is a predetermined specific word. This determination is, for example, a determination of whether or not there is a specific word such as a phrase of "Hey, listen to me," a pre-set name of the robot, or the like. In the case where there is a predetermined specific word, the processing proceeds to step S404, and in the case where there is no specific word, the processing proceeds to step S406.

[0236] In step S404, the emotion determination section 232 determines the emotion value of the user based on the specific word. That is, the specific word is extracted, and at least one of the extracted specific word and the conversation content after the specific word is used to determine the emotion value of the user. For example, at least one of a specific word such as a phrase of "Hey, listen to me," the name of the robot 100, and the conversation content thereafter is used to determine the emotion value of the user. In the case where the specific word is used to determine the emotion value of the user, the emotion value can also be determined based on the frequency of the utterance of the specific word, the speed of the utterance, or the like. In the case where the conversation content after the specific word is used to determine the emotion value of the user, the emotion value of the user can also be determined by the voice emotion recognition section 211 recognizing the emotion of the user in the conversation. Further, the emotion value can also be determined further considering the state of the user when the specific word is uttered.

[0237] On the other hand, in step S406, the same processing as S102 described above is performed. That is, the emotion determination section 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module section 210 and the state of the user 10 recognized by the user state recognition section 230.

[0238] Thus, the action determination section 236 determines the action of the robot 100 based on the emotion value determined by the emotion determination section 232.

[0239] Further, the specific word can also be applied to a phrase or the like uttered when the user is excited. The specific word such as a phrase uttered when the user is excited can also be learned in advance, for example, from the conversation of the robot 100 with the user. That is, the specific word is determined using a learning completion model learned based on the conversation content. For example, a phrase uttered by the user in an excited state is learned from the conversation content, and the specific word is extracted from the conversation content. Alternatively, the phrase can be set in advance, and it can be determined from the conversation content that the user is in an excited state.

[0240] In addition, in a case where the wake-up word is applied as the specific word, the user's action included in the conversation after the robot 100 is called by the wake-up word can also be selected based on the history data 222, and the action of the user can be set as the drawing diary. For example, when the user happily says "○○, listen to me, today I went hiking and caught a unicorn" to the robot 100, the robot 100 generates a drawing diary in which the user goes hiking and catches a unicorn. Further, "○○" is the name of the robot 100 set in advance. In addition, a diary can also be created instead of the drawing diary P.

[0241] Specifically, the drawing diary P can also be created by performing the processing illustrated in Figure 12 Figure 12 is a flowchart illustrating an example of the processing when the drawing diary is created. Further, Figure 12 The processing of the drawing diary P can be started at a time point such as the end of the day, in a case where the user's utterance is detected by the microphone 201 of the sensor section 200, or the like.

[0242] In step S500, the utterance understanding section 212 analyzes the utterance content of the user based on the history data 222. That is, the voice of the user 10 detected by the microphone 201 is analyzed, and text information indicating the utterance content of the user 10 is output.

[0243] In step S502, the emotion determination section 232 determines whether or not there is a predetermined wake-up word. In a case where there is a wake-up word, the processing proceeds to step S504, and in a case where there is no wake-up word, the processing ends.

[0244] ​In step S504, the action determination section 236 creates a picture diary P based on the conversation content of the user after the wake-up word. For example, the conversation content after the wake-up word is texted, a fixed sentence such as "a picture diary drawn with a wax crayon" is added to the text, and the picture P1 of the picture diary P is created by inputting to the image generation AI. In addition, the action determination section 236 generates the picture diary P by combining the texted text and the picture P1.

[0245] By setting the name of the robot 100 as the wake-up word, it is possible to create the picture diary P limited to content that is particularly desired to be heard.

[0246] In addition, even if the robot 100 does not have an event recorded in real time, it is possible to set a scene with a high emotional value as the picture diary P.

[0247] In the example of Figure 12 In the example of Figure 11 In the example of

[0248] In addition, when a predetermined scene is selected from events stored at a time point such as the end of the day, it is possible to create a picture diary if there is a predetermined specific word even if it is not possible to read the user's emotion. For example, it is possible to set a phrase spoken by the user when the user is excited as a predetermined keyword, and to select a scene in which the keyword is spoken to create the picture diary P.

[0249] In addition, when creating the picture diary P, it is possible to exclude content that is not desired as a picture diary. For example, in the case where the emotional value of the user determined by the emotion determination section 232 includes a scene of a predetermined emotion, it is possible to prohibit the creation of the picture diary of the selected scene. Or, in the case where there is content that violates public order and good customs or a predetermined keyword, it is possible to prohibit the creation of the picture diary P. Specifically, by performing the process shown in Figure 13 Figure 13 is a flowchart showing the flow of the process when creating the picture diary P excluding content that is not desired as the picture diary P.

[0250] That is, in step S600, the robot 100 determines whether or not to create a picture diary of the selected scene. In this determination, for example, the action determination section 236 determines whether or not the scene set as the picture diary P selected in the above (step 2) belongs to content that is not desired as the picture diary P. In the case where it does not belong to content that is not desired as the picture diary P, the process proceeds to step S602, and in the case where it belongs to content that is not desired as the picture diary P, the process proceeds to step S604.

[0251] ​In step S602, the robot 100 makes the picture diary P. That is, the action determination section 236 makes the picture diary P by making the picture P1 of the picture diary P using the image generation AI by performing the above-described (step 3).

[0252] On the other hand, in step S604, the action determination section 236 selects a scene in which the sum of the emotional value of the user 10 and the emotional value of the robot 100 is weak, and returns to step S600 to repeat the above-described processing.

[0253] Further, in Figure 13 In the above-described (step 2), the scene selected is not intended to be the content of the picture diary P, but an example in which another scene is selected to make the picture diary P, but it is also possible to make the picture diary P by excluding the content that is not intended to be the picture diary without changing the selected scene. For example, it is also possible to make the picture diary P by removing the content that violates public order and morals, the keyword.

[0254] The disclosures of Japanese Patent Application No. 2023-067894 filed on April 18, 2023, Japanese Patent Application No. 2023-128184, Japanese Patent Application No. 2023-132390, Japanese Patent Application No. 2023-133125, and Japanese Patent Application No. 2023-153912 are incorporated by reference herein in their entirety.

[0255] All of the documents, patent applications and technical standards cited in the present specification are incorporated by reference to the extent the same are treated as if each document, patent application, or technical standard were specifically and individually indicated to be incorporated by reference in its entirety.

Claims

1. An action control system, comprising: The user status identification unit identifies user status, including user actions. The emotion determination department determines the emotions of the user or the robot. The action determination unit determines the robot's actions based on an article generation model that enables dialogue between the user and the robot, and the actions of the user or the robot, according to the user's state and the user's or robot's emotions. as well as The storage control unit stores data containing the user's actions in historical data. The action determination unit further selects a scene that meets a predetermined selection criterion based on the historical data, inputs the text representing the selected scene into the image generation model, generates an image representing the scene, and outputs the image.

2. The action control system according to claim 1, wherein, The storage control unit stores the emotions determined by the emotion determination unit and the data containing the user's actions in the historical data. The action determination unit selects scenarios that meet the selection criteria related to the emotion based on the historical data.

3. The motion control system according to claim 2, wherein, The selection criteria related to the emotion include the case where the user's emotion and the robot's emotion are strongest, or the case where a specific emotion is strongest among multiple emotions, relating to the user's emotion or the robot's emotion.

4. The action control system according to claim 1, wherein, The selection criteria include the user having the most smiley faces.

5. The action control system according to claim 1, wherein, The action determination unit receives at least one of the title and article of the selected scene specified by the user, and outputs an image representing the scene and a combination of the received title and article.

6. The action control system according to claim 1, wherein, The action determination unit receives an image representing the user, inputs the text representing the user and the text representing the selected scene into the image generation model, and generates an image representing the scene.

7. The action control system according to claim 1, wherein, The robot is mounted on a cloth doll, or connected wirelessly or via a wired connection to a control device mounted on the cloth doll.

8. The action control system according to claim 7, wherein, The control device consists of a speaker installed in the mouth of the cloth doll, a camera installed in the eyes that constitute the face of the cloth doll, and a microphone installed in the ears.

9. The action control system according to claim 1, wherein, The action determination unit generates and outputs an image based on the robot's emotions without engaging in dialogue with the user.

10. The action control system according to claim 9, wherein, The robot's emotions are either randomly determined or pre-assigned to the robot, reflecting its personality. The action determination unit prepares an image of the robot in advance for each emotion of the robot, and outputs the image corresponding to the emotion of the robot.

11. The action control system according to claim 1, wherein, The motion control system also includes a storage unit that stores the sounds and images captured around the robot as events of the day. Without engaging in dialogue with the user, the action determination unit selects a scene from a day's events stored in the storage unit that meets a predetermined selection criterion, inputs the text representing the robot's emotions and the event corresponding to the selected scene into an image generation model, generates an image representing the scene, and outputs the image.

12. An action control system, comprising: The extraction unit extracts specific, pre-defined words from the user's speech; The sentiment determination unit determines the user's sentiment based on at least one of the specific words extracted by the extraction unit and the conversation content following the specific words; as well as The action determination unit determines the robot's actions based on the emotions determined by the emotion determination unit.

13. The action control system according to claim 12, wherein, The specific words were determined using a learning completion model based on user conversation content.

14. The action control system according to claim 12, wherein, The action determination unit selects a scene that meets a predetermined selection criterion based on the emotion determined by the emotion determination unit and historical data containing the user's actions. It inputs the text representing the selected scene into an image generation model to generate an image representing the scene. The generated image is then combined with an article representing the conversation content following the specific word and output.

15. The action control system according to claim 14, wherein, The emotion determination unit determines an emotion value representing the user's emotion. The action determination unit selects from the historical data scenarios where the user's emotional value, as determined by the emotion determination unit, is above a predetermined threshold.

16. The action control system according to claim 14, wherein, The action determination unit excludes pre-determined content and combines the article with an image representing the scene, then outputs it.

17. The action control system according to claim 14, wherein, The action determination unit selects the scenario in which the extraction unit extracts the specific word.

18. An action control system, comprising: The user status identification unit identifies user status, including user actions. The emotion determination department determines the emotions of the user or the robot. as well as The action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The action determination unit selects a scene that meets a predetermined selection criterion based on historical data containing the user's actions, inputs text representing the selected scene into an image generation model, generates an image representing the scene, and extracts user information containing the user's preferences based on the text, and determines the content to be suggested to the user based on the extracted user information.

19. The action control system according to claim 18, wherein, The action determination unit collects and extracts association information related to the user's information, and includes the collected association information to determine the action proposed to the user.

20. The action control system according to claim 18, wherein, The action control system further includes an action control unit, which controls the speech from a speaker installed on the robot based on the action determined by the action determination unit, thereby proposing an action to the user.

21. The motion control system according to claim 20, wherein, The action determination unit stores the extracted user information in the historical data, and the action control unit proposes an action to the user at a time point associated with the user information stored in the historical data.

22. The motion control system according to claim 18, wherein, The action determination unit stores the extracted user information in the historical data, and uses the user information stored in the historical data to match with other users.

23. An action control system, comprising: The user status identification unit identifies user status, including user actions. The emotion determination department determines the emotions of the user or the robot. as well as The action determination unit, based on an article generation model with dialogue functionality enabling the user to converse with the robot, determines the generated image as the robot's action corresponding to the user's state and the user's or robot's emotions. The action determination unit selects a scenario that meets a predetermined selection criterion based on historical data including the user's past emotions and actions. It inputs text representing the selected scenario into an image generation model, generates an image representing the scenario, and outputs the image. Based on the text, it analyzes the user's lifestyle and outputs the analysis results.

24. The action control system according to claim 23, wherein, The action determination unit, as an analysis result of the lifestyle, also scores and outputs a pre-set ideal lifestyle.

25. The motion control system according to claim 23, wherein, The action control system further includes an action control unit that performs control to enable the robot to pronounce the lifestyle analysis results analyzed by the action determination unit.

26. The motion control system according to claim 23, wherein, The action determination unit outputs the analysis results to the mobile terminals of pre-registered users.

27. The motion control system according to claim 23, wherein, The action determination unit further determines and outputs the content of the user proposal based on the analysis results.

Images