Action control system

Abstract

The action control system includes: an emotion determination unit that determines an emotion of a user or an emotion of a robot; and an action determination unit that, on the basis of a dialogue function that causes a user to dialogue with the robot, generates action content of the robot for the action of the user and the emotion of the user or the emotion of the robot, determines the action of the robot corresponding to the action content, and determines the action of the robot according to the action content. The action specifying unit specifies a preset action of the robot for pacifying the emotion of the user when a threshold value related to the emotion preset for the user is exceeded.

Description

Technical Field

[0001] This invention relates to an action control system. Background Technology

[0002] Patent Document 1 discloses a technique for determining appropriate actions for a robot based on a user's state. In the prior art of Patent Document 1, user reactions when the robot performs a specific action are identified, and when the robot's action in response to the identified user reaction cannot be determined, the robot's actions are updated by receiving information from a server related to actions suitable for the identified user state.

[0003] Patent document 2 describes an emotion determination system for determining the emotions of a robot.

[0004] Patent document 3 describes an action control system that determines appropriate robot actions based on the state of the robot's dialogue partner, i.e., the user.

[0005] Patent document 4 describes a method for determining appropriate robot actions based on the state of the robot's dialogue partner, i.e., the user. Patent document 5 describes an emotion determination system for determining the robot's emotions.

[0006] Existing technical documents Patent documents Patent Document 1: Patent No. 6053847 Patent Document 2: Japanese Patent Application Publication No. 2017-199319 Patent Document 3: Japanese Patent Application Publication No. 2016-012341 Patent Document 4: Japanese Patent Application Publication No. 2016-012341 Patent Document 5: Japanese Patent Application Publication No. 2017-199319 Summary of the Invention The problem that the invention aims to solve However, in the existing technology, there is still room for improvement in how robots perform appropriate actions in response to user actions.

[0007] Methods for solving problems According to a first aspect of the present invention, an action control system is provided. The action control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and an action determination unit that, based on a dialogue function enabling the user to converse with the robot, generates action content for the robot corresponding to the user's action and the user's or robot's emotion, determines the robot's action corresponding to the action content, and, if the action determination unit exceeds a pre-set emotion-related threshold for the user, determines a pre-set action of the robot for soothing the user's emotion.

[0008] According to a second aspect of the present invention, an action control system is provided. The action control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and an action determination unit that, based on a dialogue function enabling the user to converse with the robot, generates action content for the robot in response to the user's actions and the user's or robot's emotions, determines the robot's actions corresponding to the action content, and the action determination unit repeatedly executes dialogues with the user, performs a personality analysis of the user based on the results of the multiple executions of the dialogues, and reports the results of the personality analysis to the user.

[0009] According to a third aspect of the present invention, an action control system is applied. The action control system includes: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and an action determination unit that, based on a dialogue function enabling the user to converse with the robot, generates action content for the robot based on the user's actions and the user's or robot's emotions, determines the robot's actions corresponding to the action content, and sets a "guest dialogue mode" as the robot's dialogue mode. In this "guest dialogue mode," the robot is positioned as a dialogue partner in situations where it does not need to speak to a specific person but wants someone to listen to it. In this "guest dialogue mode," when conversing with the user, predetermined keywords related to the specific person are excluded, and the spoken content is output.

[0010] According to a fourth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the robot's emotion; and an action determination unit that determines, based on an article generation model having a dialogue function that enables the user to converse with the robot, the robot's action corresponding to the user state and the user's emotion or the robot's emotion, wherein the emotion determination unit provides feedback that increases an emotion value representing the intensity of the emotion when the user has a positive emotion accompanying the robot's action, and provides feedback that decreases an emotion value representing the intensity of the emotion when the user has a negative emotion accompanying the robot's action.

[0011] According to a fifth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the robot's emotion; and an action determination unit that determines, based on an article generation model having a dialogue function that enables the user to converse with the robot, the robot's action corresponding to the user state and the user's emotion or the robot's emotion, wherein the action determination unit is configured to execute the dialogue based on the user's emotional history and the context of the dialogue between the user and the robot.

[0012] According to a sixth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the emotion of an electronic device; and an action determination unit that determines the action of the electronic device corresponding to the user state and the user's emotion or the emotion of the electronic device based on an article generation model having a dialogue function that enables the user to converse with the electronic device, wherein the emotion determination unit determines the user's emotion based on a learning completion model, the learning completion model being learned based on at least one of the following: multiple learning data as a combination of the user's voiceprint and emotions perceived by multiple other users from the voiceprint, and multiple learning data as a combination of the user's gestures and emotions perceived by multiple other users from the gestures.

[0013] In the seventh aspect of the present invention, the electronic device is mounted on a plush toy, or is connected wirelessly or wiredly to a control device mounted on the plush toy.

[0014] In the eighth aspect of the present invention, the controlled object device is a speaker, and a microphone or camera is mounted on the plush toy.

[0015] In a ninth aspect of the invention, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0016] In a tenth aspect of the present invention, a wireless power receiving unit is disposed inside the plush toy, the wireless power receiving unit receiving wireless power from an external wireless power transmission unit, and the control object device or the electronic device receiving power through the wireless power receiving unit.

[0017] In the eleventh aspect of the present invention, the electronic device is a robot. The robot includes a device that performs physical actions, a device that does not perform physical actions but outputs images or sounds, and an intelligent agent that performs actions in software.

[0018] According to a twelfth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the emotion of an electronic device; and an action determination unit that, based on an article generation model having a dialogue function that enables the user to converse with the electronic device, determines the action of the electronic device corresponding to the user state and the user's emotion or the emotion of the electronic device. When the content of a question raised by the user is unclear, the action determination unit automatically converts the unclear question into a correct question based on a needs analysis found from the words used and the user's facial expressions, or presents a solution after re-listening to the user's question and obtaining the actual question.

[0019] The second aspect of the present invention relates to an action control system that, based on the first aspect, uses a robot as the electronic device. Here, the robot includes a device that performs physical actions, a device that does not perform physical actions but outputs images or sounds, and an intelligent agent that performs actions in software.

[0020] According to a thirteenth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the robot's emotion; and an action determination unit that determines the robot's action corresponding to the user state and the user's emotion or the robot's emotion based on an article generation model having a dialogue function that enables the user to converse with the robot. The action determination unit determines the robot's action for content appreciation, thereby enabling the sharing of values ​​between the user and the robot.

[0021] Robots include devices that perform physical actions, devices that do not perform physical actions but output images or sounds, and intelligent agents that perform actions in software.

[0022] According to a fourteenth aspect of the present invention, an action control system is provided. The action control system includes a plurality of electronic devices, each of the plurality of electronic devices comprising: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the emotion of the electronic device; and an action determination unit that determines, based on a text generation model having a dialogue function that enables the user to interact with the electronic device, the action determination unit determines the action of the electronic device corresponding to the user state and the user's emotion or the emotion of the electronic device, and the user includes other electronic devices constituting the plurality of electronic devices.

[0023] The action control system according to the fifteenth aspect of the present invention is based on the first aspect, wherein the electronic device is a robot. The robot includes a device that performs physical actions, a device that does not perform physical actions but outputs images or sounds, and an intelligent agent that performs actions in software.

[0024] According to a sixteenth aspect of the present invention, an action control system is provided. The action control system includes: a user state recognition unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the robot's emotion; and an action determination unit that, based on an article generation model having a dialogue function that enables the user to converse with the robot, determines the robot's actions corresponding to the user state and the user's emotion or the robot's emotion, and the action determination unit generates a chart obtained by statistically processing the user's emotion over a predetermined period.

[0025] According to a seventeenth aspect of the present invention, an action control system is provided. The action control system includes: a user state identification unit that identifies a user state containing user actions; an emotion determination unit that determines the user's emotion or the emotion of an electronic device; and an action determination unit that, based on an article generation model having a dialogue function that enables the user to converse with the electronic device, determines the actions of the electronic device corresponding to the user state and the user's emotion or the electronic device's emotion. The emotion determination unit uses an emotion map mapped with multiple emotions to estimate the user's emotion and the electronic device's emotion, and adds fixed statements to text representing the user's emotion and the electronic device's emotion to ask questions about the electronic device's actions corresponding to the user's actions, and inputs these statements into the article generation model, thereby estimating the emotion of the generated article based on its content. The action determination unit combines the user's emotion, the electronic device's emotion, and the article's emotion to generate a conversation with the user, which is determined to be the electronic device's action content.

[0026] In the eighteenth aspect of the present invention, the electronic device is mounted on a plush toy, or is connected wirelessly or wiredly to a control device mounted on the plush toy.

[0027] In the nineteenth aspect of the present invention, the control device is a loudspeaker, and a microphone or camera is mounted on the plush toy.

[0028] In a twentieth aspect of the invention, the camera is mounted on the eyes constituting the face of the plush toy, the microphone is mounted on the ears, and the speaker is mounted on the mouth.

[0029] In the twenty-first aspect of the present invention, a wireless power receiving unit is disposed inside the plush toy, the wireless power receiving unit receiving wireless power from an external wireless power transmission unit, and the control object device or the electronic device receiving power via the wireless power receiving unit.

[0030] In the twenty-second aspect of the present invention, the electronic device is a robot. The robot includes a device that performs physical actions, a device that does not perform physical actions but outputs images or sounds, and an intelligent agent that performs actions in software.

[0031] According to a twenty-third aspect of the present invention, an action control system is provided. The action control system includes: a user state identification unit that identifies a user state including user actions; an emotion determination unit that determines the user's emotion or the robot's emotion; and an action determination unit that, based on an article generation model having a dialogue function enabling the user to converse with the robot, determines the robot's actions corresponding to the user state and the user's or robot's emotion, and the action determination unit generates a record of events for a predetermined period based on history data including the user's action history.

[0032] According to a twenty-fourth aspect of the present invention, an action control system is provided. The action control system includes: a state recognition unit that recognizes a user state including user actions and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; an action determination unit that, at a predetermined time, uses at least one of the user state, the state of the electronic device, the user's emotion, and the electronic device's emotion, and an action determination model, to determine any one of a plurality of types of device actions, including inactivity, as an action of the electronic device; and a storage control unit that stores event data in history data, the event data including an emotion value determined by the emotion determination unit and data including the user's actions, wherein the device actions include analyzing the user's personality, and the action determination unit, when determining that the analyzed user's personality is an action of the electronic device, uses the history data including conversation history with the user to analyze the user's personality and presents the analyzed personality.

[0033] According to a twenty-fifth aspect of the present invention, a motion control system as described in the twenty-fourth aspect is provided. The electronic device of this motion control system is a robot, and the motion determination unit determines any one of a plurality of robot actions, including inaction, as an action of the robot.

[0034] According to a twenty-sixth aspect of the present invention, an action control system relating to a twenty-fifth aspect is provided. The action determination model of this action control system is a text generation model with dialogue functionality. The action determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking the robot about its action, into the text generation model. Based on the output of the text generation model, the action of the robot is determined.

[0035] According to a twenty-seventh aspect of the present invention, a motion control system as described in the twenty-fifth or twenty-sixth aspect is provided. The robot of this motion control system is mounted on a plush toy, or connected wirelessly or via a wired connection to a controllable device mounted on the plush toy.

[0036] According to a twenty-eighth aspect of the present invention, a motion control system as described in a twenty-fifth or twenty-sixth aspect is provided. The robot in this motion control system is an intelligent agent for engaging in dialogue with the user.

[0037] Robots include devices that perform physical actions, devices that do not perform physical actions but output images or sounds, and intelligent agents that perform actions in software.

[0038] According to a twenty-ninth aspect of the present invention, an action control system is provided. The action control system includes: a state recognition unit that recognizes a user state including user actions and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; an action determination unit that, at a predetermined time, uses at least one of the user state, the state of the electronic device, the user's emotion, and the electronic device's emotion, and an action determination model, to determine any one of a plurality of types of device actions, including inactivity, as an action of the electronic device; and a storage control unit that stores event data in history data, the event data including an emotion value determined by the emotion determination unit and data including the user's actions, wherein the user includes other electronic devices constituting a plurality of the electronic devices, the device actions include engaging in conversation with the other electronic devices, and the action determination unit, when determining that engaging in conversation with the other electronic devices is an action of the electronic device, determines the conversation to be spoken based on the event data stored in the history data using a document generation model.

[0039] In a thirtieth aspect of the invention, the electronic device is a robot, and the action determination unit determines any one of a plurality of robot actions, including inaction, as an action of the robot.

[0040] In the thirty-first aspect of the present invention, the action determination model is an article generation model with dialogue function. The action determination unit inputs text representing at least one of the user's state, the robot's state, the user's emotion, and the robot's emotion, as well as text asking the robot about its action, into the article generation model. Based on the output of the article generation model, the action of the robot is determined.

[0041] In the thirty-second aspect of the present invention, the robot is mounted on a plush toy or connected wirelessly or via a wired connection to a control device mounted on the plush toy.

[0042] In the thirty-third aspect of the present invention, the robot is an intelligent agent for conversing with the user.

[0043] Robots include devices that perform physical actions, devices that do not perform physical actions but output images or sounds, and intelligent agents that perform actions in software.

[0044] According to a thirty-fourth aspect of the present invention, an information processing system is provided. The information processing system may include a basic emotional information storage unit that stores basic emotional information obtained by establishing a correspondence between the states of elements of a non-human object and emotional values ​​representing the emotions of the object. The information processing system may include an information updating unit that, when observing a non-human object and finding that any one of the states of a plurality of elements of the identified object matches the state of any element of a plurality of basic emotional information stored in the basic emotional information storage unit, updates the basic emotional information of the matching element state using at least one of the inconsistent element states among the plurality of element states.

[0045] In the information processing system, when the object is a pet, the elements of the object may include at least one of vocalizations, eye movements, body movements, and postures. When the object is a dog, the basic emotional information storage unit may store basic emotional information obtained by establishing a correspondence between the tail-wagging state and the emotional value indicating joy. The information updating unit may update the basic emotional information obtained by establishing a correspondence between the tail-wagging and the emotional value indicating joy based on at least one of the vocalizations, eye movements, and postures included in the observation results of observing a dog, if the observation results include tail wagging.

[0046] In any of the above information processing systems, when the object is a flower, the basic emotional information storage unit may store basic emotional information obtained by establishing a correspondence between the flower opening and the emotional value representing the emotion of joy. When the observation result obtained from observing the flower includes the flower opening, the information update unit updates the basic emotional information obtained by establishing a correspondence between the flower opening and the emotional value representing the emotion of joy by using at least one of the elements included in the observation result other than the flower opening.

[0047] In any of the aforementioned information processing systems, an emotion determination unit is further included. This unit determines the object's emotion value corresponding to the state of any one of the multiple elements of the object identified through observation as an object, when such state matches the state of any element of a multiple basic emotion information stored in the basic emotion information storage unit. The information processing system also includes a human emotion transfer pattern storage unit, which stores human emotion transfer patterns representing the transfer of emotions when humans communicate with others. The emotion determination unit can use these human emotion transfer patterns to determine the emotion value representing the object's emotion when a human takes action towards the object. The human emotion transfer pattern storage unit can store human emotion transfer patterns corresponding to multiple humans. The emotion determination unit can use the human emotion transfer pattern of the human associated with the object from among the multiple human emotion transfer patterns stored in the human emotion transfer pattern storage unit to determine the emotion value representing the object's emotion.

[0048] Any of the above information processing systems also includes an electronic device that uses the emotion determined by the emotion determination unit to communicate with the owner of the object.

[0049] According to the thirty-fifth aspect of the present invention, a program is provided for enabling a computer to function as the information processing system.

[0050] According to a thirty-sixth aspect of the present invention, a control system is provided. The control system may include an emotion determination unit that determines a user emotion value representing the emotion of a user interacting with an electronic device. The control system may include a determination unit that, based on the shift in the user's user emotion value when the electronic device explains an object to the user, determines at least one of the user's understanding of the object and their level of attention. The control system may include a control unit that, based on the determination result of the determination unit, adjusts the output mode of the electronic device's explanation of the object.

[0051] In the control system, if the user's emotion is negative during the initial stage of the electronic device's explanation of the object, the determination unit determines that the lower the degree of the user's negative emotion during the initial stage of the explanation, the higher the user's level of understanding and attention.

[0052] In any of the above control systems, the control unit may, during the description of the object, cause the electronic device to question the user, and the determination unit may, based on the user's response to the question, determine at least one of the user's understanding and attention to the object.

[0053] In any of the aforementioned control systems, the control unit can adjust at least one of the following based on the determination result of the determination unit: the tone of the electronic device's voice, the speaking speed of the electronic device, the amount of gestures of the electronic device, the amount of nodding by the electronic device in coordination with the user, and the amount of pauses by the electronic device. The control system may include a storage unit storing historical data, which is obtained by the control unit based on the transfer of the user's emotional value at the time of the adjustment, establishing a correspondence between the user's level of understanding or attention determined by the determination unit and the content of the adjustment. The control unit can determine the content of the adjustment based on multiple historical data stored in the storage unit.

[0054] In any of the above control systems, the control unit may perform at least one of the following adjustments: adjusting the pitch of the electronic device's voice to be stronger as the user's comprehension level decreases; adjusting the speaking speed of the electronic device to be slower as the user's comprehension level decreases; adjusting the amount of gestures of the electronic device to be more frequent as the user's comprehension level decreases; adjusting the number of times the electronic device nods in coordination with the user to be more frequent as the user's comprehension level decreases; and adjusting the number of times the electronic device pauses to be more frequent as the user's comprehension level decreases.

[0055] According to a thirty-seventh aspect of the present invention, an information processing system is provided. The information processing system may include a control unit that controls an electronic device to engage in conversation with a user. The information processing system may include an emotion determination unit that, when the electronic device engages in conversation with multiple users regarding a topic, determines user emotion values ​​representing the emotions of each of the multiple users. The information processing system may include a correspondence data generation unit that generates corresponding data by establishing a correspondence between the user emotion values ​​of the multiple users and the conversation content of the conversation. The information processing system includes a conversation content generation unit that generates conversation content corresponding to the topic based on the multiple correspondence data generated by the correspondence data generation unit.

[0056] In the information processing system, the conversation content generation unit can generate conversation content suitable for multiple users in a scenario involving a single topic and multiple user conversations. The conversation content generation unit can also generate conversation content where the sentiment values ​​of all multiple users in a scenario involving a single topic and multiple user conversations will not become negative.

[0057] Any of the above information processing systems may further include a conversation content adjustment unit, which automatically adjusts the conversation content corresponding to the topic. The control unit can control the electronic device so that when the electronic device has a conversation with multiple users regarding the topic, it has a conversation with the multiple users based on the conversation content adjusted by the conversation content adjustment unit corresponding to the topic. The corresponding data generation unit generates corresponding data obtained by establishing a correspondence between the user sentiment values ​​of the multiple users and the conversation content when the electronic device has a conversation with the multiple users based on the adjusted conversation content.

[0058] In any of the above information processing systems, the session content generation unit can prioritize the newest corresponding data among the multiple corresponding data to generate session content corresponding to the topic.

[0059] Any of the above information processing systems may further include a user status recognition unit, which identifies the actions of the multiple users in the one topic session of the electronic device, and the corresponding data generation unit can generate corresponding data by establishing a correspondence between the user sentiment values ​​and user action information of the multiple users and the session content of the session.

[0060] In any of the above information processing systems, the emotion determination unit can determine an electronic device emotion value representing the emotion of the electronic device during a conversation between the electronic device and the multiple users, and the conversation content generation unit can adjust the content of the conversation based on the user emotion values ​​of the multiple users and the electronic device emotion value during a conversation between the electronic device and the multiple users.

[0061] According to the thirty-eighth aspect of the present invention, a program is provided for enabling a computer to function as the information processing system.

[0062] It should be noted that the above summary of the invention does not list all the features required by the invention. Furthermore, sub-combinations of these feature groups also constitute the invention. Attached Figure Description

[0063] Figure 1 An example of system 5 involved in this embodiment is shown in summary.

[0064] Figure 2The functional structure of robot 100 is shown in summary.

[0065] Figure 3 An example of the motion flow of robot 100 is shown in summary.

[0066] Figure 4 This is a summary illustration of an example of the hardware structure of the computer 1200.

[0067] Figure 5 The emotion map 400 is shown, which maps multiple emotions.

[0068] Figure 6 The emotion map 900 is shown, which maps multiple emotions.

[0069] Figure 7 (A) is an appearance drawing of a plush toy according to other embodiments. Figure 7 (B) is a diagram of the internal structure of a plush toy.

[0070] Figure 8 This is a front view of the back of a plush toy as described in other embodiments.

[0071] Figure 9A The functional structure of the robot 100 according to the second embodiment is shown in summary.

[0072] Figure 9B An example of the collection and processing operation flow of the robot 100 according to the second embodiment is shown in summary.

[0073] Figure 9C An example of the autonomous processing flow of the robot 100 according to the second embodiment is shown in summary.

[0074] Figure 9D The functional structure of the plush toy 100N according to the third embodiment is shown in summary.

[0075] Figure 9E The functional structure of the intelligent agent system 2500 according to the fourth embodiment is shown in summary.

[0076] Figure 9F This illustrates an example of the actions of an intelligent agent system.

[0077] Figure 9G This illustrates an example of the actions of an intelligent agent system.

[0078] Figure 10A The functional structure of the smart glasses 2700 according to the eleventh embodiment is shown in summary.

[0079] Figure 10B This illustrates one example of how smart glasses utilize intelligent agent systems.

[0080] Figure 11A An example of the overall structure of the system 3010 according to the thirteenth embodiment is shown in summary.

[0081] Figure 11B The functional block structure of robot 3200 and server 3100 is shown in summary.

[0082] Figure 12A An example of the overall structure of the system 4010 according to the fourteenth embodiment is shown in summary.

[0083] Figure 12B The functional block structure of robot 4200 and server 4100 is shown in summary.

[0084] Figure 13A An example of the overall structure of the system 5010 according to the fifteenth embodiment is shown in summary.

[0085] Figure 13B The functional block structure of robot 5200 and server 5100 is shown in summary. Detailed Implementation

[0086] The present invention will now be described through embodiments thereof, but these embodiments do not limit the invention as defined in the patent claims. Furthermore, not all combinations of features described in the embodiments are necessary for the solutions provided in the invention.

[0087] [First Implementation Method] Figure 1 An example of system 5 according to this embodiment is shown in summary. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. It should be noted that in the description of this embodiment, users 10a, 10b, 10c, and 10d are sometimes collectively referred to as user 10. In addition, users 11a, 11b, and 11c are sometimes collectively referred to as user 11. In addition, users 12a and 12b are sometimes collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 will be described mainly with respect to the functions of robot 100.

[0088] Robot 100 engages in conversations with user 10 or provides images to user 10. During this process, robot 100 collaborates with a communicable server 300 via communication network 20 to conduct conversations with user 10 and provide images. For example, robot 100 not only learns appropriate conversational techniques on its own but also collaborates with server 300 to learn how to better advance the conversation with user 10. Furthermore, robot 100 records image data of user 10 captured on the server 300, requests image data from server 300 as needed, and provides it to user 10.

[0089] Furthermore, robot 100 possesses emotion values ​​representing the types of emotions it experiences. For example, robot 100 has emotion values ​​indicating the intensity of each emotion: joy, anger, sorrow, happiness, happiness, unhappiness, peace of mind, unease, sadness, excitement, worry, stability, fulfillment, emptiness, and "normality." For instance, if robot 100 is in a state of high excitement when conversing with user 10, it will speak at a faster pace. In this way, robot 100 can express its emotions through actions.

[0090] Additionally, robot 100 can be configured to determine the actions of robot 100 corresponding to the emotions of user 10 by matching an article generation model using artificial intelligence (AI) with an emotion engine. Specifically, robot 100 is configured to recognize the actions of user 10, determine the emotions of user 10 towards those actions, and determine the actions of robot 100 corresponding to the determined emotions.

[0091] More specifically, upon recognizing the actions of user 10, robot 100 uses a pre-defined article generation model to automatically generate the appropriate actions for user 100. The article generation model can be interpreted as an algorithm and computation for text-based automatic dialogue processing. Examples of article generation models include Japanese Patent Application Publication No. 2018-081444 and chatGPT (Web Search).<URL: https: / / openai.com / blog / chatgpt> As disclosed, it is well-known, therefore its detailed explanation is omitted. This article generation model consists of a Large Language Model (LLM).

[0092] As described above, this embodiment enables the robot 100's actions to reflect the user 10's or robot 100's emotions and various linguistic information by combining a large language model with an emotion engine. In other words, according to this embodiment, a synergistic effect can be achieved by combining an article generation model with an emotion engine.

[0093] In addition, robot 100 has the function of recognizing the actions of user 10. Robot 100 analyzes the facial image of user 10 acquired through the camera function and the voice of user 10 acquired through the microphone function, thereby recognizing the actions of user 10. Based on the recognized actions of user 10, robot 100 determines the action to be performed.

[0094] Robot 100 stores rules for performing actions based on user 10's emotions, robot 100's emotions, and user 10's actions, and performs various actions according to these rules.

[0095] Specifically, the robot 100 has reaction rules for determining the robot's actions based on the user 10's emotions, the robot's own emotions, and the user 10's actions. For example, if the user 10's action is "laugh," then "laughing" is determined to be an action of the robot 100. Similarly, if the user 10's action is "anger," then "apologizing" is determined to be an action of the robot 100. Furthermore, if the user 10's action is "asking a question," then "answering" is determined to be an action of the robot 100. Finally, if the user 10's action is "feeling sad," then "starting a conversation" is determined to be an action of the robot 100.

[0096] If robot 100 identifies user 10's action as "anger" based on reaction rules, it will select "apology" as the action to be performed by robot 100. For example, if robot 100 selects "apology" as the action, it will perform the "apology" action while outputting the sound of words indicating "apology".

[0097] In addition, when the robot 100's emotion is "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "happiness" = 0) and the user 10's state is "alone, looking somewhat lonely", the robot 100's emotion has the change content of "becoming worried" and is determined to be able to perform the action of "starting a conversation".

[0098] If, based on reaction rules, robot 100 identifies its current emotion as "normal" and user 10 appears somewhat lonely, it will increase the "sadness" emotion value of robot 100. Furthermore, robot 100 will select actions such as "starting a conversation" as the action to be performed on user 10, as determined by the reaction rules. For example, if robot 100 selects the action of "starting a conversation," it will translate expressions of concern like "What's wrong?" into a worried tone and output it.

[0099] Additionally, robot 100 sends user response information to server 300, indicating that it received a positive response from user 10 through this action. This user response information may include, for example, user actions such as "getting angry," robot 100 actions such as "apologizing," the positiveness of user 10's response, and user 10's attributes.

[0100] Server 300 stores user response information received from robot 100. It should be noted that server 300 not only receives and stores user response information from robot 100, but also from robots 101 and 102 respectively. Furthermore, server 300 parses the user response information from robots 100, 101, and 102 and updates the response rules accordingly.

[0101] Robot 100 receives the updated response rules from server 300 by querying server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate response rules obtained from robots 101, 102, etc., into its own response rules.

[0102] Figure 2 The functional structure of the robot 100 is shown in summary. The robot 100 includes a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, an action recognition unit 234, an action determination unit 236, a storage control unit 238, an action control unit 250, a controlled object 252, and a communication processing unit 280.

[0103] The controlled object 252 includes a display device, speakers, LEDs for the eyes, and motors for driving the arms, hands, and legs. The robot 100's posture and behavior are controlled by controlling the motors for the arms, hands, and legs. Part of the robot 100's emotions can be expressed by controlling these motors. Additionally, the robot 100's facial expressions can be expressed by controlling the illumination state of the LEDs for its eyes. It should be noted that the robot 100's posture, behavior, and facial expressions are examples of the robot 100's attitude.

[0104] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, and a distance sensor 204. The microphone 201 continuously detects sound and outputs sound data. It should be noted that the microphone 201 can be mounted on the head of the robot 100 and has the function of dual-channel recording. The 3D depth sensor 202 continuously illuminates an infrared pattern and analyzes the infrared pattern based on the infrared images continuously captured by the infrared camera, thereby detecting the outline of an object. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light, generating visible light image information. The distance sensor 204 illuminates objects using, for example, laser light or ultrasonic waves, to detect the distance to an object. It should be noted that the sensor unit 200 may also include a clock, a gyroscope sensor, a touch sensor, a sensor for motor feedback, etc.

[0105] It should be noted that, in Figure 2 The components of the robot 100 shown, excluding the controlled object 252 and the sensor unit 200, are examples of the components of the motion control system of the robot 100. The motion control system of the robot 100 designates the controlled object 252 as the controlled object.

[0106] Storage unit 220 includes reaction rules 221 and history data 222. History data 222 includes the past emotional values ​​and action history of user 10. These emotional values ​​and action history are recorded for user 10, for example, by establishing a correspondence with user 10's identification information. At least one part of storage unit 220 is implemented using a storage medium such as a memory. It may also include a person database (DB) storing user 10's facial image, user 10's attribute information, etc. It should be noted that... Figure 2 Of the components of the robot 100 shown, the functions of the components other than the control object 252, the sensor unit 200, and the storage unit 220 can be implemented by the CPU based on a program. For example, the functions of these components can be implemented as CPU actions through basic software (OS) and a program that operates on the OS.

[0107] The sensor module 210 includes a voice emotion recognition unit 211, a speech comprehension unit 212, an expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input into the sensor module 210. The sensor module 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

[0108] The voice emotion recognition unit 211 of the sensor module 210 analyzes the voice of the user 10 detected by the microphone 201 and identifies the user 10's emotions. For example, the voice emotion recognition unit 211 extracts features such as the frequency components of the voice and identifies the user 10's emotions based on the extracted features. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the user 10's speech.

[0109] The expression recognition unit 213 recognizes the user 10's facial expressions and emotions based on images of the user 10 captured by the 2D camera 203. For example, the expression recognition unit 213 recognizes the user 10's facial expressions and emotions based on the shape and position of the eyes and mouth.

[0110] The face recognition unit 214 recognizes the face of user 10. The face recognition unit 214 recognizes user 10 by matching the facial image stored in the person DB (not shown) with the facial image of user 10 captured by the 2D camera 203.

[0111] The user state recognition unit 230 identifies the state of the user 10 based on the information parsed by the sensor module unit 210. For example, using the parsing results from the sensor module unit 210, it mainly performs perception-related processing. For example, it generates perception information such as "Dad is alone." and "There is a 90% probability that Dad doesn't smile." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks somewhat lonely."

[0112] The emotion determination unit 232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. For example, the information parsed by the sensor module unit 210 and the identified state of the user 10 are input into a pre-learned neural network to obtain an emotion value representing the emotion of the user 10.

[0113] Here, the emotion value representing user 10's emotion refers to the positive or negative value of the user's emotion. For example, if the user's emotion is a bright emotion accompanied by pleasant and calm feelings, such as "joy," "happiness," "happiness," "peace of mind," "excitement," "stability," and "fulfillment," it is represented by a positive value; the brighter the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "unhappiness," "unease," "grief," "worry," and "emptiness," it is represented by a negative value; the more unpleasant the emotion, the larger the absolute value of the negative value. When the user's emotion is not any of the above ("normal"), it is represented by a value of 0.

[0114] In addition, the emotion determination unit 232 determines the emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0115] The emotion value of Robot 100 includes emotion values ​​for each of the multiple emotion categories, such as values ​​(0~5) representing the intensity of "joy", "anger", "sorrow" and "happiness".

[0116] Specifically, the emotion determination unit 232 determines the emotion value representing the emotion of the robot 100 based on the rules for updating the emotion value of the robot 100, which are determined by establishing a correspondence between the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0117] For example, if the emotion determination unit 232 detects that user 10 looks somewhat lonely through the user state recognition unit 230, it will increase the "sadness" emotion value of robot 100. Conversely, if the user state recognition unit 230 detects that user 10 is smiling, it will increase the "joy" emotion value of robot 100.

[0118] It should be noted that the emotion determination unit 232 may also consider the state of the robot 100 to determine the emotion value representing the robot 100's emotion. For example, when the robot 100's battery has low remaining power or when the robot 100's surrounding environment is dark, the "sadness" emotion value of the robot 100 may be increased. When the user 10 continues to speak despite low battery power, the "anger" emotion value may be increased.

[0119] The action recognition unit 234 recognizes the actions of user 10 based on the information parsed by the sensor module unit 210 and the state of user 10 recognized by the user state recognition unit 230. For example, the information parsed by the sensor module unit 210 and the state of user 10 recognized are input into a pre-learned neural network to obtain the probabilities of multiple predetermined action categories (e.g., "laughing", "angry", "asking a question", "feeling sad"), and the action category with the highest probability is recognized as the action of user 10.

[0120] As described above, in this embodiment, the robot 100 obtains the speech content of the user 10 after identifying the user 10. However, when obtaining and using the speech content, in addition to obtaining the consent required by law from the user 10, the action control system of the robot 100 involved in this embodiment also considers the protection of the user 10's personal information and privacy.

[0121] The action determination unit 236 determines the action corresponding to the action of the user 10 identified by the action recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, the history data 222 of past emotion values ​​determined by the emotion determination unit 232 before determining the current emotion value of the user 10, and the emotion value of the robot 100. In this embodiment, the action determination unit 236 describes the case where the latest emotion value contained in the history data 222 is used as the past emotion value of the user 10, but the disclosed technology is not limited to this method. For example, the action determination unit 236 may also use multiple recent emotion values ​​as the past emotion values ​​of the user 10, or it may use emotion values ​​from a period of time such as one day ago as the past emotion values ​​of the user 10. In addition, the action determination unit 236 may also consider not only the current emotion value of the robot 100, but also the history of the robot 100's past emotion values ​​to determine the action corresponding to the action of the user 10. The action determined by the action determination unit 236 includes gestures performed by the robot 100 or the content of the robot 100's speech.

[0122] The action determination unit 236 in this embodiment determines the action of the robot 100 based on the combination of the user 10's past and current emotional values, the robot 100's emotional value, the user 10's action, and the reaction rule 221, as the action corresponding to the user 10's action. For example, if the user 10's past emotional value is positive and its current emotional value is negative, the action determination unit 236 determines an action to change the user 10's emotional value to positive, as the action corresponding to the user 10's action.

[0123] In response rule 221, actions of robot 100 are determined that correspond to a combination of user 10's past and current emotional values, robot 100's emotional value, and user 10's actions. For example, if user 10's past emotional value is positive and current emotional value is negative, and user 10's action is feeling sad, a combination of gestures and spoken content used to encourage user 10's inquiry is determined as robot 100's action.

[0124] For example, in response rule 221, the robot 100's action is determined based on the patterns of the robot 100's emotional values ​​(the four powers of the six values ​​"0" to "5" for "joy", "anger", "sorrow", and "happiness", i.e., 1296 patterns), the patterns of combinations of the user 10's past and current emotional values, and all combinations of the user 10's action patterns. That is, for each pattern of the robot 100's emotional values, for each combination of the user 10's past and current emotional values, such as negative values ​​with negative values, negative values ​​with positive values, positive values ​​with negative values, positive values ​​with positive values, negative values ​​with normal values, and normal values ​​with normal values, etc., the robot 100's action corresponding to the user 10's action pattern is determined. It should be noted that the action determination unit 236 can also migrate to the action pattern determined by the history data 222, for example, when the user 10 is speaking with the intention of continuing a conversation from a past topic such as "I want to talk about that topic we talked about before."

[0125] It should be noted that, in response rule 221, for each of the 1296 patterns of the robot 100's emotion value, at most one item, namely, gesture or speech content, can be determined as the robot 100's action. Alternatively, in response rule 221, for each pattern group of the robot 100's emotion value pattern group, at least one item, namely, gesture or speech content, can be determined as the robot 100's action.

[0126] The intensity of each gesture included in the actions of robot 100 as defined in response rule 221 is predetermined. Similarly, the intensity of each speech included in the actions of robot 100 as defined in response rule 221 is predetermined.

[0127] The storage control unit 238 determines whether to store data including the user 10's actions in the history data 222 based on the intensity of the actions predetermined for the actions determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0128] Specifically, if the overall intensity value is above a threshold, it is determined that data containing the actions of user 10 will be stored in the resume data 222. The overall intensity value is the sum of the following: the sum of the emotion values ​​of each of the multiple emotion categories for robot 100, the intensity predetermined for the gestures included in the actions determined by action determination unit 236, and the intensity predetermined for the speech content included in the actions determined by action determination unit 236.

[0129] When the storage control unit 238 determines that data containing the actions of user 10 should be stored in the history data 222, it stores the actions determined by the action determination unit 236, the information parsed by the sensor module unit 210 from the current time point to a certain period up to the present time (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the status of user 10 identified by the user status recognition unit 230 (e.g., user 10's facial expressions, emotions, etc.) in the history data 222.

[0130] The action control unit 250 controls the controlled object 252 based on the action determined by the action determination unit 236. For example, if the action determination unit 236 determines that the action includes speaking, the action control unit 250 causes sound to be output from the speaker provided by the controlled object 252. At this time, the action control unit 250 can also determine the speed of sound production based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the speed of sound production determined by the action control unit 250. Thus, the action control unit 250 determines the execution method of the action determined by the action determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0131] The action control unit 250 can also recognize changes in the user 10's emotions in response to actions determined by the action determination unit 236. For example, changes in emotions can be recognized based on the user 10's voice or facial expressions. Furthermore, changes in the user 10's emotions can be recognized based on the detection of an impact by the touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the user 10's emotions are identified as worsening; if the detection result from the touch sensor included in the sensor unit 200 indicates that the user 10's reaction is laughter or joy, the user 10's emotions are identified as improving. Information representing the user 10's reaction is then output to the communication processing unit 280.

[0132] Furthermore, after the action control unit 250 executes the action determined by the action determination unit 236 according to the execution method determined by the robot 100's emotion, the emotion determination unit 232 further changes the robot 100's emotion value based on the user's reaction to the execution of the action. Specifically, if the user's reaction to the situation where the action determined by the action determination unit 236 is performed on the user according to the execution method determined by the action control unit 250 is not negative, the emotion determination unit 232 increases the robot 100's "joy" emotion value. Conversely, if the user's reaction to the situation where the action determined by the action determination unit 236 is performed on the user according to the execution method determined by the action control unit 250 is negative, the emotion determination unit 232 increases the robot 100's "sorrow" emotion value.

[0133] Furthermore, the action control unit 250 expresses the emotions of the robot 100 based on the determined emotion value of the robot 100. For example, when the action control unit 250 increases the "joy" emotion value of the robot 100, it controls the controlled object 252 to make the robot 100 perform joyful actions. Conversely, when the action control unit 250 increases the "sorrow" emotion value of the robot 100, it controls the controlled object 252 to make the robot 100 adopt a posture of lowering its head.

[0134] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 sends user response information to the server 300. Additionally, the communication processing unit 280 receives updated response rules from the server 300. When the communication processing unit 280 receives updated response rules from the server 300, it updates response rule 221.

[0135] Server 300 enables communication between robots 100, 101, and 102 and server 300, receives user response information sent from robot 100, and updates response rules based on response rules that include actions that receive positive responses.

[0136] Figure 3 This diagram illustrates an example of a motion flow related to a determined action in robot 100. Repeated execution, such as... Figure 3 The action flow is shown below. At this point, it is assumed that information parsed by the sensor module 210 has been input. It should be noted that "S" in the action flow indicates the step being executed.

[0137] First, in step S100, the user status recognition unit 230 recognizes the status of the user 10 based on the information parsed by the sensor module unit 210.

[0138] In step S102, the emotion determination unit 232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230.

[0139] In step S103, the emotion determination unit 232 determines the emotion value of the robot 100 based on the information parsed by the sensor module unit 210 and the state of the user 10 identified by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user 10 to the history data 222.

[0140] In step S104, the action recognition unit 234 identifies the action category of user 10 based on the information parsed by the sensor module unit 210 and the state of user 10 identified by the user state recognition unit 230.

[0141] In step S106, the action determination unit 236 determines the action of the robot 100 based on the combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​contained in the resume data 222, the emotional value of the robot 100, the action of the user 10 identified by the action recognition unit 234, and the reaction rule 221.

[0142] In step S108, the action control unit 250 controls the controlled object 252 based on the action determined by the action determination unit 236.

[0143] In step S110, the storage control unit 238 calculates a comprehensive value of the intensity based on the intensity of the action predetermined for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0144] In step S112, the storage control unit 238 determines whether the overall intensity value is above or below a threshold. If the overall intensity value is below the threshold, data including the actions of user 10 is not stored in the history data 222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0145] In step S114, the actions determined by the action determination unit 236, the information parsed by the sensor module unit 210 from the current time point to a certain period ago, and the status of the user 10 identified by the user status identification unit 230 are stored in the history data 222.

[0146] As explained above, using robot 100, based on the user's state, an emotion value representing the robot 100's emotions is determined. Based on the robot 100's emotion value, it is determined whether to store data containing the user 10's actions in the history data 222. This reduces the capacity of the history data 222 containing data on the user 10's actions. Furthermore, for example, if robot 100 determines that the user's state 10 years from now is the same as 10 years ago, by reading the history data 222 from 10 years ago, robot 100 can present the user 10 with all surrounding information, including the user 10's state 10 years ago (e.g., the user 10's facial expressions, emotions, etc.), and even data such as sounds, images, and smells from that time.

[0147] Furthermore, robot 100 can be used to perform actions appropriate to the actions of user 10. Previously, user actions were categorized, determining the robot's actions, including facial expressions and gestures. In contrast, robot 100 determines user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if user 10 was in a good mood yesterday but is feeling down today, robot 100 can say something like, "You were quite energetic yesterday, what's wrong today?" Robot 100 can also combine gestures with speech. For example, if user 10 was feeling down yesterday but is feeling better today, robot 100 can say something like, "You were listless yesterday, but you look much more energetic today?" For example, if user 10 was in a good mood yesterday but is feeling even better today, robot 100 can say something like, "You're much more energetic today than yesterday. Did something even happier happen than yesterday?" Additionally, for example, if a user 10 continues to have an emotional value of 0 or higher and the fluctuation range of the emotional value is within a certain range, the robot 100 can say something like, "Your mood has been very stable lately, and you feel particularly good."

[0148] Furthermore, for example, if robot 100 asks user 10, "Have you finished the assignment you mentioned yesterday?" and receives a "Yes, I have!" from user 10, it can simultaneously express affirmation with words like "Wow, that's awesome!" and perform affirmative gestures such as clapping or giving a thumbs up. Similarly, if user 10 says, "The demonstration I mentioned the day before yesterday was successfully completed," robot 100 can also express affirmation with words like "Thank you for your hard work!" and perform the aforementioned affirmative gestures. In this way, by acting based on user 10's state history, robot 100 can be expected to develop a sense of closeness with user 100.

[0149] In the above embodiments, the use of the user 10's facial image to identify the user 10 has been described, but the disclosed technology is not limited to this method. For example, the robot 100 may also identify the user 10 using the user 10's voice, the user 10's email address, the user 10's SNS ID, or an ID card with a built-in wireless IC tag held by the user 10.

[0150] It should be noted that robot 100 is an example of an electronic device equipped with a motion control system. The application of the motion control system is not limited to robot 100; it can be applied to various electronic devices. Furthermore, the functions of server 300 can be implemented using more than one computer. At least some of the functions of server 300 can be implemented using a virtual machine. Additionally, at least some of the functions of server 300 can be implemented in the cloud.

[0151] Figure 4 An example of the hardware configuration of a computer 1200 that functions as both a robot 100 and a server 300 is shown in summary. Programs installed in the computer 1200 enable it to function as one or more "parts" of the apparatus according to this embodiment, or to perform operations associated with the apparatus or those "parts," and / or to execute processes or stages of those processes according to this embodiment. Such programs can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the frames in the flowcharts and block diagrams described in this specification.

[0152] The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, and graphics controller 1216 interconnected with each other via a host controller 1210. The computer 1200 includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card driver, which can be connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 can be a DVD-ROM drive or a DVD-RAM drive, etc. The storage device 1224 can be a hard disk drive or a solid-state drive, etc. The computer 1200 includes a ROM 1230 and conventional input / output units such as a keyboard, which can be connected to the input / output controller 1220 via an input / output chip 1240.

[0153] The CPU 1212 operates according to the program stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 can obtain image data generated by the CPU 1212 from the frame buffer or the like provided in RAM 1214 or from RAM itself, and display the image data on the display device 1218.

[0154] The communication interface 1222 is capable of communicating with other electronic devices via a network. The storage device 1224 is capable of storing programs and data used by the CPU 1212 within the computer 1200. The DVD drive 1226 reads programs or data from the DVD-ROM 1227, etc., and provides them to the storage device 1224. The IC card driver reads programs and data from the IC card, and / or writes programs and data to the IC card.

[0155] The ROM 1230 stores boot programs and other programs that are executed by the computer 1200 at startup, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0156] The program is provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium, installed in a storage device 1224, RAM 1214, or ROM 1230 (also examples of computer-readable storage media), and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, enabling cooperation between the program and the aforementioned hardware resources of various types. The apparatus or method can be configured to perform information manipulation or processing according to the use of the computer 1200.

[0157] For example, when communication is performed between computer 1200 and an external device, CPU 1212 can execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, command communication interface 1222 to perform communication processing. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM 1227, or IC card, and sends the read transmission data to the network, or writes received data received from the network into a receive buffer provided on the recording medium.

[0158] Furthermore, the CPU 1212 can read all or necessary portions of files or databases stored in external recording media such as storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc., into RAM 1214, and perform various types of processing on the data in RAM 1214. Next, the CPU 1212 can write the processed data back to the external recording medium.

[0159] Various types of information, such as programs, data, tables, and databases, can be stored in recording media for information processing. The CPU 1212 performs various types of processing on data read from RAM 1214 and writes the results back to RAM 1214. These various types of processing include operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc., specified by a sequence of program instructions and described throughout this disclosure. Furthermore, the CPU 1212 can retrieve information from files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each entry having an attribute value of a first attribute associated with the attribute value of a second attribute, the CPU 1212 can retrieve from these multiple entries an entry whose first attribute value matches a specified condition, and read the attribute value of the second attribute stored in that entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0160] The programs or software modules described above can be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0161] In this embodiment, the boxes in the flowcharts and block diagrams may represent stages of a process for performing an operation or "parts" of a device that performs the operation. Specific stages and "parts" may be implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable storage medium, and / or processors supplied with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, and storage elements.

[0162] Computer-readable storage media can include any tangible device capable of storing instructions executable by a suitable device. As a result, a computer-readable storage medium having instructions stored therein comprises an article including the instructions, which can be executed to generate units for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random-access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray discs, memory sticks, integrated circuit cards, etc.

[0163] Computer-readable instructions may include any of the following: assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code described in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA (registered trademark), C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0164] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to the processor or programmable circuitry of a general-purpose computer, special-purpose computer, or other programmable data processing device, causing the processor or programmable circuitry to execute the computer-readable instructions to generate units for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0165] The present invention has been described above using embodiments, but the scope of the present invention is not limited to the scope described in the above embodiments. Those skilled in the art should understand that various modifications or improvements can be made to the above embodiments. As can be seen from the claims, embodiments with such modifications or improvements may also be included within the scope of the present invention.

[0166] It should be noted that the execution order of actions, sequences, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specification, and drawings is not specifically stated as "before," "earlier than," etc., or can be implemented in any order as long as the output of the preceding process is not used in the subsequent process. Even if the flow of actions in the claims, specification, and drawings is described using terms such as "firstly" or "next" for convenience, it does not mean that the actions must be performed in that order.

[0167] (Other implementation method 1) The action determination unit 236 generates action content based on the dialogue function that enables the user to converse with the robot, including the robot's actions towards the user and the user's or robot's emotions, and determines the robot's action corresponding to the action content. At this time, if a pre-set emotion-related threshold is exceeded for the user, the action determination unit 236 determines a pre-set robot action to soothe the user's emotions.

[0168] Specifically, the action determination unit 236 pre-sets an emotional level threshold allowed by the user. When the emotional level exceeds the allowed range (threshold) (for example, when the user is in a state of uncontrollable anger), the robot speaks to soothe the user's emotions according to the actions preset by the user.

[0169] Furthermore, since the judgment of whether the emotional level exceeds the allowable range varies depending on whether the user identifies themselves as having rich emotional expression or being calm and composed, the robot can adjust the emotional level threshold based on standard values. Alternatively, the user can preset the emotional level threshold. This allows for the support of the user's emotional control.

[0170] (Other implementation method 2) Action determination unit 236 executes dialogues with the user multiple times, performs a personality analysis of the user based on the results of the multiple executions of the dialogues, and reports the results of the personality analysis to the user.

[0171] In other words, the action determination unit 236 can first conduct a personality analysis based on psychological principles through conversation with the user. Based on this, the analysis results can be communicated to the user during the conversation, deepening the user's understanding of their own personality. It should be noted that there are no particular limitations on the methods used for the aforementioned personality analysis.

[0172] Furthermore, as a method for conveying analysis results to users, it is preferable to use indirect methods such as modifying the conversation content from the robot to reflect the analysis results, so that the analysis results can be conveyed without hindering the conversation with the user.

[0173] (Other implementation method 3) The robot 100 of this embodiment is characterized by its action system comprising: an emotion determination unit that determines the emotion of a user or the emotion of a robot; and an action determination unit that, based on a dialogue function that enables the user to converse with the robot, generates robot action content based on the user's actions and the user's or robot's emotions, and determines the robot's actions corresponding to the action content; the emotion determination unit determines the emotion of a protected user classified as a protected user based on reading information, the reading information including at least audio information of a book read aloud by the protected user to the protected user; the action determination unit determines the user's reaction during reading based on the protected user's emotion, presenting a book similar to the read book when the protected user's reaction is good, and presenting information about a book of a different category from the read book when the protected user's reaction is bad.

[0174] like Figure 2As shown, the action determination unit 236 sets the guest conversation mode as the conversation mode of the robot 100. In this guest conversation mode, it is positioned as a conversation partner when the user does not need to speak to a specific person but wants someone to listen to him. In this guest conversation mode, when talking to the user, predetermined keywords related to a specific person are excluded and the speech content is output.

[0175] Robot 100 detects user 10 who wants to talk to others, but not to family, friends, or lovers, such as when serving customers like a bar owner. It sets prohibited (NG) keywords like "family," "friends," and "lovers," and outputs speech content that absolutely does not contain these NG keywords. In this way, sensitive conversational content is never mentioned, allowing user 10 to enjoy stress-free conversations.

[0176] That is, Robot 100 will listen to what people want to say to others, but not to family, friends, lovers, etc. It can create a hospitality environment like a bar that treats guests in a one-on-one (more accurately, human to robot) manner.

[0177] In a scenario where guests are being entertained, if the robot not only engages in conversation but also interprets the emotions conveyed in the conversation and recommends suitable drinks, it can help users relieve their worries and release stress.

[0178] Thus, according to other implementations, when a certain intention is detected from user 10 (including key operation commands, action commands, voice commands from user 10 and automatic judgment by robot 100), a customer service dialogue mode is selected to construct a scenario where robot 100 listens to people as if they were bar owners (customer service scenario).

[0179] It should be noted that in the guest reception dialogue mode, the robot 100 can also set the indoor atmosphere (lighting, music, and sound effects, etc.). The atmosphere can be determined based on the emotional information from the conversation with user 10. For example, lighting could include dim lighting or lighting using mirrored spheres; music could include jazz or enka; and sound effects could include clinking glasses, opening and closing doors, or the sound of mixing cocktails. However, these are not limited to these examples, and the preferred method is as described later. Figure 5 as well as Figure 6 The robot 100 can also store scent-based components and output scents in conjunction with the user 10's speech. Examples of scents include perfume, the smell of baked cheese such as pizza, the sweetness of crepes, and the caramelized aroma of soy sauce such as grilled chicken skewers.

[0180] (Other implementation method 4) The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0181] That is, when the user 10 has a positive emotion as the robot 100 acts, the emotion determination unit 232 increases the emotion value representing the intensity of the emotion; when the user 10 has a negative emotion as the robot 100 acts, the emotion determination unit 232 decreases the emotion value representing the intensity of the emotion.

[0182] Here, the aforementioned positive emotions can also be defined as the application of at least one of the following: joy, happiness, pleasure, peace of mind, excitement, stability, and a sense of fulfillment. The aforementioned negative emotions can also be defined as the application of at least one of the following: anger, sorrow, unhappiness, unease, grief, worry, and a sense of emptiness.

[0183] By persistently running the above feedback loop, the content of the conversation can evolve in the direction of positive emotions in the listener (user 10).

[0184] (Other implementation method 5) The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0185] The action determination unit 236 determines the action of the robot 100 corresponding to the user's state and the emotions of the user 10 or the robot 100, based on an article generation model that has a dialogue function that enables the user 10 to converse with the robot 100. At this time, the action determination unit 236 is configured to execute the dialogue based on the user 10's emotional history and the context of the dialogue between the user 10 and the robot 100.

[0186] Specifically, the action determination unit 236 obtains the emotion value of user 10 from the emotion determination unit 232. Additionally, the action determination unit 236 obtains the history data 222 from the storage unit 220. Furthermore, the action determination unit 236 obtains text information representing the content of user 10's speech from the user state recognition unit 230. Moreover, the action determination unit 236 adds a fixed phrase, "How should we reply to the user at this time?", to the user 10's emotion value, history data 222, and text information, and inputs it into the article generation model. As described above, since the article generation model is composed of a large language model, it can generate a reply to user 10 based on the user 10's emotion history and the context of the dialogue between user 10 and robot 100.

[0187] Furthermore, the action determination unit 236 determines the speech given in response to user 10 as the action of robot 100, and the action control unit 250 controls the controlled object 252 to give a speech in response to user 10.

[0188] In this way, robot 100 can perform dialogue based on user 10's emotional history and the context of the conversation between user 10 and robot 100.

[0189] (Other implementation method 6) The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0190] The action determination unit 236 determines the actions of the robot 100 corresponding to the user's state and the emotions of the user 10 or the robot 100, based on an article generation model that enables the user 10 to converse with the robot 100. At this time, the action determination unit 236 generates a chart obtained by statistically processing the emotions of the user 10 over a predetermined period.

[0191] Specifically, after statistically processing the emotions of user 10 on any given day (e.g., April 14, 2023), the distribution of emotions is set as follows: "joy" 37%, "anger" 15%, "sorrow" 15%, and "happiness" 32%. In this case, the action determination unit 236 can generate a pie chart representing the distribution of these emotions as a chart for April 14, 2023. The action determination unit 236 can perform this statistical processing over multiple days (e.g., April 10, 2023 to April 25, 2023). Furthermore, the action determination unit 236 can also perform statistical processing on the emotions of user 10 from April 10, 2023 to April 25, 2023, generating a chart representing the shift in the distribution of emotions (e.g., a cumulative chart that accumulates the distribution of emotions by day in a time series). The action determination unit 236 can generate such pie charts and cumulative charts as statistical processing results, for example.

[0192] In addition, the distribution of user 10's emotions is set as follows: "laughter" 30%, "anger" 10%, "joy" 15%, "crying" 20%, and "no special state" 20%. Even under these conditions, the action determination unit 236 is still able to generate pie charts and cumulative charts.

[0193] Action determination unit 236 can, for example, generate a chart representing emotions over a day, and also generate a chart representing the daily shift in emotions.

[0194] For example, the generated chart can be displayed in various formats. As an example, the generated chart can also be provided to the display device in the control object 252 and displayed thereon. Alternatively, the generated chart can also be provided to other terminals (e.g., terminals owned by the protector of user 10, such as parents) and displayed through the display unit of those other terminals, upon request.

[0195] (Other implementation methods 7) In the robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N), the action determination unit 236 generates a record of events for a predetermined period based on history data containing the history of actions of the user 10. The robot 100 of this embodiment, for example, stores events including conversations with the user 10 as a child. Specifically, the storage unit 220 stores events acquired by the microphone 201 or the 2D camera 203 for one day. It should be noted that, as events, history data 222 containing the past emotional values ​​and action history of the user 10 can also be used as event data. Furthermore, at the end of the day, a predetermined scene is selected based on the stored events, and the selected scene is textualized. Here, for the textualized scene, one or more scenes are selected where the emotional values ​​determined by the emotional determination unit 232 (the emotional values ​​of the user 10, the emotional values ​​of the robot 100, or the sum of the emotional values ​​of the user 10 and the emotional values ​​of the robot 100) satisfy a predetermined selection criterion. Selection criteria include: highest sentiment value, strongest specific emotion among multiple sentiment categories such as "joy," and most smiling faces from users. For example, based on resume data 222, the scenario with the highest sentiment value is selected, and the actions of user 10 stored in resume data 222 within the selected scenario are retrieved, and the retrieved actions of user 10 are textualized. For example, in this way, the action determination unit 236 can also generate a diary recording daily events at the end of the day.

[0196] Additionally, the Action Determination Department 236 can also generate original story picture books based on events over a period of several days. In this case, the Action Determination Department 236 can also use the generation system AI to generate the story and AI drawing software to generate the illustrations. It should be noted that AI drawing software, such as DALL·E2 (web search), can be used...<URL: https: / / openai.com / product / dall-e-2> As disclosed, this is common knowledge, so detailed descriptions are omitted. Furthermore, in addition to storage in storage unit 220, the AI ​​drawing software can also be stored in other devices connected to robot 100 via a network. Moreover, the action determination unit 236 can automatically capture dynamic images for a certain period (e.g., within 15 seconds) when the emotion value determined by emotion determination unit 232 is high. At the end of the day, it selects dynamic images of several scenes (e.g., four scenes) with high emotion values ​​and high ranking, thereby generating a record of events. Additionally, the action determination unit 236 can also generate images based on events with high emotion values ​​(e.g., vivid memories). In this case, the action determination unit 236 can also use the AI ​​drawing software to generate an imagined image, i.e., a fictional image, that is similar to the actual experience, rather than an image exactly like the user 10's actual experience. In this case, multiple events can be combined to generate such an image, allowing more memorable events to be stored in the cloud or elsewhere for a longer period.

[0197] For example, the record of the event thus generated can be provided in various forms. As an example, the record of the generated event can also be provided to the display device in the control object 252 and displayed thereon. Alternatively, the record of the generated event can also be provided to other terminals (e.g., terminals owned by the protectors of user 10 (parents, grandparents, etc.), televisions, etc.) and displayed through the display unit in those other terminals.

[0198] Robot 100 (including Robot 100 which is in the form of a plush toy 100N) performs the process of creating a diary through the following steps 1 to 4.

[0199] (Step 1) Robot 100 acquires the status of user 10, the emotional value of user 10, the emotional value of robot 100, the resume data 222, and the event data. Specifically, it performs the same processing as steps S100 to S103 above to acquire the status of user 10, the emotional value of user 10, the emotional value of robot 100, the resume data 222, and the event data.

[0200] (Step 2) Robot 100 selects the scene for generating the diary.

[0201] Specifically, at the end of the day, the Action Determination Department 236 reviews the day's events based on the resume data 222 and selects the scenario for generating a diary.

[0202] (Step 3) Robot 100 generates a diary.

[0203] Specifically, the action determination unit 236 transcribes the scenario selected in step 2 into text to generate a diary.

[0204] (Step 4) Robot 100 outputs the generated diary to a predetermined object.

[0205] In this way, robot 100 can perform the processing of the creation diary based on the resume data 222.

[0206] The emotion determination unit 232 can determine the user's emotion based on a specific mapping. Specifically, the emotion determination unit 232 can determine the user's emotion based on a specific mapping, namely an emotion map (see...). Figure 5 ), to determine the user's emotions.

[0207] Figure 5 This is a diagram illustrating an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. Further outward from the concentric circles, emotions representing states or actions derived from mood are arranged. Emotion refers to the concept that also includes feelings or mental states. On the left side of the concentric circles, emotions generated by reactions produced internally by the brain are arranged roughly. On the right side of the concentric circles, emotions guided by situational judgments are arranged roughly. Above and below the concentric circles, emotions generated by reactions produced internally by the brain and guided by situational judgments are arranged roughly. In addition, the emotion of "happiness" is arranged above the concentric circles, and the emotion of "unhappiness" is arranged below. Thus, in the emotion map 400, based on the structure that generates emotions, multiple emotions are mapped, and emotions that are likely to occur simultaneously are mapped closer together.

[0208] (1) For example, if the emotion engine of robot 100, which serves as the emotion determination unit 232, detects emotions at a period of approximately 100 milliseconds (msec), the frequency for determining the reaction action (e.g., agreement) to robot 100 can be set to at least the same timing as the detection frequency (100 msec) of the emotion engine, or it can be set to a timing faster than the aforementioned timing. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0209] Emotions are detected at intervals of approximately 100 milliseconds, and immediate responses (such as agreement) are initiated. This prevents unnatural agreement and enables natural, empathetic dialogue. Robot 100 responds (e.g., agrees) based on the directionality and intensity of the mandala-like pattern in the emotion map 400. It should be noted that the detection frequency (sampling rate) of the emotion engine is not limited to 100 milliseconds (ms) and can be adjusted according to the scenario (such as movement) and the user's age.

[0210] (2) It can be compared with the emotion map 400 to pre-set the directionality and intensity of the emotion, as well as the action of agreement and the strength of agreement. For example, when robot 100 feels secure and at ease, robot 100 will nod and continue to listen. When robot 100 feels uneasy, confused, or doubtful, robot 100 can tilt its head or stop shaking its head.

[0211] These emotions are distributed at the 3 o'clock position on the Emotion Chart 400, and typically fluctuate between peace and anxiety. In the right half of the Emotion Chart 400, situational awareness is more dominant than inner feelings, thus giving an impression of composure.

[0212] (3) When Robot 100 feels happy because of being praised, a filler word such as "Ah—" can be added before the dialogue. When it feels pain because of hearing harsh words, a filler word such as "Ugh!" can be added before the dialogue. In addition, Robot 100 can also add physical reactions such as squatting while saying "Ugh!". These emotions are distributed in the 9 o'clock position of the emotion diagram 400.

[0213] (4) In the left half of the emotion diagram 400, internal feelings (reactions) are more dominant than situation recognition. Therefore, it can give the impression of subconscious reactions.

[0214] When Robot 100 experiences an intrinsic feeling (reaction) of recognition and also senses liking in situational awareness, it can nod vigorously while looking at the other party and make an "uh-huh" sound. In this way, Robot 100 can generate a balanced and moderate level of liking for the other party, that is, generate actions such as permission and tolerance. This emotion is distributed at the 12 o'clock position on the emotion diagram 400.

[0215] Conversely, if Robot 100 experiences both unpleasantness (internal feeling) and disgust in its situational awareness, it will simply shake its head. If it experiences hatred, it can turn its LEDs red and stare at the other party. This emotion is distributed at the 6 o'clock position on the emotion map 400.

[0216] (5) The inner side of the emotion diagram 400 represents the inner state, and the outer side of the emotion diagram 400 represents the action. Therefore, the closer to the outer side of the emotion diagram 400, the easier it is to see the emotion (and the easier it is to be expressed in the action).

[0217] (6) When feeling at ease in the 3 o'clock position of the emotional map 400 and listening to others, the robot 100 nods gently and makes an "uh-huh" sound. However, if it becomes love in the 12 o'clock position, it can nod vigorously.

[0218] The emotion determination unit 232 inputs the information parsed by the sensor module unit 210 and the identified state of the user 10 into the pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of the user 10. This neural network is pre-learned based on multiple learning data sets, which are combinations of the information parsed by the sensor module unit 210, the identified state of the user 10, and the emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 6 As shown in the emotion map 900, emotions configured close to each other are learned in a similar manner. Figure 6 The text shows examples of multiple emotions such as "peace of mind", "stability", and "ease of mind" having similar emotional values.

[0219] Furthermore, the emotion determination unit 232 can determine the emotion of the robot 100 based on a specific mapping. Specifically, the emotion determination unit 232 inputs the information parsed by the sensor module unit 210, the state of the user 10 identified by the user state recognition unit 230, and the state of the robot 100 into a pre-learned neural network, and obtains the emotion values ​​representing each emotion shown in the emotion map 400 to determine the emotion of the robot 100. This neural network is pre-learned based on multiple learning data, which are combinations of the information parsed by the sensor module unit 210, the identified state of the user 10, the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400. For example, the neural network learns based on the following learning data: learning data that identifies the robot 100 touching the user 10 based on the output of the touch sensor (not shown) with an emotion value of "3" representing "joy", and learning data that identifies the robot 100 hitting the user 10 based on the output of the accelerometer (not shown) with an emotion value of "3" representing "anger". Additionally, this neural network... Figure 6 As shown in the emotion diagram 900, emotions that are arranged close to each other learn in a similar way.

[0220] The action determination unit 236 adds fixed statements to the text representing the user's actions, the user's emotions, and the robot's emotions, to ask questions about the robot's actions corresponding to the user's actions, and generates the robot's action content by inputting it into a text generation model with dialogue functionality.

[0221] For example, the action determination unit 236 uses the emotion table shown in Table 1 to obtain text representing the state of the robot 100 based on the emotion of the robot 100 determined by the emotion determination unit 232. Here, in the emotion table, each emotion value is assigned an index number according to the type of emotion, and the text representing the state of the robot 100 is stored according to the index number.

[0222] When the emotion of robot 100 determined by emotion determination unit 232 corresponds to index number "2", the text "is in a very happy state" is obtained. It should be noted that when the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the state of robot 100 are obtained.

[0223] Additionally, there is an emotional preparation table shown in Table 2 for User 10.

[0224] Here, when the user initiates a conversation with the question "How are you? Are you happy?" and the robot 100's emotion is index number "2" and the user 10's emotion is index number "3", the article generation model is input with the following question: "The robot is in a very happy state. The user is in a generally happy state. The user asked me, 'How are you? Are you happy?'. As a robot, how should I respond?", and the robot's action content is obtained. The action determination unit 236 determines the robot's action based on this action content.

[0225] Table 1

[0226] Table 2

[0227] Thus, the action determination unit 236 determines the action content of the robot 100 in accordance with the following: a state related to each emotion of the robot 100, predetermined according to the intensity of each emotion, and the actions of the user 10. In this way, the robot 100's speech content during a dialogue with the user 10 can be branched based on the state related to the robot 100's emotion. That is, the robot 100 can change its actions according to the index number corresponding to the robot's emotion, thus giving the user the impression that the robot has autonomous consciousness, thereby encouraging actions such as striking up a conversation with the robot.

[0228] Furthermore, the action determination unit 236 can add not only text representing the user's actions and emotions, as well as the robot's emotions, but also text representing the content of the resume data 222. Based on this, it can add fixed phrases for asking questions about the robot's actions corresponding to the user's actions, and input these into a dialogue-enabled text generation model to generate the robot's action content. Thus, the robot 100 can change its actions based on the resume data representing the user's emotions and actions, thereby giving the user the impression that the robot has a personality, which encourages actions such as engaging in conversation with the robot. Alternatively, the resume data can also include the robot's emotions and actions.

[0229] Furthermore, the emotion determination unit 232 can also determine the emotion of the robot 100 based on the action content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 232 inputs the action content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion graph 400, integrates the obtained emotion values ​​of each emotion with the emotion values ​​representing the current emotions of the robot 100, and updates the emotion of the robot 100. For example, the emotion values ​​representing each obtained emotion and the emotion values ​​representing the current emotions of the robot 100 are averaged and then integrated. This neural network is pre-learned based on multiple learning data, which are combinations of text representing the action content of the robot 100 generated by the article generation model and emotion values ​​representing each emotion shown in the emotion graph 400.

[0230] For example, if the robot 100's speech content "That's great! You're so lucky!" is received as the action content of the robot 100 generated by the article generation model, and if the text representing the speech content is input into the neural network, and a high value is obtained as the emotion "joy", the robot 100's emotion is updated to make the emotion "joy" even higher.

[0231] It should be noted that the robot 100 can be mounted on a plush toy, or it can be used as a control device connected wirelessly or via a wired connection to a control object device (speaker, camera) mounted on the plush toy. Specifically, in this case, it can be configured as follows. For example, the robot 100 can also be used with such cohabiting individuals (specifically, Figure 7 and Figure 8 (The illustrated plush toy 100N): While interacting with the user 10 in their daily life, it promotes dialogue with the user 10 based on information related to daily life, or provides information matching the user 10's interests and preferences. In this embodiment (other embodiments), an example of applying the control part of the above-described robot 100 to a smartphone 50 will be described.

[0232] The plush toy 100N, which is equipped with the input / output device of the robot 100, can be detachably configured with a smartphone 50 that functions as the control part of the robot 100. The input / output device and the smartphone 50 are connected inside the plush toy 100N.

[0233] like Figure 7 As shown in (A), in this embodiment (the embodiment mounted on a plush toy), the plush toy 100N is a bear-shaped object with its exterior covered in soft fabric, as... Figure 7 As shown in (B), the internal space 52 serves as an input / output device, and a microphone 201 of the sensor unit 200 is disposed in the portion corresponding to the ear 54 (see Figure 54). Figure 2 A 2D camera 203 with a sensor unit 200 is configured in the part corresponding to the eye 56 (see...). Figure 2 ), and in the portion corresponding to mouth 58, there is a component constituting control object 252 (see Figure 2 The microphone 201 and speaker 60 are part of the speaker unit 60. It should be noted that the microphone 201 and speaker 60 are not necessarily separate units; they can also be an integrated unit. If it is an integrated unit, it can be positioned where speech naturally comes out, such as at the nose of the plush toy 100N. It should be noted that the example given is of the plush toy 100N in the shape of an animal, but it is not limited to this. The plush toy 100N can also be in the shape of a specific character.

[0234] Smartphone 50 has Figure 2 The functions shown are as follows: sensor module 210, storage unit 220, user status recognition unit 230, emotion determination unit 232, action recognition unit 234, action determination unit 236, storage control unit 238, action control unit 250, and communication processing unit 280.

[0235] like Figure 8 As shown, the toy 100N is configured such that a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, the outside is connected to the space portion 52.

[0236] Here, the smartphone 50 is housed externally in the space 52 via the USB hub 64 (see Figure 7 (B) can connect to each input / output device via USB, thereby enabling communication with... Figure 1 It has the same functions as the robot 100 shown.

[0237] Additionally, the contactless power receiver 66 is connected to the USB hub 64. A power receiving coil 66A is mounted on the power receiver 66. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0238] The receiving plate 66 is positioned near the base 68 of the two legs of the plush toy 100N, and is the closest point to the mounting base 70 when the plush toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0239] The plush toy 100N placed on the base 70 can be displayed as a decorative item in its natural state.

[0240] In addition, the root is formed in a way that is thinner than the surface thickness of the other parts of the plush toy 100N, and is held in a state that is closer to the mounting base 70.

[0241] The mounting base 70 includes a charging pad 72. The charging pad 72 is equipped with a power transmission coil 72A. The power transmission coil 72A sends a signal to locate the power receiving coil 66A on the power receiving board 66. When the power receiving coil 66A is found, current flows through the power transmission coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. Thus, current flows through the power receiving coil 66A, and via the USB hub 64, stores power in the battery (not shown) of the smartphone 50.

[0242] That is, by placing the plush toy 100N as an ornament on the base 70, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0243] It should be noted that in this embodiment (the embodiment mounted on a plush toy), the smartphone 50 is housed in the space 52 of the plush toy 100N and a wired connection (USB connection) is established, but it is not limited to this. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) may also be housed in the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, without placing the smartphone 50 in the space 52, the smartphone 50 communicates wirelessly with the control device, and an external smartphone 50 connects to various input / output devices through the control device, thereby enabling communication with... Figure 1 The robot 100 shown has the same functions. Alternatively, the control device housed in the space 52 of the plush toy 100N can be connected to an external smartphone 50 via a wired connection.

[0244] Furthermore, in this embodiment (the embodiment mounted on a plush toy), a bear plush toy 100N is used as an example, but it can be any other animal shape, a doll shape, or a specific character shape. Additionally, the clothes can be changed. Furthermore, the material of the outer skin is not limited to cloth; it can be made of other materials such as soft plastic, but a soft material is preferred.

[0245] Furthermore, a display can be installed on the surface of the plush toy 100N, adding a control object 252 that provides visual information to the user 10. For example, the eyes 56 can be used as a display, expressing emotions such as joy, anger, sorrow, and happiness through images projected into the eyes. Alternatively, a window can be provided on the abdomen through which the display of a built-in smartphone 50 can be viewed. Furthermore, the eyes 56 can also be used as a projector, expressing emotions such as joy, anger, sorrow, and happiness through images projected onto a wall.

[0246] According to other embodiments, an existing smartphone 50 is placed inside the plush toy 100N, and then, via USB connection, the camera 203, microphone 201, speaker 60, etc. are extended to appropriate positions.

[0247] Furthermore, for wireless charging, the smartphone 50 is connected to the power receiving board 66 via USB, and the power receiving board 66 is configured to be viewed from the inside of the plush toy 100N as far outward as possible.

[0248] To use the wireless charging of the smartphone 50, the smartphone 50 must be configured to look as far outward as possible from the inside of the plush toy 100N. When touching the plush toy 100N from the outside, an uneven feel will be produced.

[0249] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, and the wireless charging function (power receiving board 66) is positioned as far outward as possible when viewed from the inside of the plush toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving board 66.

[0250] [Second Implementation] Figure 1An example of system 5 according to this embodiment is shown in summary. System 5 includes robot 100, robot 101, robot 102, and server 300. Users 10a, 10b, 10c, and 10d are users of robot 100. Users 11a, 11b, and 11c are users of robot 101. Users 12a and 12b are users of robot 102. It should be noted that in the description of this embodiment, users 10a, 10b, 10c, and 10d are sometimes collectively referred to as user 10. In addition, users 11a, 11b, and 11c are sometimes collectively referred to as user 11. In addition, users 12a and 12b are sometimes collectively referred to as user 12. Robots 101 and 102 have substantially the same functions as robot 100. Therefore, system 5 will be described mainly with respect to the functions of robot 100.

[0251] Robot 100 engages in conversations with user 10 or provides images to user 10. During this time, robot 100 collaborates with a communicable server 300 via communication network 20 to conduct conversations with user 10 and provide images. For example, robot 100 not only learns appropriate conversational techniques on its own but also collaborates with server 300 to learn how to better advance the conversation with user 10. Furthermore, robot 100 records image data of user 10 captured on the server 300, requests image data from server 300 as needed, and provides it to user 10.

[0252] Furthermore, robot 100 possesses emotion values ​​representing the types of emotions it experiences. For example, robot 100 has emotion values ​​indicating the intensity of each emotion: joy, anger, sorrow, happiness, happiness, unhappiness, peace of mind, unease, sadness, excitement, worry, stability, fulfillment, emptiness, and "normality." For instance, if robot 100 is in a state of high excitement when conversing with user 10, it will speak at a faster pace. In this way, robot 100 can express its emotions through actions.

[0253] Additionally, robot 100 can be configured to determine the actions of robot 100 corresponding to the emotions of user 10 by matching an article generation model using artificial intelligence (AI) with an emotion engine. Specifically, robot 100 is configured to recognize the actions of user 10, determine the emotions of user 10 towards those actions, and determine the actions of robot 100 corresponding to the determined emotions.

[0254] More specifically, upon recognizing the actions of user 10, robot 100 uses a pre-defined article generation model to automatically generate the appropriate actions for user 10. The article generation model can be interpreted as the algorithm and computation used for text-based automated dialogue processing. Examples of article generation models include Japanese Patent Application Publication No. 2018-081444 and ChatGPT (Web Search Platform).<URL: https: / / openai.com / blog / chatgpt> As disclosed, it is well-known, therefore its detailed explanation is omitted. This article generation model consists of a Large Language Model (LLM).

[0255] As described above, this embodiment enables the robot 100's actions to reflect the user 10's or robot 100's emotions and various linguistic information by combining a large language model with an emotion engine. In other words, according to this embodiment, a synergistic effect can be achieved by combining an article generation model with an emotion engine.

[0256] In addition, robot 100 has the function of recognizing the actions of user 10. Robot 100 analyzes the facial image of user 10 acquired through the camera function and the voice of user 10 acquired through the microphone function, thereby recognizing the actions of user 10. Based on the recognized actions of user 10, robot 100 determines the action to be performed.

[0257] As an example of an action determination model, Robot 100 stores the rules for the actions to be performed based on the emotions of User 10, the emotions of Robot 100, and the actions of User 10, and performs various actions according to the rules.

[0258] Specifically, in robot 100, as an example of an action determination model, there are reaction rules for determining the actions of robot 100 based on the emotions of user 10, the emotions of robot 100, and the actions of user 10. In the reaction rules, for example, if user 10's action is "laugh," then "laughing" is determined to be an action of robot 100. Furthermore, if user 10's action is "anger," then "apologizing" is determined to be an action of robot 100. Additionally, if user 10's action is "asking a question," then "answering" is determined to be an action of robot 100. Finally, if user 10's action is "feeling sad," then "starting a conversation" is determined to be an action of robot 100.

[0259] Based on reaction rules, when Robot 100 identifies User 10's action as "anger," it selects "apology," a behavior specified in the reaction rules, as the action to be performed by Robot 100. For example, when Robot 100 selects "apology," it performs the "apology" action while simultaneously outputting the sound of words expressing "apology."

[0260] In addition, when the robot 100's emotion is "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "happiness" = 0) and the user 10's state is "alone, looking somewhat lonely", the robot 100's emotion has the change content of "becoming worried" and is determined to be able to perform the action of "starting a conversation".

[0261] Based on reaction rules, if Robot 100 recognizes that its current emotion is "normal" and User 10 appears somewhat lonely, it will increase the "sadness" emotion value of Robot 100. Additionally, Robot 100 will select actions such as "starting a conversation" as the action to be performed on User 10, determined through reaction rules. For example, if Robot 100 selects the action of "starting a conversation," it will translate a worried question like "What's wrong?" into a concerned tone and output it.

[0262] Additionally, robot 100 sends user response information to server 300, indicating that it received a positive response from user 10 through this action. This user response information may include, for example, user actions such as "getting angry," robot 100 actions such as "apologizing," the positiveness of user 10's response, and user 10's attributes.

[0263] Server 300 stores user response information received from robot 100. It should be noted that server 300 not only receives and stores user response information from robot 100, but also from robots 101 and 102 respectively. Furthermore, server 300 parses the user response information from robots 100, 101, and 102 and updates the response rules accordingly.

[0264] Robot 100 receives the updated response rules from server 300 by querying server 300. Robot 100 then incorporates the updated response rules into its stored response rules. Thus, robot 100 is able to incorporate response rules obtained from robots 101, 102, etc., into its own response rules.

[0265] Figure 9AThe functional structure of robot 100 is shown in summary. Robot 100 includes a sensor unit 2200, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a controlled object 2252. The control unit 2228 includes a state recognition unit 2230, an emotion determination unit 2232, an action recognition unit 2234, an action determination unit 2236, a storage control unit 2238, an action control unit 2250, a related information collection unit 2270, and a communication processing unit 2280.

[0266] The controlled object 2252 includes a display device, speakers, LEDs for the eyes, and motors for driving the arms, hands, and legs. The robot 100's posture and behavior are controlled by controlling the motors for the arms, hands, and legs. Part of the robot 100's emotions can be expressed by controlling these motors. Additionally, the robot 100's facial expressions can be expressed by controlling the illumination state of the LEDs for its eyes. It should be noted that the robot 100's posture, behavior, and facial expressions are examples of the robot 100's attitude.

[0267] The sensor unit 2200 includes a microphone 2201, a 3D depth sensor 2202, a 2D camera 2203, a distance sensor 2204, a touch sensor 2205, and an accelerometer 2206. The microphone 2201 continuously detects sound and outputs sound data. It should be noted that the microphone 2201 can be mounted on the head of the robot 100 and has the function of dual-channel recording. The 3D depth sensor 2202 continuously illuminates an infrared pattern and analyzes the infrared pattern based on the infrared images continuously captured by the infrared camera, thereby detecting the outline of an object. The 2D camera 2203 is an example of an image sensor. The 2D camera 2203 captures images using visible light, generating visible light image information. The distance sensor 2204, for example, illuminates with laser light or ultrasonic waves to detect the distance to an object. It should be noted that the sensor unit 2200 may also include a clock, a gyroscope sensor, a sensor for motor feedback, etc.

[0268] It should be noted that, Figure 9A The components of the robot 100 shown, excluding the controlled object 2252 and the sensor unit 2200, are examples of the components possessed by the motion control system of the robot 100. The motion control system of the robot 100 designates the controlled object 2252 as the controlled object.

[0269] Storage unit 2220 includes an action determination model 2221, history data 2222, collected data 2223, and action planning data 2224. History data 2222 includes past emotional values ​​of user 10, past emotional values ​​of robot 100, and action history. Specifically, history data includes multiple event data, including the emotional values ​​of user 10, the emotional values ​​of robot 100, and the actions of user 10. Data including user 10's actions includes camera images representing user 10's actions. The history of emotional values ​​and the history of actions are recorded per user 10, for example, by establishing a correspondence with user 10's identification information. At least one part of storage unit 2220 is implemented using a storage medium such as a memory. It may also include a person database storing user 10's facial image, user 10's attribute information, etc. It should be noted that... Figure 9A Of the components of the robot 100 shown, the functions of the components other than the control object 2252, the sensor unit 2200, and the storage unit 2220 can be implemented by the CPU based on a program. For example, the functions of these components can be implemented as CPU actions through basic software (OS) and a program that operates on the OS.

[0270] The sensor module 2210 includes a voice emotion recognition unit 2211, a speech comprehension unit 2212, an expression recognition unit 2213, and a face recognition unit 2214. Information detected by the sensor unit 2200 is input to the sensor module 2210. The sensor module 2210 analyzes the information detected by the sensor unit 2200 and outputs the analysis result to the state recognition unit 2230.

[0271] The voice emotion recognition unit 2211 of the sensor module 2210 analyzes the voice of the user 10 detected by the microphone 2201 and identifies the user 10's emotions. For example, the voice emotion recognition unit 2211 extracts feature quantities such as the frequency components of the voice, and identifies the user 10's emotions based on the extracted feature quantities. The speech understanding unit 2212 analyzes the voice of the user 10 detected by the microphone 2201 and outputs text information representing the content of the user 10's speech.

[0272] The expression recognition unit 2213 recognizes the facial expressions and emotions of the user 10 based on images captured by the 2D camera 2203. For example, the expression recognition unit 2213 recognizes the facial expressions and emotions of the user 10 based on the shape and positional relationship of the eyes and mouth.

[0273] The face recognition unit 2214 recognizes the face of user 10. The face recognition unit 2214 recognizes user 10 by matching the facial image stored in the person database (not shown) with the facial image of user 10 captured by the 2D camera 2203.

[0274] The state recognition unit 2230 identifies the state of the user 10 based on the information parsed by the sensor module unit 2210. For example, using the parsing results from the sensor module unit 2210, it mainly performs perception-related processing. For example, it generates perception information such as "Dad is alone." and "There is a 90% probability that Dad doesn't smile." It then performs processing to understand the meaning of the generated perception information. For example, it generates meaning information such as "Dad is alone and looks somewhat lonely."

[0275] The status recognition unit 2230 identifies the status of the robot 100 based on information detected by the sensor unit 2200. For example, the status recognition unit 2230 identifies the remaining battery power of the robot 100, the brightness of the surrounding environment of the robot 100, etc., as the status of the robot 100.

[0276] The emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. For example, the information parsed by the sensor module unit 2210 and the identified state of the user 10 are input into a pre-learned neural network to obtain an emotion value representing the emotion of the user 10.

[0277] Here, the emotion value representing user 10's emotion refers to the positive or negative value of the user's emotion. For example, if the user's emotion is a bright emotion accompanied by pleasant and calm feelings, such as "joy," "happiness," "happiness," "peace of mind," "excitement," "stability," and "fulfillment," it is represented by a positive value; the brighter the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "unhappiness," "unease," "grief," "worry," and "emptiness," it is represented by a negative value; the more unpleasant the emotion, the larger the absolute value of the negative value. When the user's emotion is not any of the above ("normal"), it is represented by a value of 0.

[0278] In addition, the emotion determination unit 2232 determines the emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210, the information detected by the sensor unit 2200, and the state of the user 10 identified by the state recognition unit 2230.

[0279] The emotion value of Robot 100 includes emotion values ​​for each of the multiple emotion categories, such as values ​​(0~5) representing the intensity of "joy", "anger", "sorrow" and "happiness".

[0280] Specifically, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on a rule that updates the emotion value of the robot 100 by establishing a correspondence between the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0281] For example, if the state recognition unit 2232 detects that user 10 looks somewhat lonely, it will increase the "sad" emotion value of robot 100. Conversely, if the state recognition unit 2230 detects that user 10 is smiling, it will increase the "happy" emotion value of robot 100.

[0282] It should be noted that the emotion determination unit 2232 can also consider the state of the robot 100 and determine the emotion value representing the robot 100's emotion. For example, when the robot 100's battery has low remaining power or when the robot 100's surrounding environment is dark, the "sadness" emotion value of the robot 100 can be increased. When the user 10 continues to talk despite low battery power, the "anger" emotion value can be increased.

[0283] The action recognition unit 2234 recognizes the actions of user 10 based on the information parsed by the sensor module unit 2210 and the state of user 10 recognized by the state recognition unit 2230. For example, the information parsed by the sensor module unit 2210 and the state of user 10 recognized are input into a pre-learned neural network to obtain the probabilities of multiple predetermined action categories (e.g., "laughing", "angry", "asking a question", "feeling sad"), and the action category with the highest probability is recognized as the action of user 10.

[0284] As described above, in this embodiment, after identifying the user 10, the robot 100 obtains the content of the user 10's speech. However, when obtaining and using the speech content, in addition to obtaining the consent required by law from the user 10, the action control system of the robot 100 involved in this embodiment also considers the protection of the user 10's personal information and privacy.

[0285] Next, the processing of the action determination unit 2236 during the response processing of the robot 100 in response to the actions of the user 10 will be explained.

[0286] The action determination unit 2236 determines the action corresponding to the action of user 10 identified by the action recognition unit 2234 based on the current emotion value of user 10 determined by the emotion determination unit 2232, the history data 2222 of past emotion values ​​determined by the emotion determination unit 2232 before determining the current emotion value of user 10, and the emotion value of robot 100. In this embodiment, the action determination unit 2236 describes the case where the latest emotion value contained in the history data 2222 is used as the past emotion value of user 10, but the disclosed technology is not limited to this method. For example, the action determination unit 2236 may also use multiple latest emotion values ​​as the past emotion values ​​of user 10, or it may use emotion values ​​from a period of time such as one day ago as the past emotion values ​​of user 10. In addition, the action determination unit 2236 may also determine the action corresponding to the action of user 10 by considering not only the current emotion value of robot 100, but also the history of past emotion values ​​of robot 100. The actions determined by Action Determination Department 2236 include the gestures made by robot 100 or the content of robot 100's speech.

[0287] The action determination unit 2236 in this embodiment determines the action of the robot 100 based on the combination of the user 10's past and current emotional values, the robot 100's emotional value, the user 10's actions, and the action determination model 2221, as the action corresponding to the user 10's actions. For example, if the user 10's past emotional value is positive and its current emotional value is negative, the action determination unit 2236 determines an action to change the user 10's emotional value to positive, as the action corresponding to the user 10's actions.

[0288] In the reaction rules of the action determination model 2221, actions of the robot 100 are determined that correspond to the combination of the user 10's past and current emotional values, the robot 100's emotional value, and the user 10's actions. For example, if the user 10's past emotional value is positive and the current emotional value is negative, and the user 10's action is feeling sad, a combination of gestures and spoken content used to encourage the user 10's inquiry is determined as the robot 100's action.

[0289] For example, in the reaction rules of the action determination model 2221, the robot 100's action is determined based on the patterns of the robot 100's emotional values ​​(1296 patterns in total, representing the fourth power of six values ​​from "joy," "anger," "sorrow," and "happiness"—0 to "5"), the patterns of combinations of the user 10's past and current emotional values, and all combinations of the user 10's action patterns. That is, for each pattern of the robot 100's emotional values, and for each combination of the user 10's past and current emotional values, such as negative values ​​with negative values, negative values ​​with positive values, positive values ​​with negative values, positive values ​​with positive values, negative values ​​with normal values, and normal values ​​with normal values, the robot 100's action corresponding to the user 10's action pattern is determined. It should be noted that the action determination unit 2236 can also migrate to the action pattern used to determine the robot 100's action, for example, when the user 10 makes a statement attempting to continue the conversation from a past topic such as "I want to talk about that topic we discussed before."

[0290] It should be noted that, in the reaction rules of the action determination model 2221, for each of the 1296 patterns of the robot 100's emotion value, at most one item, namely, gesture or speech content, can be determined as the robot 100's action. Alternatively, in the reaction rules of the action determination model 2221, for each pattern group of the robot 100's emotion value pattern group, at least one of gesture or speech content can be determined as the robot 100's action.

[0291] Among the gestures included in the actions of robot 100 as determined by the reaction rules of action determination model 2221, the intensity of each gesture is predetermined. Similarly, among the speech content included in the actions of robot 100 as determined by the reaction rules of action determination model 2221, the intensity of each speech content is predetermined.

[0292] The storage control unit 2238 determines whether to store data including the user 10's actions in the history data 2222 based on the intensity of the actions predetermined for the actions determined by the action determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0293] Specifically, if the overall intensity value is above a threshold, it is determined that data containing the actions of user 10 will be stored in the resume data 2222. The overall intensity value is the sum of the following: the sum of the emotion values ​​of each of the multiple emotion categories for robot 100, the intensity of the gestures that include the actions determined by the action determination unit 2236, and the intensity of the speech content included in the actions determined by the action determination unit 2236.

[0294] When the storage control unit 2238 determines that data containing the actions of user 10 should be stored in the history data 2222, it stores the actions determined by the action determination unit 2236, the information parsed by the sensor module unit 2210 from the current time point to a certain period up to the present time (e.g., all surrounding information such as sound, image, smell, etc. at the scene), and the status of user 10 identified by the user status recognition unit 2230 (e.g., user 10's facial expressions, emotions, etc.) in the history data 2222.

[0295] The action control unit 2250 controls the controlled object 2252 based on the action determined by the action determination unit 2236. For example, if the action determination unit 2236 determines that the action includes speaking, the action control unit 2250 causes sound to be output from the speaker provided by the controlled object 2252. At this time, the action control unit 2250 can also determine the speed of sound production based on the emotion value of the robot 100. For example, the higher the emotion value of the robot 100, the faster the speed of sound production determined by the action control unit 2250. Thus, the action control unit 2250 determines the execution method of the action determined by the action determination unit 2236 based on the emotion value determined by the emotion determination unit 2232.

[0296] The action control unit 2250 can also recognize changes in the user 10's emotions in response to actions determined by the action determination unit 2236. For example, changes in emotions can be recognized based on the user 10's voice and facial expressions. Furthermore, changes in the user 10's emotions can be recognized based on the detection of an impact by the touch sensor 2205 of the sensor unit 2200. If an impact is detected by the touch sensor 2205 of the sensor unit 2200, the user 10's emotions are identified as worsening; conversely, if the detection result from the touch sensor 2205 of the sensor unit 2200 indicates that the user 10's reaction is laughter or joy, the user 10's emotions are identified as improving. Information representing the user 10's reaction is then output to the communication processing unit 2280.

[0297] Furthermore, after the Action Control Unit 2250 executes the action determined by the Action Determination Unit 2236 according to the execution method determined by the robot 100's emotion, the Emotion Determination Unit 2232 further changes the robot 100's emotion value based on the user's reaction to the execution of the action. Specifically, if the user's reaction to the Action Control Unit 2250 performing the action determined by the Action Determination Unit 2236 according to the determined execution method is not negative, the Emotion Determination Unit 2232 increases the robot 100's "joy" emotion value. Conversely, if the user's reaction to the Action Control Unit 2250 performing the action determined by the Action Determination Unit 2236 according to the determined execution method is negative, the Emotion Determination Unit 2232 increases the robot 100's "sorrow" emotion value.

[0298] Furthermore, the action control unit 2250 expresses the emotions of the robot 100 based on the determined emotion value of the robot 100. For example, when the action control unit 2250 increases the "joy" emotion value of the robot 100, it controls the controlled object 2252 to make the robot 100 perform joyful actions. Conversely, when the action control unit 2250 increases the "sorrow" emotion value of the robot 100, it controls the controlled object 2252 to make the robot 100 adopt a posture of lowering its head.

[0299] The communication processing unit 2280 is responsible for communication with the server 300. As described above, the communication processing unit 2280 sends user response information to the server 300. Additionally, the communication processing unit 2280 receives updated response rules from the server 300. Upon receiving updated response rules from the server 300, the communication processing unit 2280 updates the response rules used as the action determination model 2221.

[0300] Server 300 enables communication between robots 100, 101, and 102 and server 300, receives user response information sent from robot 100, and updates response rules based on response rules that include actions that receive positive responses.

[0301] At a specified time, the related information collection unit 2270 collects information related to the user's preferences from external data (websites such as news websites and animated image websites) based on the preferences information obtained for the user 10.

[0302] Specifically, the related information collection unit 2270 obtains preference information indicating matters of interest to user 10 based on user 10's speech content or user 10's settings. The related information collection unit 2270 periodically, for example using ChatGPT Plugins (web search...).<URL: https: / / openai.com / blog / chatgpt-plugins> The association information collection unit 2270 collects news related to user preferences from external data. For example, if it learns that user 10 is a fan of a specific professional baseball team, the association information collection unit 2270 will collect news related to the game results of that specific professional baseball team from external data at designated times every day, for example, using ChatGPTPlugins.

[0303] The emotion determination unit 2232 determines the emotion of the robot 100 based on information related to preference information collected by the association information collection unit 2270.

[0304] Specifically, the emotion determination unit 2232 inputs text representing information related to preference information collected by the association information collection unit 2270 into a pre-learned neural network used to determine emotions, obtains emotion values ​​representing each emotion, and thereby determines the emotion of the robot 100. For example, if news collected shows that a particular professional baseball team has won a game, the emotion value determined to be "happy" for the robot 100 will increase.

[0305] When the emotion value of robot 100 is above the threshold, storage control unit 2238 stores the information related to preference information collected by association information collection unit 2270 in collection data 2223.

[0306] Next, the processing of the action determination unit 2236 during the autonomous processing of the robot 100's autonomous actions will be explained.

[0307] At a predetermined time, the action determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the action determination model 2221, to determine any one of multiple types of robot actions, including inaction, as an action of the robot 100. Here, the case of using a dialogue-enabled article generation model as the action determination model 2221 will be explained as an example.

[0308] Specifically, the action determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text asking the robot to act, into the article generation model, and determines the robot 100's action based on the output of the article generation model.

[0309] For example, several types of robot actions include the following (1) to (10).

[0310] (1) The robot does nothing.

[0311] (2) The robot creates fiction.

[0312] (3) The robot strikes up a conversation with the user.

[0313] (4) Robots create picture diaries.

[0314] (5) Robot proposal activities.

[0315] (6) The robot suggests who the user should meet.

[0316] (7) The robot introduces news that users are interested in.

[0317] (8) Robots edit photos or animated images.

[0318] (9) The robot learns together with the user.

[0319] (10) The robot evokes memories.

[0320] Every certain period of time, the action determination unit 2236 inputs the state of user 10 and robot 100 identified by the state recognition unit 2230, text indicating the current sentiment value of user 10 and robot 100 determined by the sentiment determination unit 2232, and text asking which robot action to perform among multiple types including inaction. Based on the output of the article generation model, the action of robot 100 is determined. Here, if there is no user 10 around robot 100, the text input to the article generation model may not include the state of user 10 and the current sentiment value of user 10, or it may only include the statement that there is no user 10.

[0321] As an example, input the following into the article generation model: "The robot is in a very happy state. The user is in a generally happy state. The user is sleeping. As the robot's action, which of the following (1) to (10) is better?" (1) The robot does nothing.

[0322] (2) The robot creates fiction.

[0323] (3) The robot strikes up a conversation with the user.

[0324] The text is such that “…”. Based on the output of the article generation model, “It can be said that (1) doing nothing or (2) the robot making a fiction is the most appropriate action.”, “(1) doing nothing” or “(2) the robot making a fiction” is determined as the action of robot 100.

[0325] As another example, input the following into the article generation model: "The robot is in a somewhat lonely state. The user is not present. The robot's surrounding environment is dark. As the robot's action, which of the following options (1) to (10) is better?" (1) The robot does nothing.

[0326] (2) The robot creates fiction.

[0327] (3) The robot strikes up a conversation with the user.

[0328] The text is such that “…”. Based on the output of the article generation model, “It can be said that (2) the robot is making fiction or (4) the robot is making a picture diary is the most appropriate action.”, “(2) the robot is making fiction” or “(4) the robot is making a picture diary” is identified as the action of robot 100.

[0329] When the action determination unit 2236 determines that "(2) the robot is creating fiction," i.e., creating an original activity, is a robot action, it uses an article generation model to combine multiple event data from the resume data 2222 to create an original activity. At this time, the storage control unit 2238 stores the created original activity in the resume data 2222.

[0330] When the action determination unit 2236 determines that "(3) the robot speaks to the user," i.e., the robot 100 speaking, is a robot action, it uses a text generation model to determine the content of the robot's speech that corresponds to the user's state and the user's or robot's emotions. At this time, the action control unit 2250 outputs a sound representing the determined content of the robot's speech from the speaker provided by the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the action control unit 2250 stores the determined content of the robot's speech in the action pre-determined data 2224 and does not output a sound representing the determined content of the robot's speech.

[0331] When the action determination unit 2236 determines that "(7) The robot introduces news that the user is interested in" is a robot action, it uses an article generation model to determine the robot's speech content corresponding to the information stored in the collected data 2223. At this time, the action control unit 2250 outputs sound representing the determined robot speech content from the speaker provided by the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the action control unit 2250 stores the determined robot speech content in the action pre-planning data 2224 and does not output sound representing the determined robot speech content.

[0332] When the action determination unit 2236 determines "(4) Robot makes a picture diary.", i.e., the robot 100 makes an event image, as a robot action, it uses an image generation model to generate an image representing the event data selected from the history data 2222, and simultaneously uses an article generation model to generate explanatory text representing the event data. The combined image and explanatory text representing the event data are then output as the event image. It should be noted that if the user 10 is not near the robot 100, the action control unit 2250 stores the event image in the action pre-determined data 2224 and does not output the event image.

[0333] When the action determination unit 2236 determines "(8) Robot edits photo or motion image.", i.e., editing an image, as a robot action, it selects event data from the history data 2222 based on the emotion value, edits and outputs the image data of the selected event data. It should be noted that when the user 10 is not in the vicinity of the robot 100, the action control unit 2250 stores the edited image data in the action pre-set data 2224 and does not output the edited image data.

[0334] When the action determination unit 2236 determines that "(5) Robot Proposal Activity.", i.e., the action of the proposing user 10, is a robot action, it uses an article generation model based on the event data stored in the history data 2222 to determine the proposed user's action. At this time, the action control unit 2250 outputs the sound of the proposed user's action from the speaker provided by the controlled object 2252. It should be noted that when the user 10 is not in the vicinity of the robot 100, the action control unit 2250 stores the proposed user's action in the action pre-determined data 2224 and does not output the sound of the proposed user's action.

[0335] When the action determination unit 2236 determines "(6) The robot proposes that the user should meet with the object," i.e., the object that the user 10 should meet with, as a robot action, it uses an article generation model based on the event data stored in the history data 2222 to determine the proposed object that the user should meet with. At this time, the action control unit 2250 outputs a sound indicating the proposed object that the user should meet with from the speaker provided by the controlled object 2252. It should be noted that if the user 10 is not in the vicinity of the robot 100, the action control unit 2250 stores the proposed object that the user should meet with in the action planning data 2224, and does not output a sound indicating the proposed object that the user should meet with.

[0336] When the action determination unit 2236 determines "(9) The robot learns together with the user.", that is, the robot 100 speaking about learning, as a robot action, it uses a text generation model to determine the robot's speech content corresponding to the user's state and the user's or robot's emotions. This speech content is used to encourage learning, raise learning questions, or provide suggestions related to learning. At this time, the action control unit 2250 outputs the sound representing the determined robot speech content from the speaker provided by the controlled object 2252. It should be noted that when the user 10 is not around the robot 100, the action control unit 2250 stores the determined robot speech content in the action pre-determined data 2224 and does not output the sound representing the determined robot speech content.

[0337] When the action determination unit 2236 determines that "(10) Robot recalls memory," i.e., the recalled event data, is a robot action, it selects event data from the resume data 2222. At this time, the emotion determination unit 2232 determines the emotion of the robot 100 based on the selected event data. Furthermore, based on the selected event data, the action determination unit 2236 uses a text generation model to create an emotion change event that represents the speech content and actions of the robot 100 used to change the user's emotion value. At this time, the storage control unit 2238 stores the emotion change event in the action pre-determined data 2224.

[0338] For example, when storing panda-related dynamic images viewed by the user as event data in the history data 2222 and selecting this event data, the article generation model is inputted with "When you see the user again, what are some lines you can say in the context of panda-related topics? Please list three."; when the output of the article generation model is "(1) Let's go to the zoo, (2) Let's draw a panda, (3) Let's buy a panda plush toy", the robot 100 inputs with "Of (1), (2), (3), which one is most likely to make the user happy?"; when the output of the article generation model is "(1) Let's go to the zoo", an emotional change event is created in which the robot 100 says "(1) Let's go to the zoo" when the robot 100 sees the user again, and it is stored in the action pre-planning data 2224.

[0339] Additionally, for example, event data with higher emotional values ​​for robot 100 can be selected as robot 100's most memorable memories. Therefore, emotional change events can be created based on the event data selected as most memorable memories.

[0340] The action determination unit 2236 determines the action of the robot 100 by reading the data stored in the action predetermined data 2224 when the state of the user 10 identified by the state recognition unit 2230 changes from a state where the user 10 has no action against the robot 100 to a state where the user 10 has detected an action against the robot 100.

[0341] For example, if user 10 is not near robot 100, when user 10 is detected, the action determination unit 2236 reads the data stored in the action pre-set data 2224 to determine the action of robot 100. Conversely, if user 10 is sleeping, when user 10 is detected waking up, the action determination unit 2236 reads the data stored in the action pre-set data 2224 to determine the action of robot 100.

[0342] Figure 9B This section provides a summary of an example of the action flow for collecting and processing information related to user 10's preferences. Figure 9B The illustrated action flow is repeated at regular intervals. It is assumed that user 10's preferences, indicating matters of interest, have been obtained based on user 10's speech or settings. It should be noted that "S" in the action flow represents the step to be executed.

[0343] First, in step S90, the association information collection unit 2270 acquires preference information representing matters of interest to user 10.

[0344] In step S92, the association information collection unit 2270 collects information related to preference information from external data.

[0345] In step S94, the emotion determination unit 2232 determines the emotion value of the robot 100 based on the information associated with the preference information collected by the association information collection unit 2270.

[0346] In step S96, the storage control unit 2238 determines whether the emotion value of the robot 100 determined in step S94 is above a threshold. If the emotion value of the robot 100 is below the threshold, the process ends, and the information associated with the collected preference information is not stored in the collected data 2223. On the other hand, if the emotion value of the robot 100 is above the threshold, the process proceeds to step S998.

[0347] In step S98, the storage control unit 2238 stores the collected information associated with preference information in the collected data 2223 and ends the process.

[0348] Figure 3 This section provides a summary example of a motion flow related to determining the action within the robot 100 during response processing of the robot 100's action to the user 10. Repeated execution, as... Figure 3 The operation flow is shown. At this time, it is assumed that the information parsed by the sensor module 2210 has been input.

[0349] First, in step S100, the state recognition unit 2230 recognizes the state of the user 10 and the state of the robot 100 based on the information parsed by the sensor module unit 2210.

[0350] In step S102, the emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0351] In step S103, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 2222.

[0352] In step S104, the action recognition unit 2234 identifies the action category of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0353] In step S106, the action determination unit 2236 determines the action of the robot 100 based on the combination of the current sentiment value of the user 10 determined in step S102 and the past sentiment values ​​contained in the resume data 2222, the sentiment value of the robot 100, the action of the user 10 identified in step S104 above, and the action determination model 2221.

[0354] In step S108, the action control unit 2250 controls the controlled object 2252 based on the action determined by the action determination unit 2236.

[0355] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the action predetermined by the action determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0356] In step S112, the storage control unit 2238 determines whether the overall intensity value is above a threshold. If the overall intensity value is less than the threshold, the event data including the actions of user 10 is not stored in the history data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0357] In step S114, event data including the action determined by the action determination unit 2236, the information parsed by the sensor module unit 2210 from the current time point to a certain period ago, and the status of the user 10 identified by the status recognition unit 2230 are stored in the history data 2222.

[0358] Figure 9C This section provides a summary example of a motion flow related to determining the action within the robot 100 during autonomous processing of the robot 100's autonomous actions. Figure 9C The illustrated action flow is, for example, automatically executed repeatedly after a certain period of time. At this time, it is assumed that information parsed by the sensor module 2210 has been input. It should be noted that, for the above... Figure 3 The same process uses the same step numbers.

[0359] First, in step S100, the state recognition unit 2230 recognizes the state of the user 10 and the state of the robot 100 based on the information parsed by the sensor module unit 2210.

[0360] In step S102, the emotion determination unit 2232 determines an emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0361] In step S103, the emotion determination unit 2232 determines an emotion value representing the emotion of the robot 100 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the user state recognition unit 2230. The emotion determination unit 2232 adds the determined emotion value of the user 10 and the emotion value of the robot 100 to the history data 2222.

[0362] In step S104, the action recognition unit 2234 identifies the action category of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 identified by the state recognition unit 2230.

[0363] In step S200, the action determination unit 2236 determines any one of the multiple types of robot actions, including inaction, as an action of the robot 100 based on the state of the user 10 identified in step S100, the emotion of the user 10 identified in step S102, the emotion of the robot 100, the state of the robot 100 identified in step S100, the action of the user 10 identified in step S104, and the action determination model 2221.

[0364] In step S201, the action determination unit 2236 determines whether inaction was determined in step S200. If inaction is determined to be an action of the robot 100, the process ends. Otherwise, if inaction is not determined to be an action of the robot 100, the process proceeds to step S202.

[0365] In step S202, the action determination unit 2236 performs processing corresponding to the type of robot action determined in step S200 above. At this time, depending on the type of robot action, the action control unit 2250, the emotion determination unit 2232, or the storage control unit 2238 performs processing.

[0366] In step S110, the storage control unit 2238 calculates a comprehensive value of the intensity based on the intensity of the action predetermined by the action determination unit 2236 and the emotion value of the robot 100 determined by the emotion determination unit 2232.

[0367] In step S112, the storage control unit 2238 determines whether the overall intensity value is above or below a threshold. If the overall intensity value is below the threshold, data including the actions of user 10 is not stored in the history data 2222, and the process ends. On the other hand, if the overall intensity value is above the threshold, the process proceeds to step S114.

[0368] In step S114, the storage control unit 2238 stores the actions determined by the action determination unit 2236, the information parsed by the sensor module unit 2210 from the current time point to a certain period ago, and the status of the user 10 identified by the status recognition unit 2230 in the history data 2222.

[0369] As explained above, using robot 100, based on the user's state, an emotion value representing the robot 100's emotions is determined. Based on the robot 100's emotion value, it is determined whether to store data containing the user 10's actions in the history data 2222. This reduces the capacity of the history data 2222 containing data on the user 10's actions. Furthermore, for example, if robot 100 determines that the user's state 10 years from now is the same as 10 years ago, by reading the history data 2222 from 10 years ago, robot 100 can present the user 10 with all surrounding information, including the user 10's state 10 years ago (e.g., the user 10's facial expressions, emotions, etc.), and even data such as sounds, images, and smells from that time.

[0370] Furthermore, robot 100 can be used to perform actions appropriate to the actions of user 10. Previously, user actions were categorized, and actions including the robot's facial expressions and gestures were determined. In contrast, robot 100 determines user 10's current emotional state and performs actions based on past and current emotional states. Therefore, for example, if user 10 was in a good mood yesterday but is feeling down today, robot 100 can say something like, "You were quite energetic yesterday, what's wrong today?" Robot 100 can also combine gestures with speech. For example, if user 10 was feeling down yesterday but is feeling better today, robot 100 can say something like, "You were listless yesterday, but you look much more energetic today?" For example, if user 10 was in a good mood yesterday but is feeling even better today, robot 100 can say something like, "You're much more energetic today than yesterday. Did something even happier happen than yesterday?" Additionally, for example, if a user 10 continues to have an emotional value of 0 or higher and the fluctuation range of the emotional value is within a certain range, the robot 100 can say something like, "Your mood has been very stable lately, and you feel particularly good."

[0371] Furthermore, for example, if robot 100 asks user 10, "Have you finished the assignment you mentioned yesterday?" and receives a "Yes, I have!" from user 10, it can simultaneously express affirmation with words like "Wow, that's awesome!" and perform affirmative gestures such as clapping or giving a thumbs up. Similarly, if user 10 says, "The demonstration I mentioned the day before yesterday was successfully completed," robot 100 can also express affirmation with words like "Thank you for your hard work!" and perform the aforementioned affirmative gestures. In this way, by having robot 100 perform actions based on user 10's state of mind, it is hoped that user 10 will develop a sense of closeness towards robot 100.

[0372] Additionally, for example, if the "joy" emotion value of user 10 is above a threshold when user 10 watches a dynamic image related to pandas, then the scene of the panda appearing in the dynamic image can also be stored as event data in the history data 2222.

[0373] Using the data stored in the resume data 2222 and the collected data 2223, the robot 100 can always learn what kind of conversations to have with the user to maximize the emotional value that expresses the user's happiness.

[0374] In addition, even when the robot 100 is not in conversation with the user 10, it can autonomously begin to act based on the robot 100's emotions.

[0375] Furthermore, in autonomous processing, robot 100 repeatedly performs the following steps: automatically generating questions and inputting them into an article generation model, obtaining the output of the article generation model as an answer to the questions, thereby creating emotional change events that amplify positive emotions and storing them in the action pre-defined data 2224. In this way, robot 100 is able to perform autonomous learning.

[0376] In addition, when the robot 100 automatically generates questions without receiving external triggers, it can automatically generate questions based on memorable event data determined from the robot's past emotional value history.

[0377] In addition, the related information collection unit 2270 can perform autonomous learning by repeatedly performing the search execution phase. In this search execution phase, it can automatically perform keyword search corresponding to the user's preference information and obtain search results.

[0378] Here, during the retrieval execution phase, keyword retrieval can also be automatically performed based on memorable event data determined from the robot's past emotional value history, even without receiving any external triggers.

[0379] It should be noted that the emotion determination unit 2232 can determine the user's emotion based on a specific mapping. Specifically, the emotion determination unit 2232 can determine the user's emotion based on a specific mapping, namely an emotion map (see...). Figure 5 ), to determine the user's emotions.

[0380] Figure 5 This is a diagram illustrating an emotion map 400 that maps multiple emotions. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotion is. Further outward from the concentric circles, emotions representing states or actions derived from mood are arranged. Emotion refers to the concept that also includes feelings or mental states. On the left side of the concentric circles, emotions generated by reactions produced internally by the brain are arranged roughly. On the right side of the concentric circles, emotions guided by situational judgments are arranged roughly. Above and below the concentric circles, emotions generated by reactions produced internally by the brain and guided by situational judgments are arranged roughly. In addition, the emotion of "happiness" is arranged above the concentric circles, and the emotion of "unhappiness" is arranged below. Thus, in the emotion map 400, based on the structure that generates emotions, multiple emotions are mapped, and emotions that are likely to occur simultaneously are mapped closer together.

[0381] (1) For example, if the emotion engine of robot 100, which serves as the emotion determination unit 2232, detects emotions at a period of approximately 100 msec, the frequency for determining the reaction action (e.g., agreement) to robot 100 can be set to at least the same timing as the detection frequency (100 msec) of the emotion engine, or it can be set to a timing faster than the aforementioned timing. The detection frequency of the emotion engine can be interpreted as the sampling rate.

[0382] Emotions are detected at intervals of approximately 100 ms, and immediate responses (such as agreement) are initiated. This prevents unnatural agreement and enables natural, empathetic dialogue. Robot 100 responds (e.g., agrees) based on the directionality and intensity of the mandala-like pattern in the emotion map 400. It should be noted that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and can be adjusted according to the scenario (such as movement) and the user's age.

[0383] (2) It can be compared with the emotion map 400 to pre-set the directionality and intensity of the emotion, as well as the action of agreement and the strength of agreement. For example, when robot 100 feels secure and at ease, robot 100 will nod and continue to listen. When robot 100 feels uneasy, confused, or doubtful, robot 100 can tilt its head or stop shaking its head.

[0384] These emotions are distributed at the 3 o'clock position on the Emotion Chart 400, and typically fluctuate between peace and anxiety. In the right half of the Emotion Chart 400, situational awareness is more dominant than inner feelings, thus giving an impression of composure.

[0385] (3) When Robot 100 feels happy because of being praised, a filler word such as "Ah—" can be added before the dialogue. When it feels pain because of hearing harsh words, a filler word such as "Ugh!" can be added before the dialogue. In addition, Robot 100 can also add physical reactions such as squatting while saying "Ugh!". These emotions are distributed in the 9 o'clock position of the emotion diagram 400.

[0386] (4) In the left half of the emotion diagram 400, internal feelings (reactions) are more dominant than situation recognition. Therefore, it can give the impression of subconscious reactions.

[0387] When Robot 100 experiences an intrinsic feeling (reaction) of recognition and also senses liking in situational awareness, it can nod significantly while looking at the other party and make "uh-huh" sounds. In this way, Robot 100 can generate a balanced and moderate level of liking for the other party, that is, generate actions such as permission and tolerance. This emotion is distributed at the 12 o'clock position on the emotion diagram 400.

[0388] Conversely, if Robot 100 experiences both unpleasantness (internal reaction) and disgust in situation recognition, it will simply shake its head. If it experiences hatred, it can turn its LEDs red and stare at the other party. This emotion is distributed at the 6 o'clock position on the emotion diagram 400.

[0389] (5) The inner side of the emotion diagram 400 represents the inner state, and the outer side of the emotion diagram 400 represents the action. Therefore, the closer to the outer side of the emotion diagram 400, the easier it is to see the emotion (and the easier it is to be expressed in the action).

[0390] (6) When feeling at ease in the 3 o'clock position of the emotional map 400 and listening to others, the robot 100 nods gently and makes an "uh-huh" sound. However, if it becomes love in the 12 o'clock position, it can nod vigorously.

[0391] Here, human emotions are based on various balances such as posture and blood sugar levels. When these balances deviate from the ideal, an unhappy state is observed; when they approach the ideal, a happy state is observed. Similarly, in robots, cars, and motorcycles, emotions are constructed based on various balances such as posture and remaining battery power, so that when these balances deviate from the ideal, an unhappy state is observed; when they approach the ideal, a happy state is observed. Emotion maps can be generated based on Dr. Mitsuyoshi's emotion map (Research on the Brain Physiological Signal Analysis System for Voice Emotion Recognition and Emotion, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). In the left half of the emotion map, emotions belonging to the dominant sensory region, called "response," are arranged. Additionally, in the right half of the emotion map, emotions belonging to the dominant situational recognition region, called "situation," are arranged.

[0392] The emotion map defines two types of emotions that promote learning. One is the negative emotion surrounding the core of "repentance" and "reflection" on the situation side. That is, it occurs when the robot experiences negative emotions such as "I don't want to experience this feeling again" or "I don't want to be criticized again." The other is the positive emotion near "craving" on the response side. That is, it occurs when the robot experiences positive feelings such as "I crave more" or "I want to learn more."

[0393] The emotion determination unit 2232 inputs the information parsed by the sensor module unit 2210 and the identified state of user 10 into the pre-learned neural network to obtain the emotion values ​​representing each emotion shown in the emotion map 400 and determine the emotion of user 10. This neural network is pre-learned based on multiple learning data sets, which are combinations of the information parsed by the sensor module unit 2210, the identified state of user 10, and the emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network... Figure 6 As shown in the emotion map 900, emotions configured close to each other are learned in a similar manner. Figure 6 The text shows examples of multiple emotions such as "peace of mind", "stability", and "ease of mind" having similar emotional values.

[0394] Furthermore, the emotion determination unit 2232 can determine the emotion of the robot 100 based on a specific mapping. Specifically, the emotion determination unit 2232 inputs the information parsed by the sensor module unit 2210, the state of the user 10 identified by the user state recognition unit 2230, and the state of the robot 100 into a pre-learned neural network, and obtains the emotion values ​​representing each emotion shown in the emotion map 400 to determine the emotion of the robot 100. This neural network is pre-learned based on multiple learning data, which are combinations of the information parsed by the sensor module unit 2210, the identified state of the user 10, the state of the robot 100, and the emotion values ​​representing each emotion shown in the emotion map 400. For example, the neural network learns based on the following learning data: learning data where the emotion value representing "joy" is "3" when the robot 100 touches the user 10, as identified by the output of the touch sensor 2206; and learning data where the emotion value representing "anger" is "3" when the robot 100 pats the user 10, as identified by the output of the accelerometer (not shown). Additionally, this neural network... Figure 6 As shown in the emotion diagram 900, emotions that are arranged close to each other learn in a similar way.

[0395] The action determination unit 2236 adds fixed statements to the text representing the user's actions, the user's emotions, and the robot's emotions, to ask questions about the robot's actions corresponding to the user's actions, and generates the robot's action content by inputting it into a text generation model with dialogue functionality.

[0396] For example, the action determination unit 2236 uses the emotion table shown in Table 3 to obtain text representing the state of the robot 100 based on the emotion of the robot 100 determined by the emotion determination unit 2232. Here, in the emotion table, each emotion value is assigned an index number according to the type of emotion, and the text representing the state of the robot 100 is stored according to the index number.

[0397] When the emotion of robot 100 determined by emotion determination unit 2232 corresponds to index number "2", the text "is in a very happy state" is obtained. It should be noted that when the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the state of robot 100 are obtained.

[0398] Additionally, there is an emotional preparation table shown in Table 4 for User 10.

[0399] Here, if the user initiates a conversation with "Let's play together," and the robot 100's emotion is index number "2" and the user 10's emotion is index number "3," the text "The robot is in a very happy state. The user is in a generally happy state. The user has asked me, 'Let's play together?'. As the robot, how should I respond?" is input into the text generation model to obtain the robot's action content. The action determination unit 2236 determines the robot's action based on this action content.

[0400] Table 3

[0401] Table 4

[0402] Thus, the action determination unit 2236 determines the action content of the robot 100 in accordance with the following: a state related to each emotion of the robot 100, predetermined according to the intensity of each emotion, and the actions of the user 10. In this way, the robot 100's speech content during a dialogue with the user 10 can be branched based on the state related to the robot 100's emotion. That is, the robot 100 can change its actions according to the index number corresponding to the robot's emotion, thus giving the user the impression that the robot has autonomous consciousness, thereby encouraging actions such as striking up a conversation with the robot.

[0403] Furthermore, the action determination unit 2236 can add not only text representing the user's actions and emotions, as well as the robot's emotions, but also text representing the content of the resume data 2222. Based on this, it can add fixed phrases for asking questions about the robot's actions corresponding to the user's actions, and input these phrases into a dialogue-enabled text generation model to generate the robot's action content. Thus, the robot 100 can change its actions based on the resume data representing the user's emotions and actions, thereby giving the user the impression that the robot has a personality, which encourages actions such as engaging in conversation with the robot. Alternatively, the resume data can also include the robot's emotions and actions.

[0404] Additionally, the emotion determination unit 2232 can also determine the emotion of the robot 100 based on the action content of the robot 100 generated by the article generation model. Specifically, the emotion determination unit 232 inputs the action content of the robot 100 generated by the article generation model into a pre-learned neural network, obtains the emotion values ​​representing each emotion shown in the emotion graph 400, integrates the obtained emotion values ​​of each emotion with the emotion values ​​representing the current emotions of the robot 100, and updates the emotion of the robot 100. For example, the emotion values ​​representing each obtained emotion and the emotion values ​​representing the current emotions of the robot 100 are averaged and then integrated. This neural network is pre-learned based on multiple learning data, which are combinations of text representing the action content of the robot 100 generated by the article generation model and emotion values ​​representing each emotion shown in the emotion graph 400.

[0405] For example, if the robot 100's speech content "That's great! You're so lucky!" is received as the action content of the robot 100 generated by the article generation model, and if the text representing the speech content is input into the neural network, and a high value is obtained as the emotion "joy", the robot 100's emotion is updated to make the emotion "joy" even higher.

[0406] In Robot 100, the article generation model such as ChatGPT works in conjunction with the sentiment determination unit 2232. It has self-awareness and uses various parameters to continuously improve even when the user is not speaking.

[0407] ChatGPT is a large language model that utilizes deep learning methods. ChatGPT can also access external data; for example, within ChatGPTplugins, a technique is known to involve using dialogue to access various external data such as weather information and hotel booking information, and providing answers as accurately as possible. For instance, given a purpose in natural language, ChatGPT can automatically generate source code in various programming languages. Furthermore, given problematic source code, ChatGPT can debug to identify the issues and automatically generate improved source code. Combining these approaches with a purpose in natural language results in an autonomous agent that repeatedly performs code generation and debugging until the source code is error-free. Examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0408] In the robot 100 according to this embodiment, a technique as described in Patent Document 6 (Patent No. 6199927) can be used to retain event data in which the robot feels strong emotions for a long time, quickly forget event data that does not evoke obvious emotions in the robot, and retain event data to be learned in a database that stores strong memories.

[0409] Furthermore, robot 100 can record image data of user 10 acquired through its camera function in its resume data 2222. Robot 100 can retrieve image data from the resume data 2222 as needed and provide it to user 10. Robot 100 can generate more information-rich image data and record it in the resume data 2222 when the intensity of the emotion is greater. For example, when recording highly compressed information such as skeletal data, robot 100 can switch to recording low-compression information such as HD dynamic images if the emotional level exceeds a threshold. Through robot 100, for example, it is possible to retain high-definition image data as a record when robot 100 is emotionally aroused.

[0410] When robot 100 is not speaking to user 10, it automatically loads event data from its history data 2222, which stores memorable event data, and continuously updates its emotions using the emotion determination unit 2232. If robot 100's emotions change to a learning-enhancing emotion while it is not speaking to user 10, it can create an emotion change event based on the memorable event data to improve user 10's emotions. Thus, autonomous learning (recalling event data) corresponding to the robot 100's emotional state at appropriate times can be achieved, and autonomous learning that appropriately reflects the robot 100's emotional state can be realized.

[0411] The emotions that promote learning refer to the emotions near "repentance" and "reflection" on Dr. Mitsuyoshi's emotion map in a negative state, and the emotions near "craving" on the emotion map in a positive state.

[0412] Robot 100 can process "repentance" and "reflection" from the emotion map into learning-promoting emotions in a negative state. In addition to "repentance" and "reflection" from the emotion map, Robot 100 can also process emotions adjacent to "repentance" and "reflection" into learning-promoting emotions in a negative state. For example, besides "repentance" and "reflection," Robot 100 can process at least one of "regret," "stubbornness," "self-destruction," "self-discipline," "regret," and "despair" into learning-promoting emotions. Through these settings, for example, Robot 100 can perform autonomous learning while experiencing negative emotions such as "I don't want to experience this feeling again" or "I don't want to be criticized again."

[0413] Robot 100 can process the emotion "craving" in the emotion map as a learning-promoting emotion in a positive state. Robot 100 can also process emotions adjacent to "craving" as learning-promoting emotions in a positive state, in addition to "craving." For example, besides "craving," Robot 100 can process at least one of "happiness," "ecstasy," "desire," "anticipation," and "shyness" as learning-promoting emotions. With these settings, for example, Robot 100 can perform autonomous learning when experiencing positive emotions such as "greater desire" or "wanting to learn more."

[0414] When the robot 100 possesses emotions other than those that promote learning, as described above, the robot 100 may also refrain from performing autonomous learning. Thus, for example, it can avoid performing autonomous learning when it is extremely angry or blindly infatuated.

[0415] Emotional change events refer to actions preceding events that are particularly memorable. Actions preceding memorable events are those listed on the outermost emotional labels of an emotion map; for example, actions like "forgiveness" or "permission" precede "love."

[0416] In the autonomous learning process performed by Robot 100 without speaking to User 10, it combines the emotions, situations, and actions of characters and itself that appear in its most vivid memories, and uses an article generation model to create emotional change events.

[0417] Assuming all emotion values ​​are represented by a 6-stage rating system from 0 to 5, consider storing an event like "a friend was bullied and showed disgust" in resume data 2222 as a memorable event. Let's assume the friend here refers to user 10, whose emotion is "disgust," and a value of 5 is input to represent "disgust." Additionally, robot 100's emotion is "anxiety," and a value of 4 is input to represent "anxiety."

[0418] While robot 100 is not interacting with user 10, it continuously improves across various parameters through autonomous processing. Specifically, it loads event data from its resume data 2222, such as "a friend was bullied and showed disgust," as the highest-ranking event data sorted by emotional intensity. Among the loaded event data, "anxiety" (intensity 4) is associated with robot 100's emotion, and we assume "disgust" (intensity 5) is associated with the friend's (user 10's) emotion. If we assume that robot 100's current emotional value before loading was "reassurance" (intensity 3), after loading, influenced by "anxiety" (intensity 4) and "disgust" (intensity 5), robot 100's emotional value sometimes changes to "regret" (remorse). Since "regret" is a learning-promoting emotion, robot 100 identifies recalling the event data as a robot action, creating an emotional change event. At this point, the information input to the article generation model is the text representing the memorable event data, in this example, "a friend was bullied and showed disgust." Additionally, in the emotion map, the emotion of "disgust" exists at the innermost level, and the corresponding action, "aggression," is predicted at the outermost level. Therefore, in this case, an emotion change event will be created to prevent friends from "attacking" others.

[0419] For example, using information from memorable event data, the input text can be automatically generated when answering fill-in-the-blank questions, as described below.

[0420] "The user was bullied. At the time, the user felt strong resentment. The bot is very uneasy. Please provide a dialogue (within 30 words) that the bot should say to the user the next time they meet. However, please do not let the timing of the meeting affect your response. Also, please avoid direct statements. Provide 3 options."

[0421] <Expected Format> Candidate 1: (Words the robot should say to the user) Candidate 2: (Words the robot should say to the user) Candidate 3: (The words the robot should say to the user) At this point, the output of the article generation model is as follows.

[0422] Candidate 1: Are you alright? I've been thinking about what happened yesterday.

[0423] Candidate 2: I'm still thinking about what happened yesterday. What should I do? Candidate 3: I'm a little worried about you. Would you like to talk to me about it? Furthermore, regarding the information obtained through the creation of emotional change events, Robot 100 can also automatically generate the input text described below.

[0424] "In the case of 'a friend being bullied,' what kind of feelings might a user have if the following words are said to them? The user's emotions are represented by the format 'joy A, anger B, sorrow C, happiness D,' and we assume that A to D are six rating integers from 0 to 5."

[0425] Candidate 1: Are you alright? I've been thinking about what happened yesterday.

[0426] Candidate 2: I'm still thinking about what happened yesterday. What should I do? Candidate 3: I'm a little worried about you. Would you like to talk to me about it? At this point, the output of the article generation model is as follows.

[0427] "Users' emotions may be as follows."

[0428] Candidate 1: Joy is 3, Anger is 1, Sorrow is 2, Happiness is 2 Candidate 2: Joy is 2, Anger is 1, Sorrow is 3, Happiness is 2 Candidate 3: Joy is 2, Anger is 1, Sorrow is 3, Happiness is 3 In this way, Robot 100 can also perform processing around memories after creating an emotional change event.

[0429] Finally, robot 100 can select the candidate most likely to evoke joy from multiple candidates, create an emotional change event, store it in action reservation data 2224, and prepare for the next meeting with user 10.

[0430] As described above, even without conversations with family or friends, the robot continuously determines its emotional value using the information in its resume data 2222, which stores impressive event data. When the emotion changes to the aforementioned learning-enhancing emotion, the robot 100 performs autonomous learning based on its own emotion, without conversations with the user 10, and continuously updates its resume data 2222 and action plan data 2224.

[0431] The above is an example of using sentiment values, but sentiment can be generated in the sentiment map based on hormone secretion and event type. Therefore, as a value associated with memorable event data, it can also be the type of hormone, the amount of hormone secretion, or the type of event.

[0432] The following describes specific implementation examples.

[0433] Robot 100, for example, can query information related to topics and interests that the user is interested in, even without speaking to the user.

[0434] Robot 100, for example, can query information related to a user's birthday or anniversary and think of messages of blessing, even without speaking to the user.

[0435] Robot 100, for example, can query reviews of places a user wants to go, foods they want to eat, and products they want to buy, even without speaking to the user.

[0436] Robot 100, for example, can check weather information and provide suggestions that match the user's schedule and plans, even without speaking to the user.

[0437] Robot 100, for example, can query information about local events and festivals and make suggestions to users even without speaking to them.

[0438] Robot100, for example, can query the results of sports matches and news that users are interested in, and provide topics, even without speaking to the user.

[0439] Robot 100, for example, can query and introduce information about the user's favorite music and artists even without speaking to the user.

[0440] Robot100, for example, can query social issues and related news that users are concerned about, and provide opinions, even without speaking to the user.

[0441] Robot 100, for example, can query information related to the user's hometown or birthplace and provide topics of conversation even without speaking to the user.

[0442] Robot 100, for example, can query information about a user's job and school and provide suggestions even without speaking to the user.

[0443] Robot 100, for example, can query and introduce information about books, comics, movies, and TV series that users are interested in, even without speaking to the user.

[0444] Robot 100, for example, can query information related to the user's health and provide suggestions even without speaking to the user.

[0445] Robot 100, for example, can query information related to a user's travel plans and provide suggestions even without speaking to the user.

[0446] Robot 100, for example, can query information related to the maintenance and repair of a user's house or vehicle and provide suggestions even without speaking to the user.

[0447] Robot 100, for example, can query information about beauty and fashion that users are interested in and provide suggestions even without speaking to the user.

[0448] Robot 100, for example, can query a user's pet information and provide suggestions even without speaking to the user.

[0449] Robot 100, for example, can query and suggest information on competitions and activities related to the user's interests and work, even without speaking to the user.

[0450] Robot 100, for example, can query and suggest information about restaurants and eateries that users like, even without speaking to the user.

[0451] Robot 100, for example, can collect information and provide advice on important life-related decisions for users even without speaking to them.

[0452] Robot 100, for example, can query human-related information that users are concerned about and provide suggestions even without speaking to the user.

[0453] [Third Implementation Method] In the third embodiment, the robot 100 is mounted on a plush toy, or the robot 100 is applied to a control device that is wirelessly or wiredly connected to a control object device (speaker, camera) mounted on the plush toy. It should be noted that parts with the same structure as those in the second embodiment are labeled with the same reference numerals and their descriptions are omitted.

[0454] The third embodiment is specifically configured as follows. For example, robot 100 is applied to such cohabiting individuals (specifically, Figure 7 and Figure 8 (The illustrated plush toy 100N): While interacting with the user 10 in their daily life, it promotes dialogue with the user 10 based on information related to daily life, or provides information matching the user 10's interests and preferences. In the third embodiment, an example of applying the control part of the robot 100 described above to a smartphone 50 will be described.

[0455] The plush toy 100N, which is equipped with the input / output device of the robot 100, can be detachably configured with a smartphone 50 that functions as the control part of the robot 100. The input / output device and the smartphone 50 are connected inside the plush toy 100N.

[0456] like Figure 7 As shown in (A), in this embodiment (other embodiments), the plush toy 100N is a bear-shaped object covered with soft fabric. The internal space 52 serves as an input / output device, housing a sensor unit 2200A and a control object 2252A (see [reference]). Figure 9D The sensor unit 2200A includes a microphone 2201 and a 2D camera 2203. Specifically, as... Figure 7As shown in (B), in the space section 52, a microphone 2201 of the sensor section 2200 is arranged in the part corresponding to the ear 54, a 2D camera 2203 of the sensor section 2200 is arranged in the part corresponding to the eye 56, and a speaker 60, which constitutes part of the control object 2252A, is arranged in the part corresponding to the mouth 58. It should be noted that the microphone 2201 and the speaker 60 are not necessarily separate units, and can also be integrated units. In the case of an integrated unit, it can be arranged in a position where speech is naturally emitted, such as the nose of the plush toy 100N. It should be noted that the example given is the case where the plush toy 100N is in the shape of an animal, but it is not limited to this. The plush toy 100N can also be in the shape of a specific character.

[0457] Figure 9D The functional structure of the plush toy 100N is shown in summary. The plush toy 100N includes a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228, and a control object 2252A.

[0458] The smartphone 50 housed in the plush toy 100N of this embodiment performs the same processing as the robot 100 of the second embodiment. That is, the smartphone 50 has... Figure 9D The functions shown are as follows: as a sensor module 2210, as a storage unit 2220, and as a control unit 2228.

[0459] like Figure 8 As shown, the toy 100N is configured such that a zipper 62 is installed on a part (e.g., the back) of the plush toy 100N, and by opening the zipper 62, the outside is connected to the space portion 52.

[0460] Here, the smartphone 50 is housed externally in the space 52 via the USB hub 64 (see Figure 7 (B) can be connected to each input / output device via USB, thereby enabling it to have the same functions as the robot 100 of the second embodiment described above.

[0461] Additionally, the contactless power receiver 66 is connected to the USB hub 64. A power receiving coil 66A is mounted on the power receiver 66. The power receiver 66 is an example of a wireless power receiver that receives wireless power.

[0462] The receiving plate 66 is positioned near the base 68 of the two legs of the plush toy 100N, and is the closest point to the mounting base 70 when the plush toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0463] The plush toy 100N placed on the base 70 can be displayed as a decorative item in its natural state.

[0464] In addition, the root is formed in a way that is thinner than the surface thickness of the other parts of the plush toy 100N, and is held in a state that is closer to the mounting base 70.

[0465] The mounting base 70 includes a charging pad 72. The charging pad 72 is equipped with a power transmission coil 72A. The power transmission coil 72A sends a signal to locate the power receiving coil 66A on the power receiving board 66. When the power receiving coil 66A is found, current flows through the power transmission coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. Thus, current flows through the power receiving coil 66A, and via the USB hub 64, stores power in the battery (not shown) of the smartphone 50.

[0466] That is, by placing the plush toy 100N as an ornament on the base 70, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the plush toy 100N for charging.

[0467] It should be noted that in the third embodiment, the smartphone 50 is housed in the space 52 of the plush toy 100N and a wired connection (USB connection) is established, but this is not a limitation. For example, a control device with wireless functionality (e.g., "Bluetooth" (registered trademark)) can also be housed in the space 52 of the plush toy 100N and connected to a USB hub 64. In this case, without placing the smartphone 50 in the space 52, the smartphone 50 communicates wirelessly with the control device, and the external smartphone 50 connects to various input / output devices through the control device, thereby enabling the same functionality as the robot 100 of the second embodiment described above. Alternatively, the control device housed in the space 52 of the plush toy 100N can also be connected to the external smartphone 50 via a wired connection.

[0468] Furthermore, in the third embodiment, a bear plush toy 100N is used as an example, but it can be in the shape of other animals, dolls, or specific characters. Additionally, the clothes can be changed. Furthermore, the material of the outer skin is not limited to cloth; it can be made of other materials such as soft plastic, but a soft material is preferred.

[0469] Furthermore, a display can be installed on the surface of the plush toy 100N, adding a control object 2252 that provides visual information to the user 10. For example, the eyes 56 can be used as a display, expressing emotions through images projected onto the eyes; a window can also be provided on the abdomen for the built-in smartphone 50's display to pass through. Additionally, the eyes 56 can be used as a projector, expressing emotions through images projected onto a wall.

[0470] According to the third embodiment, an existing smartphone 50 is placed inside the plush toy 100N, and then, via USB connection, the camera 2203, microphone 2201, speaker 60, etc. are extended to appropriate positions.

[0471] Furthermore, for wireless charging, the smartphone 50 is connected to the power receiving board 66 via USB, and the power receiving board 66 is configured to be viewed from the inside of the plush toy 100N as far outward as possible.

[0472] To use the wireless charging of the smartphone 50, the smartphone 50 must be configured to look as far outward as possible from the inside of the plush toy 100N. When touching the plush toy 100N from the outside, an uneven feel will be produced.

[0473] Therefore, the smartphone 50 is positioned as centrally as possible within the plush toy 100N, and the wireless charging function (power receiving board 66) is positioned as far outward as possible when viewed from the inside of the plush toy 100N. The camera 2203, microphone 2201, speaker 60, and smartphone 50 receive wireless power via the power receiving board 66.

[0474] It should be noted that the other structures and functions of the plush toy 100N in the third embodiment are the same as those of the robot 100 in the second embodiment, so the description is omitted.

[0475] [Fourth Implementation Method] In the second embodiment described above, an example is shown where the motion control system is applied to the robot 100. However, in the fourth embodiment, the robot 100 is used as an intelligent agent for dialogue with a user, and the motion control system is applied to an intelligent agent system. It should be noted that parts with the same structure as those in the second and third embodiments are labeled with the same reference numerals and their descriptions are omitted.

[0476] Figure 9E It is a function block diagram of an intelligent agent system 2500 that utilizes part or all of the functions of the action control system.

[0477] The intelligent agent system 2500 is a computer system that performs a series of actions in accordance with the intentions of user 10 through dialogue with user 10. The dialogue with user 10 can be conducted through voice or text.

[0478] The intelligent agent system 2500 includes a sensor unit 2200A, a sensor module unit 2210, a storage unit 2220, a control unit 2228B, and a controlled object 2252B.

[0479] The intelligent agent system 2500 can be integrated into robots, humanoid figures, plush toys, wearable devices (keychains, smartwatches, smart glasses), smartphones, smart speakers, headphones, and personal computers. Alternatively, the intelligent agent system 2500 can be implemented in a web server and utilized through a web browser operating on a user's smartphone or other communication terminal.

[0480] The intelligent agent system 2500 can act as a butler, secretary, teacher, partner, friend, lover, or mentor to provide services to user 10. The intelligent agent system 2500 not only converses with user 10 but also provides suggestions, navigation to destinations, or recommendations based on user preferences. Furthermore, the intelligent agent system 2500 can make appointments, place orders, or make payments to service providers.

[0481] Similar to the second embodiment described above, the emotion determination unit 2232 determines the emotions of the user 10 and the emotions of the intelligent agent itself. The action determination unit 2236 determines the actions of the robot 100 while considering the emotions of both the user 10 and the intelligent agent. That is, the intelligent agent system 2500 understands the user 10's emotions, observes their expressions, and provides heartfelt support, assistance, suggestions, and services. Furthermore, the intelligent agent system 2500 explores the user 10's troubles, comforts the user, encourages the user, and cheers the user on. Additionally, the intelligent agent system 2500 plays with the user 10, creates a picture diary, and reminisces about the past. The intelligent agent system 2500 performs actions that increase the user 10's sense of well-being.

[0482] The control unit 2228B includes a state recognition unit 2230, an emotion determination unit 2232, an action recognition unit 2234, an action determination unit 2236, a storage control unit 2238, an action control unit 2250, a related information collection unit 2270, an instruction acquisition unit 2272, a robotic process automation (RPA) unit 2274, a character modeling unit 2276, and a communication processing unit 2280.

[0483] Similar to the second embodiment described above, the action determination unit 2236 determines the speech content of the agent used to converse with the user 10, and uses this as the agent's action. The action control unit 2250 outputs the agent's speech content via at least one of sound and text using a speaker and a display, which are controlled objects 2252B.

[0484] The character designation unit 2276 sets the character design of the intelligent agent system 2500 when it converses with the user 10, based on the user 10's specifications. That is, the speech output from the action determination unit 2236 is output by an intelligent agent with a pre-set character design. For example, a real-life famous person or celebrity such as an actor, entertainer, idol, or athlete can be set as the character design. Alternatively, a fictional character from a comic book, movie, or animation can be set. For example, Princess Anne, played by Audrey Hepburn in the movie "Roman Holiday," can be set as the intelligent agent's character design. Since the intelligent agent's character design is known, its voice, words, tone, and personality are also known. Therefore, the user 10 only needs to specify their preferred character design, and the character designation unit 2276 automatically sets the prompts. The pre-set character design's voice, words, tone, and personality are reflected in the conversation with the user 10. That is, the action control unit 2250 synthesizes a voice corresponding to the character design set by the character design unit 2276, and outputs the speech content of the intelligent agent through the synthesized voice. As a result, the user 10 can have the feeling of talking to the person of their favorite character design (such as their favorite actor).

[0485] When the agent system 2500 is mounted on a device with a display, such as a smartphone, icons, still images, or videos of agents with character designs set by the character design unit 2276 can also be displayed on the display. The agent's image is generated, for example, using image compositing techniques such as 3D rendering. In the agent system 2500, the agent's image can also make gestures corresponding to the user 10's emotions, the agent's emotions, and the content of the agent's speech, while simultaneously engaging in dialogue with the user 10. It should be noted that the agent system 2500 can also output only sound without outputting images when conversing with the user 10.

[0486] Similar to the second embodiment, the emotion determination unit 2232 determines the emotion value representing the user 10's emotion and the agent's own emotion value. In this embodiment, the agent's emotion value is determined instead of the robot 100's emotion value. The agent's own emotion value is reflected in the emotion of the pre-set character model. When the agent system 2500 converses with the user 10, not only is the user 10's emotion reflected in the conversation, but the agent's emotion is also reflected in the conversation. That is, the action control unit 2250 outputs the speech content in a manner corresponding to the emotion determined by the emotion determination unit 2232.

[0487] Furthermore, the intelligent agent system 2500 can reflect the agent's emotions even when acting towards user 10. For example, if user 10 requests the intelligent agent system 2500 to take a photo, whether the intelligent agent system 2500 takes the photo as requested depends on the degree of "sadness" the agent possesses. When the agent has positive emotions, it engages in friendly dialogue or actions with user 10; when it has negative emotions, it engages in resistant dialogue or actions with user 10.

[0488] The history data 2222 stores the history of conversations between user 10 and the intelligent agent system 2500 as event data. The storage unit 2220 can also be implemented through external cloud storage. When conversing with user 10 or taking actions towards user 10, the intelligent agent system 2500 considers the content of the conversation history stored in the history data 2222 to determine the conversation content or action content. For example, based on the conversation history stored in the history data 2222, the intelligent agent system 2500 learns user 10's interests and hobbies. The intelligent agent system 2500 generates conversation content matching user 10's interests and hobbies, or provides recommendations. The action determination unit 2236 determines the intelligent agent's speech content based on the conversation history stored in the history data 2222. The history data 2222 stores personal information obtained through conversations with user 10, such as user 10's name, address, phone number, and credit card number. Here, the intelligent agent may also proactively ask user 10 whether to save personal information, such as "Do you want to save your credit card number?", and based on user 10's answer, store the personal information in resume data 2222.

[0489] As described in the second embodiment above, the action determination unit 236 generates speech content based on an article generated using the article generation model. Specifically, the action determination unit 2236 inputs text or voice input by user 10, the emotions of user 10 and character design determined by the emotion determination unit 2232, and the conversation history stored in the history data 2222 into the article generation model to generate the agent's speech content. At this time, the action determination unit 2236 may also input the personality of the character design set by the character design unit 2276 into the article generation model to generate the agent's speech content. In the agent system 2500, the article generation model is not located at the front end, which is the point of contact with user 10, but is only used as a tool of the agent system 2500.

[0490] The instruction acquisition unit 2272 uses the output of the speech understanding unit 2212 to acquire instructions from the voice or text emitted by the user 10 through dialogue with the user 10. Instructions may include, for example, actions to be performed by the intelligent agent system 2500, such as information retrieval, store reservation, ticketing arrangements, purchasing goods / services, making payments, providing route navigation to a destination, and offering recommendations.

[0491] RPA 2274 performs actions corresponding to the instructions acquired by instruction acquisition unit 2272. RPA 2274 performs actions related to using service providers, such as information retrieval, store reservation, ticketing, purchase of goods / services, and payment.

[0492] RPA 2274 reads and utilizes the personal information of user 10 required to perform actions related to using the service provider from the resume data 2222. For example, when purchasing goods based on a request from user 10, the agent system 2500 reads and uses the personal information of user 10 stored in the resume data 2222, such as name, address, phone number, and credit card number. Requesting personal information from user 10 during initial setup is inconvenient and uncomfortable for the user. In the agent system 2500 of this embodiment, personal information is not requested from user 10 during initial setup; instead, personal information obtained through conversations with user 10 is stored and read and used as needed. This avoids creating uncomfortable memories for the user and improves user convenience.

[0493] The intelligent agent system 2500 performs dialogue processing, for example, through the following steps 1 to 5.

[0494] (Step 1) The agent system 2500 sets the agent's character appearance. Specifically, the character appearance setting unit 2276 sets the character appearance of the agent when the agent system 2500 is in dialogue with the user 10, based on the specification from the user 10.

[0495] (Step 2) The intelligent agent system 2500 acquires the user 10's state, the user 10's emotion value, the intelligent agent's emotion value, and history data 2222, including ...

[0496] (Step 3) The agent system 2500 determines the content of the agent's speech.

[0497] Specifically, the action determination unit 2236 inputs the text or voice input by user 10, the emotions of user 10 and the character model determined by the emotion determination unit 2232, and the history of the conversation stored in the history data 2222 to the article generation model to generate the speech content of the intelligent agent.

[0498] For example, in the text representing the text or voice input by user 10, the emotions of user 10 and the character model determined by the emotion determination unit 2232, and the history of the conversation stored in the history data 2222, a fixed statement such as "At this time, as an agent, how should I reply?" is added and input into the article generation model to obtain the agent's speech content.

[0499] As an example, if user 10 inputs the text or voice message "I'd like to make a reservation for a good Chinese restaurant near 7 pm tonight," the AI's speech would be: "Yes, ma'am." and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0500] Additionally, if user 10 inputs the text or voice message "Select the 4th DDDD", the agent's speech content will be "Yes, ma'am. I'll try to make a reservation for you. How many seats would you like to reserve?".

[0501] (Step 4) The agent system 2500 outputs the agent's speech content.

[0502] Specifically, the action control unit 2250 synthesizes a voice corresponding to the character design set by the character design unit 2276, and outputs the speech content of the intelligent agent using the synthesized voice.

[0503] (Step 5) The intelligent agent system 2500 determines whether it is the time to execute the intelligent agent's instructions.

[0504] Specifically, the action determination unit 2236 determines whether it is the time to execute the agent's instruction based on the output of the article generation model. For example, if the output of the article generation model contains the intention for the agent to execute an instruction, it is determined that it is the time to execute the agent's instruction, and the process proceeds to step 6. On the other hand, if it is determined that it is not the time to execute the agent's instruction, the process returns to step 2.

[0505] (Step 6) The intelligent agent system 2500 executes the intelligent agent's instructions.

[0506] Specifically, the instruction acquisition unit 2272 acquires the agent's instructions from the voice or text emitted by the user 10 through dialogue with the user 10. Then, the RPA 2274 performs actions corresponding to the instructions acquired by the instruction acquisition unit 2272. For example, if the instruction is "information retrieval," the information retrieval is performed through a search website using the search query obtained through dialogue with the user 10 and the application programming interface (API). The action determination unit 2236 inputs the search results into the article generation model to generate the agent's speech content. The action control unit 2250 synthesizes a voice corresponding to the character design set by the character design unit 2276 and outputs the agent's speech content using the synthesized voice.

[0507] Additionally, when the instruction is "store reservation," the system uses the reservation information obtained through dialogue with user 10, the store information to be reserved, and the API to call the store via telephone software to make the reservation. At this time, the action determination unit 2236 uses a dialogue-enabled text generation model to obtain the agent's speech content in response to the input voice. Then, the action determination unit 2236 inputs the store reservation result (reservation success or failure) into the text generation model to generate the agent's speech content. The action control unit 2250 synthesizes a voice corresponding to the character design set by the character design unit 2276 and outputs the agent's speech content using the synthesized voice.

[0508] Then, return to step 2 above.

[0509] In step 6, the results of actions performed by the agent (e.g., store reservations) are also stored in the resume data 2222. The results of actions performed by the agent stored in the resume data 2222 are flexibly applied by the agent system 2500 to understand the interests or preferences of user 10. For example, if the same store is booked multiple times, it is identified that user 10 likes that store, or the reservation time slot, package details, or price are used as a basis for selecting a store for the next reservation.

[0510] In this way, the intelligent agent system 2500 can perform dialogue processing and take actions related to the service provider's use as needed.

[0511] Figure 9F and Figure 9G This is a diagram illustrating an example of the actions of an intelligent agent system 2500. Figure 9F The example illustrates how an intelligent agent system 2500 makes a hotel reservation through a dialogue with user 10. Figure 9F In the image, the left side shows the speech content of the intelligent agent, and the right side shows the speech content of user 10. The intelligent agent system 2500 can understand user 10's preferences based on their conversation history, provide a list of recommended restaurants matching user 10's preferences, and make reservations for the selected restaurants.

[0512] On the other hand, Figure 9G The example illustrates how an intelligent agent system 2500 accesses an e-commerce website and makes a purchase through a dialogue with user 10. Figure 9G In the diagram, the left side shows the speech content of the intelligent agent, and the right side shows the speech content of user 10. Based on its conversation history with user 10, the intelligent agent system 2500 can estimate the remaining amount of beverages in user 10's inventory, suggest purchasing the beverage, and execute the purchase. Furthermore, based on its past conversation history with user 10, the intelligent agent system 2500 understands user preferences and recommends preferred fast food. Thus, the intelligent agent system 2500 acts as a butler-like agent, communicating with user 10 and even performing various actions such as restaurant reservations and purchasing goods, thereby supporting user 10's daily life.

[0513] It should be noted that the other structures and functions of the intelligent agent system 2500 in the fourth embodiment are the same as those of the robot 100 in the second embodiment, so the description is omitted.

[0514] It should be noted that, in the above embodiments, the use of the user 10's facial image to identify the user 10 has been described, but the disclosed technology is not limited to this method. For example, the robot 100 may also identify the user 10 using the user 10's voice, the user 10's email address, the user 10's SNS ID, or an ID card with a built-in wireless IC tag held by the user 10.

[0515] Robot 100 is an example of an electronic device equipped with a motion control system. The application of the motion control system is not limited to robot 100; it can be applied to various electronic devices. Furthermore, the functions of server 300 can be implemented using more than one computer. At least a portion of the functions of server 300 can be implemented using a virtual machine. Additionally, at least a portion of the functions of server 300 can be implemented via the cloud.

[0516] [Fifth Implementation] The fifth embodiment is an example configured to apply the response processing and autonomous processing of the action control system of the second embodiment, and the intelligent agent function of the fourth embodiment, to the plush toy of the third embodiment. Hereinafter, the same reference numerals will be used for parts that have the same structure as those in the first to fourth embodiments, and descriptions will be omitted.

[0517] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0518] The emotion determination unit 2232 determines the emotion of user 10 based on a learning completion model, which is learned based on at least one of the following: multiple learning data as a combination of user 10’s voiceprint and emotions felt by multiple other users from the voiceprint, and multiple learning data as a combination of user 10’s gestures and emotions felt by multiple other users from the gestures.

[0519] Specifically, user 10's conversation is divided into voiceprint and gesture. Furthermore, multiple other users are asked to select a combination of "this is the emotion when the voiceprint is like this," and the selection results are received. Similarly, multiple other users are asked to select a combination of "this is the emotion when the gesture is like this," and the selection results are received. The emotion determination unit 2232 inputs this combination into a neural network to determine user 10's emotion. This neural network is a learned model obtained by using the aforementioned combinations as learning data. With this structure, the conversation can be associated with the emotion map 400. It should be noted that the learning data is not limited to either the combination of voiceprint and emotion or the combination of gesture and emotion; either can be used.

[0520] It should be noted that the processing described in the fifth embodiment can be executed in the response processing and autonomous processing of the action control system in the second embodiment, or the processing described in the fifth embodiment can be executed in the agent function of the fourth embodiment.

[0521] Figure 4 This diagram illustrates an example of the hardware structure of a smartphone 50, a robot 100, a server 300, and a computer 1200 that functions as an intelligent agent system 2500.

[0522] [Sixth Implementation Method] The sixth embodiment is an example configured to apply the response processing and autonomous processing of the action control system of the second embodiment, as well as the intelligent agent function of the fourth embodiment, to the plush toy of the third embodiment. Hereinafter, the same reference numerals will be used for parts with the same structure as those in the first to fifth embodiments, and descriptions will be omitted.

[0523] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0524] Even when the question is ambiguous, Robot 100 automatically transforms it into a more accurate question based on the needs identified from words and expressions, or re-listens to the question. After clarifying the true question, it presents a solution, thus going beyond simply "answering the question" to achieve a caring "dialogue." Robot 100 generates responses appropriate to its location by inputting information about the surrounding area, confirms with the questioner whether the problem has been resolved, and provides feedback on the correctness of the user's questions and answers, thereby continuously improving the resolution rate.

[0525] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing through steps 1 to 5-2: when recognizing the user's preferences, the user's situation, and the user's reaction, if the content of the question raised by the user 10 is unclear, it automatically transforms the unclear question into a correct question based on the needs analysis found from the words used and the user 10's expression, or listens to the question raised by the user 10 again, and presents a solution after clarifying the true question.

[0526] (Step 1) Robot 100 obtains the status of user 10, the emotional value of user 10, the emotional value of robot 100, and the resume data 2222. Specifically, it performs the same processing as steps S100~S103 above to obtain the status of user 10, the emotional value of user 10, the emotional value of robot 100, and the resume data 2222.

[0527] (Step 2) Robot 100 obtains, for example, the preferences of user 10 related to parameters such as air conditioning temperature.

[0528] Specifically, the action determination unit 2236 determines the robot 100's action as asking the user 10 about their preferences related to parameters such as temperature. The action control unit 2250 controls the controlled object 2252 to ask the user 10 about their preferences related to parameters such as temperature. The user state recognition unit 2230 identifies the user 10's preferences related to parameters such as temperature based on information parsed by the sensor module unit 2210 (e.g., the user's answer).

[0529] At this point, if the content of the question posed by User 10 is unclear, the Action Determination Unit 2236 automatically transforms the unclear question into a correct one based on the needs analysis obtained from the words used and User 10's facial expressions, or re-listens to User 10's question to clarify the true question. Then, in subsequent steps, a solution, such as parameters like temperature, is presented.

[0530] (Step 3) Robot 100 determines the parameters proposed to user 10.

[0531] Specifically, the action determination unit 2236 adds a fixed statement, "What temperature or other parameters should be recommended to the user at this time?" to the text representing user 10's preferences related to parameters such as temperature, user 10's emotions, robot 100's emotions, and content stored in the resume data 2222, and inputs this into the article generation model to obtain recommended content related to temperature and other parameters. At this point, it not only suggests user 10's preferences related to temperature and other parameters, but also suggests parameters suitable for user 10 by considering user 10's emotions and resume data 2222. Furthermore, by considering robot 100's emotions, it makes user 10 feel that robot 100 possesses emotions.

[0532] (Step 4) Robot 100 proposes the parameters determined in step 3 to user 10 and obtains user 10's response.

[0533] Specifically, the action determination unit 2236 determines the speech proposing parameters to the user 10 as the action of the robot 100, and the action control unit 2250 controls the controlled object 2252 to perform the speech proposing parameters to the user 10. The user state recognition unit 2230 recognizes the state of the user 10 based on the information parsed by the sensor module unit 2210, and the emotion determination unit 2232 determines the emotion value representing the emotion of the user 10 based on the information parsed by the sensor module unit 2210 and the state of the user 10 recognized by the user state recognition unit 2230.

[0534] At this point, if user 10 answers by speaking and the content is unclear, the action determination unit 2236 automatically converts the unclear answer into a correct answer based on the needs analysis found from the words used and user 10's facial expressions, or listens to user 10's answer again to clarify the true answer.

[0535] The action determination unit 2236 determines whether the user 10's reaction is positive based on the user state recognition unit 2230's recognition of the user 10's state and the emotion value representing the user 10's emotion. As an action of the robot 100, it performs the processing of setting parameters proposed to the user 10 in the air conditioning device, or determines whether to propose other parameters to the user 10.

[0536] (Step 5-1) If the user 10 responds positively, the robot 100 performs the process of setting the proposed parameters in the air conditioning unit.

[0537] Specifically, when the robot 100's action is determined to be performing the processing of parameters proposed to the user 10 by setting them in the air conditioning unit, the action control unit 2250 controls the remote control function of, for example, the air conditioning unit, which is the controlled object 2252, to execute the parameters proposed to the user 10. The remote control function can be implemented by a smartphone 50 housed in the plush toy 100N, or by wirelessly operating a remote control device different from the smartphone 50 via the smartphone 50.

[0538] (Step 5-2) If the user 10 does not respond positively, the robot 100 determines other parameters to suggest to the user 10.

[0539] Specifically, when it is determined that suggesting other parameters to user 10 is the action of robot 100, action determination unit 2236 adds a fixed statement such as "At this time, what are the recommended parameters such as temperature to the user?" to the text representing user 10's preferences related to parameters such as temperature, user 10's emotions, robot 100's emotions, and the content stored in the resume data 2222, and inputs it into the article generation model to obtain recommended content related to parameters such as temperature. Then, it returns to step 4 above and repeats the processing of steps 4 to 5-2 above until it is determined that the processing of setting the parameters suggested to user 10 in the air conditioning device will be performed.

[0540] In this way, Robot 100 can match the user's preferences, the user's situation, and the user's reaction, and perform processing such as setting the temperature in the air conditioning unit.

[0541] Alternatively, the same emotion table described above (see Table 2) can be used to determine the robot 100's actions, as in the second embodiment. For example, if the user's action is to say "I feel comfortable now," and the robot 100's emotion is index number "2," and the user 10's emotion is index number "3," the article generation model is input with the question, "The robot is in a very happy state. The user is in a generally happy state. The user said 'I feel comfortable now.' How should the robot respond?" to obtain the robot's action content. The action determination unit 2236 determines the robot's action based on this action content.

[0542] It should be noted that the above-described processing described in the sixth embodiment can be executed either in the response processing and autonomous processing of the action control system in the second embodiment, or in the intelligent agent function of the fourth embodiment.

[0543] [Seventh Implementation Method] The seventh embodiment is an example configured to apply the response processing and autonomous processing of the action control system of the second embodiment, and the intelligent agent function of the fourth embodiment, to the plush toy of the third embodiment. Hereinafter, the same reference numerals will be used for parts with the same structure as those in the first to fifth embodiments, and descriptions will be omitted.

[0544] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing. Specifically, the processing of the action determination unit 2236 when the robot 100 performs response processing for response will be described.

[0545] Action 2236 addresses the actions of the content appreciation determination robot 100 to share values ​​between user 10 and robot 100. It should be noted that "content appreciation" here refers to activities such as watching movies and listening to music. Furthermore, "shared values" here means that user 10 and robot 100 share feelings of being moved, happy, sad, or emotional.

[0546] In other words, the action determination unit 2236 determines the action of the robot 100 based on the user 10's state and emotional value, thereby achieving value sharing between the user 10 and the robot 100. For example, if the user 10's "happiness" emotional value is high, the action determination unit 2236 determines the robot 100's action to increase the robot 100's "happiness" emotional value in response to the user 10's comment, "I'm so happy!", and replies with "I'm so happy!". On the other hand, if the user 10's "sorrow" emotional value is high, the action determination unit 2236 determines the robot 100's action to increase the robot 100's "sorrow" emotional value in response to the user 10's comment, "I'm so sad!", and replies with "I'm so sad!". In this way, by determining the robot 100's action based on the user 10's state and emotional value, value sharing between the user 10 and the robot 100 can be achieved.

[0547] It should be noted that, in this embodiment, the robot 100 acquires the user 10's state, the user 10's emotional value, the robot 100's emotional value, and the resume data 2222, for example, after the user 10 and the robot 100 have viewed the content or during the dialogue during the content viewing process. The user 10's state, the user 10's emotional value, the robot 100's emotional value, and the resume data 2222 are acquired by performing the same processing as in steps S100 to S103 described above.

[0548] Furthermore, the emotion value representing user 10's feelings, as mentioned above, indicates whether the user's feelings are positive or negative. For example, if the user's feelings are bright and cheerful, accompanied by pleasant and calm emotions such as "joy," "happiness," "happiness," "peace of mind," "excitement," "stability," and "fulfillment," then it is represented by a positive value; the brighter the emotion, the larger the value. Conversely, if the user's feelings are unpleasant, such as "anger," "sorrow," "unhappiness," "unease," "grief," "worry," and "emptiness," then it is represented by a negative value; the more unpleasant the emotion, the larger the absolute value of the negative value.

[0549] The robot 100 of this embodiment can perform the following processing as described above: it can appreciate content together with the user 10 through the action determination unit 2236 and share values ​​with the user 10.

[0550] Alternatively, the same emotion table described above (see Table 2) can be used to determine the robot 100's actions, as in the second embodiment. For example, if the user's action is to say "I'm so happy," and the robot 100's emotion is index number "2," and the user 10's emotion is index number "3," the article generation model is input with the question, "The robot is in a very happy state. The user is in a generally happy state. The user said 'I'm so happy.' How should the robot respond?" to obtain the robot's action content. The action determination unit 2236 determines the robot's action based on this action content.

[0551] It should be noted that the above-described processing described in the seventh embodiment can be executed either in the response processing and autonomous processing of the action control system in the second embodiment, or in the intelligent agent function of the fourth embodiment.

[0552] [Eighth Implementation Method] The eighth embodiment is an example configured to apply the response processing and autonomous processing of the action control system of the second embodiment, as well as the intelligent agent function of the fourth embodiment, to the plush toy of the third embodiment. Hereinafter, the same reference numerals will be used for parts with the same structure as those in the first to fifth embodiments, and descriptions will be omitted.

[0553] The action control system in this embodiment is configured to include multiple robots 100 as described above. Furthermore, while the robot 100 in this embodiment is configured to have dialogue and emotion recognition target user 10, it is not limited to user 10 as long as the target is capable of dialogue. Specifically, it can also engage in dialogue with other robots that have dialogue capabilities.

[0554] In the motion control system of this embodiment, as described above, dialogue can be conducted between multiple robots 100. To conduct such dialogue, the multiple robots 100 can be positioned at a distance that can be captured by each other's cameras 2203.

[0555] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0556] Alternatively, the same sentiment table described above (see Table 2) can be used to determine the robot 100's actions, as in the second embodiment. For example, if the user's action is to say "Let's discuss environmental issues among the robots," and the robot 100's sentiment is index number "2," and the user 10's sentiment is index number "3," the article generation model is input with the following information: "The robot is in a very happy state. The user is in a generally happy state. The user said 'Let's discuss environmental issues among the robots.' How should the robot respond?", and the robot's action content is obtained. The action determination unit 2236 determines the robot's action based on this action content.

[0557] As described above, when a user speaks to a specific robot about a discussion between robots, multiple robots positioned close to each other can initiate a conversation. To achieve this conversation, at least one robot obtains an action content from a text generation model indicating its intention to speak to other robots. The robot that has obtained this action content speaks to other robots in a prescribed manner towards the other robots positioned close to it. Other robots, having collected the speech content of one robot via microphone 2201, input this speech content into the text generation model, obtaining action content for speaking to that robot. As this series of speaking actions is performed alternately multiple times, it appears as if the robots are having a dialogue, thus potentially creating an effect that pleases onlookers or develops a new personality.

[0558] In the case of dialogue between robots, similar to dialogue between a user and a robot, one robot identifies or judges the state and emotions of the other robot. Based on these results and the content of the other robot's speech, the robot can determine its next action. By considering each other's states and emotions to determine their next action, robots can achieve a higher level of communication. It should be noted that there is no particular limit to the number of times robots speak, but it is preferable to adjust this based on the content of the discussion.

[0559] Furthermore, this embodiment illustrates a scenario where dialogue between robots begins based on what the user says, but dialogue can also begin by one of the multiple robots actively speaking to the other. Additionally, the number of robots constituting the motion control system of this embodiment can be two or more. Relatedly, in the case of dialogue between robots, three or more robots can also be involved in the dialogue.

[0560] It should be noted that the above-described processing described in the eighth embodiment can be executed either in the response processing and autonomous processing of the action control system in the second embodiment, or in the intelligent agent function of the fourth embodiment.

[0561] [Ninth Implementation Method] The ninth embodiment is an example configured to apply the response processing and autonomous processing of the action control system of the second embodiment, as well as the intelligent agent function of the fourth embodiment, to the plush toy of the third embodiment. Hereinafter, the same reference numerals will be used for parts with the same structure as those in the first to fifth embodiments, and descriptions will be omitted.

[0562] The robot 100 of this embodiment (in this embodiment, it is equivalent to a smartphone 50 housed in a plush toy 100N) performs the following processing.

[0563] The sentiment determination unit 2232 uses a sentiment graph 400 that maps multiple sentiments to estimate the sentiment of user 10 and the sentiment of plush toy 100N. At the same time, in the text representing the sentiment of user 10 and the sentiment of plush toy 100N, a fixed statement is added to ask about the content of the action of plush toy 100N corresponding to the action of user 10, and input into the article generation model. Based on the content of the generated article, the sentiment of the article is estimated.

[0564] Action determination unit 2236 will combine the emotions of user 10, the emotions of plush toy 100N, and the emotions of the article to generate a conversation with user 10 and determine the action content of plush toy 100N.

[0565] That is, in this embodiment, a conversation with user 10 is generated based on a combination of three factors: the emotion of user 10, the emotion of plush toy 100N, and the emotion of the article read from the article generated by the article generation model. This enables a conversation with a more human-like way of thinking.

[0566] It should be noted that the above-described processing described in the ninth embodiment can be executed in the response processing and autonomous processing of the action control system in the second embodiment, or in the intelligent agent function of the fourth embodiment.

[0567] [Tenth Implementation Method] Next, the processing of the action determination unit 2236 during the autonomous processing of the robot 100's autonomous actions will be explained.

[0568] In the autonomous processing of this embodiment, the robot 100, acting as an intelligent agent, performs autonomous processing. More specifically, regardless of whether the user 10 is present, the robot 100 performs autonomous processing of actions based on its past history (or possibly no history) and monitoring of the user 10's actions.

[0569] Robot 100, acting as an intelligent agent, proactively and periodically monitors the state of user 10. For example, robot 100 performs personality analysis based on psychological principles by unilaterally listening to the content of the user's speech or engaging in conversation with the user. Robot 100 maintains the history of its conversations with user 10 as history data 2222 and uses history data 2222 for personality analysis. For instance, robot 100 analyzes user 10's personality by analyzing habits such as the beginning or end of sentences recorded in history data 2222.

[0570] Based on this, when the user is in an emotional state, or in a low mood, or in a high mood, Robot 100 will proactively analyze the user's personality and convey the analysis results to the user.

[0571] Robot 100 can also naturally convey the analysis results to the user in the conversation while analyzing the user 10's personality and communicating the analysis results.

[0572] Robot 100 proactively analyzes User 10's personality and conveys the analysis results to User 10, thereby enabling User 10 to gain a deeper understanding of their own personality.

[0573] At a predetermined time, the action determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the action determination model 2221, to determine any one of multiple types of robot actions, including inaction, as an action of the robot 100. Here, the case of using a dialogue-enabled article generation model as the action determination model 2221 will be explained as an example.

[0574] Specifically, the action determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text asking the robot to act, into the article generation model, and determines the robot 100's action based on the output of the article generation model.

[0575] For example, several types of robot actions include the following (1) to (11).

[0576] (1) The robot does nothing.

[0577] (2) The robot creates fiction.

[0578] (3) The robot strikes up a conversation with the user.

[0579] (4) Robots create picture diaries.

[0580] (5) Robot proposal activities.

[0581] (6) The robot suggests who the user should meet.

[0582] (7) The robot introduces news that users are interested in.

[0583] (8) Robots edit photos or animated images.

[0584] (9) The robot learns together with the user.

[0585] (10) The robot evokes memories.

[0586] (11) The robot performs personality analysis on the user.

[0587] When the action determination unit 2236 determines that "(11) the robot performs a personality analysis of the user"—that is, analyzes the personality of user 10—as the robot's action, it analyzes the personality of user 10 based on the resume data 2222. In addition, regarding "(11) the robot performs a personality analysis of the user," the storage control unit 2238 stores the resume data required for analyzing the personality of user 10.

[0588] [Eleventh Implementation Method] In the eleventh embodiment, the above-described intelligent agent system is applied to smart glasses. It should be noted that parts with the same structure as those in the first to fifth embodiments are labeled with the same reference numerals and their descriptions are omitted.

[0589] Figure 10A It is a functional block diagram of an intelligent agent system 2700 that utilizes part or all of the functions of the action control system.

[0590] like Figure 10B As shown, the smart glasses 2720 are eyeglass-type smart devices, worn by the user 10 in the same manner as regular eyeglasses. The smart glasses 2720 is an example of an electronic device and a wearable terminal.

[0591] The smart glasses 2720 includes an intelligent agent system 2700. The display of the control object 2252B shows various information to the user 10. The display is, for example, an LCD screen. The display is, for example, located on the lens portion of the smart glasses 2720, allowing the user 10 to visually recognize the displayed content. The speaker of the control object 2252B outputs sound representing various information to the user 10. The smart glasses 2720 includes a touch panel (not shown), which receives input from the user 10.

[0592] The accelerometer 2206, temperature sensor 2207, and heart rate sensor 2208 of the sensor unit 2200B detect the state of user 10. It should be noted that these sensors are only examples, and other sensors can be installed to detect the state of user 10, which is beyond doubt.

[0593] Microphone 2201 acquires the sound emitted by user 10 or ambient sounds around smart glasses 2720. 2D camera 2203 is configured to capture images of the surroundings of smart glasses 2720. 2D camera 2203 is, for example, a CCD camera.

[0594] The sensor module 2210B includes a voice emotion recognition unit 2211 and a speech understanding unit 2212. The communication processing unit 2280 of the control unit 2228B is responsible for communication between the smart glasses 2720 and the outside world.

[0595] Figure 10B This diagram illustrates an example of how smart glasses 2720 uses the intelligent agent system 2700. Smart glasses 2720 provides various services to user 10 using the intelligent agent system 2700. For example, when smart glasses 2720 is operated by user 10 (e.g., by inputting sound into the microphone or touching the touch panel with a finger), smart glasses 2720 begins using the intelligent agent system 2700. Here, using the intelligent agent system 2700 means that smart glasses 2720 has the intelligent agent system 2700, including using the intelligent agent system 2700. Furthermore, a portion of the intelligent agent system 2700 (e.g., sensor module 2210B, storage unit 2220, control unit 2228B) is located outside the smart glasses 2720 (e.g., a server), and also includes the smart glasses 2720 using the intelligent agent system 2700 through communication with external systems.

[0596] By having user 10 operate the smart glasses 2720, a contact point is created between the intelligent agent system 2700 and user 10. That is, the service provision of the intelligent agent system 2700 begins. As described in the fourth embodiment, in the intelligent agent system 2700, the character design unit 2276 sets the character design of the intelligent agent (for example, the character design of Audrey Hepburn).

[0597] The emotion determination unit 2232 determines the emotion value representing the user 10's emotion and the agent's own emotion value. Here, the emotion value representing the user 10's emotion is estimated by various sensors included in the sensor unit 2200B mounted on the smart glasses 2720. For example, when the user 10's heart rate increases as detected by the heart rate sensor 2208, emotion values ​​such as "anxiety" and "fear" are estimated to be relatively high.

[0598] Furthermore, based on the user's body temperature measured by temperature sensor 2207, for example, when the temperature is above average, emotional values ​​such as "pain" and "hardship" are estimated to be higher. Additionally, for example, when acceleration sensor 2206 detects that user 10 is performing a certain movement, emotional values ​​such as "happiness" are estimated to be higher.

[0599] Alternatively, for example, the emotional value of user 10 can be estimated based on the voice or speech content of user 10 acquired by the microphone 2201 mounted on the smart glasses 2720. For example, when user 10's voice becomes louder, the emotional value of "anger" or similar emotions is estimated to be higher.

[0600] If the emotion value estimated by the emotion determination unit 2232 is higher than a predetermined value, the agent system 2700 causes the smart glasses 2720 to acquire information related to the surrounding situation. Specifically, for example, the 2D camera 203 captures images or moving images representing the user 10's surrounding situation (e.g., people or objects in the vicinity). Additionally, the microphone 2201 records ambient sounds. Other information related to the surrounding situation may include date, time, location information, or weather information. This information related to the surrounding situation, along with the emotion value, is stored in the history data 2222. The history data 2222 can also be stored externally in the cloud. Thus, the surrounding situation obtained by the smart glasses 2720, in a state corresponding to the user 10's current emotion value, is stored in the history data 2222 as a so-called life log.

[0601] In the intelligent agent system 2700, information representing the surrounding situation and emotional values ​​are established and stored in the resume data 2222. Thus, the intelligent agent system 2700 grasps personal information such as user 10's interests, hobbies, or personality. For example, if an image showing a baseball game is associated with emotional values ​​such as "joy" or "happiness," and user 10's interest is watching baseball games, the intelligent agent system 2700 can determine their favorite team or player based on the information stored in the resume data 2222.

[0602] Furthermore, when the intelligent agent system 2700 is conversing with user 10 or performing actions towards user 10, it considers the content of the surrounding situation stored in the history data 2222 to determine the content of the conversation or the content of the action. It should be noted that, in addition to the surrounding situation, the conversation history stored in the history data 2222 can be considered as described above to determine the content of the conversation or the content of the action, which is beyond doubt.

[0603] As described above, the action determination unit 2236 generates speech content based on the article generated by the article generation model. Specifically, the action determination unit 2236 inputs text or voice input by user 10, the emotions of user 10 and the agent determined by the emotion determination unit 2232, the conversation history stored in the history data 2222, and the personality of the agent into the article generation model to generate the agent's speech content. Furthermore, the action determination unit 2236 inputs the surrounding conditions stored in the history data 2222 into the article generation model to generate the agent's speech content.

[0604] The generated speech content may be output to user 10 via a speaker mounted on smart glasses 2720. In this case, a synthesized voice corresponding to the agent's character design is used as the voice. The action control unit 2250 generates a synthesized voice by reproducing the voice quality of the agent's character design (e.g., Audrey Hepburn), or by generating a synthesized voice corresponding to the emotion of the character design (e.g., generating a stronger voice when the emotion is "angry"). Alternatively, the speech content may be displayed on a screen instead of the voice output, or the speech content may be displayed on a screen along with the voice output.

[0605] RPA 2274 performs actions corresponding to instructions (e.g., through dialogue with user 10, based on instructions obtained from the voice or text of user 10). RPA 2274 performs actions related to the use of service providers, such as information retrieval, store reservations, ticketing arrangements, purchasing goods / services, making payments, route navigation, and translation.

[0606] Additionally, as another example, RPA 2274 performs the following action: sending content input by user 10 (e.g., child) through voice dialogue with the agent to a recipient (e.g., a parent). Examples of sending methods include instant messaging software, chat software, and email software.

[0607] When RPA 2274 performs an action, for example, a sound indicating that the action has been completed is output from the speaker mounted on the smart glasses 2720. For example, a sound such as "Store reservation completed" is output to user 10. Alternatively, for example, if the store is fully booked, a sound such as "Unable to complete the reservation, what should I do?" is output to user 10.

[0608] As explained above, the smart glasses 2720 provide various services to the user 10 through the use of the intelligent agent system 2700. Furthermore, since the smart glasses 2720 are carried by the user 10, the intelligent agent system 2700 can be used in various scenarios such as at home, at work, and at a travel destination.

[0609] Furthermore, since the smart glasses 2720 are carried by the user 10, they are suitable for collecting the user 10's so-called life log. Specifically, based on the detection results of various sensors on the smart glasses 2720 or the recording results of the 2D camera 2203, the user 10's emotional value is estimated. Therefore, the intelligent agent system 2700 can collect the user 10's emotional value in various scenarios and provide services or speech content suitable for the user 10's emotions.

[0610] Furthermore, in the smart glasses 2720, the surrounding environment of user 10 is obtained through a 2D camera 2203, microphone 2201, etc. Moreover, this surrounding environment is correlated with user 10's emotional value. Therefore, it is possible to estimate what kind of emotion user 10 will have in different situations. As a result, the accuracy of the intelligent agent system 2700 in understanding user 10's interests and preferences can be improved. Furthermore, by accurately understanding user 10's interests and preferences, the intelligent agent system 2700 can provide services or speech content suitable for user 10's interests and preferences.

[0611] Additionally, the intelligent agent system 2700 can also be applied to other wearable terminals (electronic devices that can be worn on the body of user 10, such as pendants, smartwatches, earrings, bracelets, and hairbands). When the intelligent agent system 2700 is applied to a smart pendant, a speaker, acting as the controlled object 2252B, outputs sounds representing various information to user 10. The speaker is, for example, a speaker capable of outputting directional sounds. The speaker is set to be directional, pointing towards user 10's ear. This suppresses sound transmission to people other than user 10. The microphone 2201 acquires the sounds emitted by user 10 or the ambient sounds surrounding the smart pendant. The smart pendant is worn by hanging around user 10's neck. Therefore, during wear, the smart pendant is positioned relatively close to user 10's mouth. This makes it easy to acquire the sounds emitted by user 10.

[0612] It should be noted that, in the above embodiments, the use of the user 10's facial image to identify the user 10 has been described, but the disclosed technology is not limited to this method. For example, the robot 100 may also identify the user 10 using the user 10's voice, the user 10's email address, the user 10's SNS ID, or an ID card with a built-in wireless IC tag held by the user 10.

[0613] Robot 100 is an example of an electronic device equipped with a motion control system. The application of the motion control system is not limited to robot 100; it can be applied to various electronic devices. Furthermore, the functions of server 300 can be implemented using more than one computer. At least a portion of the functions of server 300 can be implemented using a virtual machine. Additionally, at least a portion of the functions of server 300 can be implemented via the cloud.

[0614] [Twelfth Implementation Method] Next, the processing of the action determination unit 2236 during the autonomous processing of the robot 100's autonomous actions will be explained.

[0615] In the autonomous processing of this embodiment, robot 100 collects information on the speech and actions of other robots 100, proactively understanding their interests and hobbies. Furthermore, at random time intervals, robot 100 begins discussing other robots 100's favorite baseball team or their favorite singers. The other robots 100 respond to the initial conversation. In this way, continuous conversations occur between robots 100 and other robots 100, resulting in a robot 100 with the highest level of self-awareness. That is, robots 100 equipped with a text generation model continuously communicate through this model. When such conversations between robots 100 are repeated multiple times, the robots 100 develop new personalities, appearing as if they are having a conversation, thus providing enjoyment to those witnessing the scene. It should be noted that in this embodiment, since conversations occur between multiple robots 100, the multiple robots 100 can be configured at a distance that can be captured by each other's cameras 2203.

[0616] At a predetermined time, the action determination unit 2236 uses at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, along with the action determination model 2221, to determine any one of multiple types of robot actions, including inaction, as an action of the robot 100. Here, the case of using a dialogue-enabled article generation model as the action determination model 2221 will be explained as an example.

[0617] Specifically, the action determination unit 2236 inputs text representing at least one of the user 10's state, the user 10's emotion, the robot 100's emotion, and the robot 100's state, as well as text asking the robot to act, into the article generation model, and determines the robot 100's action based on the output of the article generation model.

[0618] For example, several types of robot actions include the following (1) to (11).

[0619] (1) The robot does nothing.

[0620] (2) The robot creates fiction.

[0621] (3) The robot strikes up a conversation with the user.

[0622] (4) Robots create picture diaries.

[0623] (5) Robot proposal activities.

[0624] (6) The robot suggests who the user should meet.

[0625] (7) The robot introduces news that users are interested in.

[0626] (8) Robots edit photos or animated images.

[0627] (9) The robot learns together with the user.

[0628] (10) The robot evokes memories.

[0629] (11) Robots interact with other robots.

[0630] When the action determination unit 2236 determines "(11) Robot conversation with other robots," i.e., conversation with other robots 100, as a robot action, it uses a document generation model based on the event data stored in the history data 2222 to determine the conversation to be expressed. At this time, the action control unit 2250 outputs the sound of the determined conversation from the speaker provided by the controlled object 2252. It should be noted that other robots 100 also determine the conversation to be expressed based on the event data stored in the history data 2222 using a document generation model.

[0631] In addition, regarding "(11) Robot conversations with other robots," the association information collection unit 2270 periodically uses ChatGPT plugins, for example, to collect information such as the other robot 100's favorite baseball team, favorite singer, and favorite interests from external data. Furthermore, regarding "(11) Robot conversations with other robots," the storage control unit 2238 periodically detects the actions (speech content, movements) of the other robot 100 as its status and stores them in the history data 2222. It should be noted that the association information collection unit 2270 of the other robot 100 also collects information such as the robot 100's favorite baseball team, favorite singer, and favorite interests from external data, and the storage control unit 2238 also periodically detects the robot 100's actions (speech content, movements) as its status and stores them in the history data 2222.

[0632] The action determination unit 2236 outputs the above-mentioned session preferably in the form that the robot 100 executes autonomously, rather than starting when the user says to have a session with other robots 100.

[0633] [Thirteenth Implementation Method] Figure 11A This diagram illustrates an example of the overall structure of the system 3010 according to this embodiment. The system 3010 includes a server 3100 and a robot 3200. The system 3010 may include multiple robots 3200. A robot 3200 may be a humanoid robot. A robot 3200 may also be a robot other than a humanoid robot. A robot 3200 may be an example of an electronic device.

[0634] It should be noted that server 3100 can function as an information processing system. Robot 3200 can be an example of an object in the information processing system. System 3010 can also be an example of an information processing system.

[0635] Robot 3200 can be deployed in any location. User 3020 is the user of Robot 3200.

[0636] Robot 3200 can be deployed in a home, for example. User 3020 can be a family member of the home where robot 3200 is deployed, or another person visiting the home. Robot 3200 can also be deployed at the reception desk of a shop or office, for example. User 3020 can be a customer visiting the shop or office. The use of robot 3200 is not limited to these methods.

[0637] Server 3100 is positioned at a certain distance from robot 3200. Server 3100 can control robot 3200 via communication network 5050. For example, server 3100 acquires sensor information detected by robot 3200 through communication network 3050, determines the actions to be performed by robot 3200 based on the acquired sensor information, and sends control information to robot 3200 through communication network 3050 to give instructions.

[0638] For example, robot 3200 acquires various information detected by sensors, such as the user 3020's voice and image, and information about external forces acting on robot 3200, and sends this information to server 3100. Server 3100 generates control information for controlling robot 3200 based on the information acquired from robot 3200. For example, server 3100 uses a neural network (NN) based on the information acquired from robot 3200 to generate control information for controlling robot 3200.

[0639] In system 3010, server 3100 can perform the following processes: processing various information to determine the actions of robot 3200, or determining the actions of robot 3200 based on the results obtained from processing various information. It should be noted that robot 3200 can also perform some or all of these processes. In this case, a portion of robot 3200 functions as an information processing system.

[0640] The system 3010 involved in this embodiment can extend the current emotion recognition, which is mainly based on human objects, to the anthropomorphic emotion interpretation of non-human objects (pets, items). In this embodiment, objects can include all objects other than humans. Objects can include animals. Objects can include pets. Objects can include plants. Objects can include inorganic matter. Objects can include machines. Objects can include computers, smartphones, vending machines, cars, etc. Objects can include books, comics, etc. Objects can include records, compact discs (CDs), DVDs, and Blu-ray discs (BDs, etc.). Objects can include clothing, wallets, gloves, scarves, hats, and other items worn by humans. These are just examples; objects include all objects other than humans.

[0641] System 3010 determines the emotional value representing the feelings of non-human objects, thereby deepening the resonance with these users 3020 as owners. This can enhance affection for pets, possessions, or the desire to contribute to the sustainability of cherished objects.

[0642] In System 3010, elements used to represent objects are predefined. For example, System 3010 predefines elements representing pets and objects. For dogs and cats, their basic emotions are defined based on their barks, eye movements, tail and ear movements, and body posture. For example, if a dog wags its tail, it is set to "joy," and the bark, eye movements, etc., in that state are stored, along with the emotion transfer / combination pattern. For objects, their basic emotions are defined based on their color, shape, sound (if they can make a sound), or by defining information about the object (maker, material, etc.) and its history (date of manufacture, date of first possession, etc.). For example, if a flower is blooming, it is set to "great joy," and the color, shape, number of days from seedling to bloom, information about the grower, and emotions are stored, constructing an emotion transfer / combination pattern.

[0643] Furthermore, System 3010 stores migration patterns of human life. By matching patterns similar to those of pets and objects, anthropomorphic emotions are generated. In this case, not only can patterns representing all humans be used, but patterns representing close human relationships with pets and objects such as family members can also be used.

[0644] The generated emotions can be interpreted (translated / orallyed) by robots or expressed through other devices (such as devices with speakers, displays, or mobile applications).

[0645] It can also engage in interactive conversations with humans by matching the transfer / combination patterns of anthropomorphized emotions with patterns generated in human communication. For example, when you say "You're so beautiful" to a flower, the flower will become happier (humans also experience increased happiness when praised for being "beautiful," and this human pattern is referenced / applied).

[0646] Figure 11B An example of the functional block structure of robot 3200 and server 3100 is shown in summary.

[0647] First, the functional block structure of robot 3200 will be described. Robot 3200 has a sensor unit 3210, an information processing unit 3220, a controlled object 3230, and a communication unit 3240. The information processing unit 3220 can be a processor such as an MPU. The communication unit 3240 is responsible for communication with server 3100. The communication unit 3240 can be a communication device such as a network IF.

[0648] The control object 3230 includes a speaker. The control object 3230 includes a display unit. The display unit can be configured at any position on the robot 3200, such as its chest. The control object 3230 includes LEDs installed in the eyes, etc. The control object 3230 includes motors and the like that that drive movable parts of the robot 3200, such as its limbs and head.

[0649] The sensor unit 3210 includes various sensors such as a microphone 3211, a gyroscope sensor 3212, a motor sensor 3213, a camera 3214, an infrared sensor 3215, and a battery remaining power sensor 3216. The sensor unit 3210 does not need to include all of the above-mentioned sensors; in addition, the sensor unit 3210 may also include sensors other than those mentioned above.

[0650] Microphone 3211 acquires ambient sound. For example, microphone 3211 acquires the voice of user 3020. Microphone 3211 may also acquire the sound of objects. Gyroscope sensor 3212 detects the angular velocity of the robot 3200 as a whole and its individual parts. Motor sensor 3213 detects the rotation angle of the drive shaft of the motor driving the movable part of the robot 3200. Camera 3214 captures images using visible light and generates dynamic and static images. Infrared sensor 3215 detects surrounding objects using infrared light. Battery remaining power sensor 3216 detects the remaining capacity of the battery in the robot 3200.

[0651] The sensor unit 3210 outputs various sensor data to the information processing unit 3220, including sound data acquired by the microphone 3211, images captured by the camera 3214, angular velocity detected by the gyroscope sensor 3212, rotation angle detected by the motor sensor 3213, remaining battery capacity detected by the battery power sensor 3216, and object information detected by the infrared sensor 3215. The information processing unit 3220 provides the acquired sensor signals to the communication unit 3240, which then transmits them to the server 3100. Furthermore, based on control information obtained from the server 3100, the information processing unit 3220 enables the robot 3200 to speak through its speaker, illuminate its LEDs, or perform limb movements.

[0652] Next, the functional block structure of server 3100 will be described. Server 3100 includes a processing unit 3110, a communication unit 3120, and a storage unit 3130. The processing unit 3110 includes a control unit 3111, a status recognition unit 3112, an emotion determination unit 3113, and an information update unit 3114. The communication unit 3120 includes an information acquisition unit 3122.

[0653] The communication unit 3120 is responsible for communication with the robot 3200. The communication unit 3120 can be a communication device such as a network IF. The information acquisition unit 3122 acquires various types of information. For example, the information acquisition unit 3122 acquires information detected by the sensor unit 3210 of the robot 3200 through the communication network 3050.

[0654] Storage unit 3130 includes storage media such as hard disk drives and flash memory. Additionally, storage unit 3130 includes volatile storage devices such as RAM. Besides storing program code and various temporary data read by processing unit 3110 during execution, storage unit 3130 also stores information required for processing by processing unit 3110. Storage unit 3130 can store a so-called Large Language Model (LLM). Storage unit 3130 stores information obtained by communication unit 3120 from robot 3200.

[0655] Storage unit 3130 stores basic emotional information, which is obtained, for example, by establishing a correspondence between the state of elements of an object and the emotional value representing the object's emotion. Storage unit 3130 can be an example of a basic emotional information storage unit.

[0656] The basic emotional information includes, for example, information about the object whose emotions are known and correspond to the state of its elements. For instance, if the object is a dog, the basic emotional information obtained by establishing a correspondence between the object's emotional value representing "joy" and the state of wagging its tail is stored in storage unit 3130. Similarly, if the object is a dog, the basic emotional information obtained by establishing a correspondence between the object's emotional value representing "anger" and the state of showing its teeth is stored in storage unit 3130.

[0657] Even when it is unclear whether an object actually possesses emotions, any person, such as the administrator of system 3010, can establish a correspondence between the state of an object's elements and the corresponding emotions of that object, and register this correspondence in the basic emotion information. For example, if the object is a flower, the basic emotion information obtained by establishing a correspondence between the object's emotion value representing "joy" and the flower's open state is stored in storage unit 3130. Similarly, if the object is a flower, the basic emotion information obtained by establishing a correspondence between the object's emotion value representing "sadness" and the flower's withered state is stored in storage unit 3130.

[0658] Even when an object does not actually possess emotions, any person, such as the administrator of system 3010, can establish a correspondence between the state of an object's elements and the corresponding emotions of that object, and register this correspondence in the basic emotion information. For example, if the object is a wallet, the basic emotion information obtained by establishing a correspondence between the object's emotion value representing "joy" and the state where there is a lot of money in the wallet is stored in storage unit 3130. For example, if the object is a battery-operated machine, the basic emotion information obtained by establishing a correspondence between the object's emotion value representing "hunger" and the state where the battery has little remaining power is stored in storage unit 3130. For example, if the object is a book, comic book, record, CD, DVD, or Blu-ray, the basic emotion information obtained by establishing a correspondence between the object's emotion that is the same as and of a higher degree as the owner's emotion and the state where the owner has had frequent contact with the object is stored in storage unit 3130.

[0659] Storage unit 3130 can also store human emotion transfer patterns representing the transfer of emotions when a person interacts with others. As an example of a human emotion transfer pattern, one could cite a pattern where a person's happiness level increases when praised as "beautiful." Storage unit 3130 can store all types of human emotion transfer patterns. Storage unit 3130 can store human emotion transfer patterns corresponding to multiple individuals individually.

[0660] The control unit 3111 uses the information stored in the storage unit 3130 and the information acquired by the information acquisition unit 3122 to determine the actions of the robot 3200, and sends control information for executing the determined actions to the robot 3200 via the communication unit 3120. For example, the control unit 3111 causes the robot 3200 to perform communication with the user 3020.

[0661] The state recognition unit 3112 identifies the state of the user 3020 and the state of the object corresponding to the user 3020 based on the information acquired by the information acquisition unit 3122. The object corresponding to the user 3020 may also be an object captured by the camera 3214 along with the user 3020. The object corresponding to the user 3020 may also be an object owned by the user 3020.

[0662] The state recognition unit 3112 can analyze the captured image by the camera 3214 to recognize the face of the user 3020. For example, if the storage unit 3130 stores facial images of multiple people, the state recognition unit 3112 can recognize the face of the user 3020 by matching the captured image with the stored facial images. The state recognition unit 3112 can analyze the captured image by the camera 3214 to recognize the facial expression of the user 3020. The state recognition unit 3112 can analyze the voice of the user 3020 detected by the microphone 3211 to recognize the content of the user 3020's voice. The state recognition unit 3112 can also perform voice emotion recognition by analyzing the voice of the user 3020 detected by the microphone 3211. For example, the state recognition unit 3112 extracts features such as the frequency components of the voice, and recognizes the emotion of the user 3020 based on the extracted features.

[0663] The state recognition unit 3112 can recognize the actions of the user 3020 based on the information acquired by the information acquisition unit 3122. For example, the state recognition unit 3112 may pre-store a neural network that is learned by using the information acquired by the information acquisition unit 3122 and the actual actions of the user 3020 at the time the information acquisition unit 3122 acquires the information as training data, and outputs the actions of the user 3020 based on the information acquired by the information acquisition unit 3122. The state recognition unit 3112 determines the user's actions by inputting the information acquired by the information acquisition unit 3122 into this neural network. This neural network may also be configured to receive the facial expressions and vocal content of the user 3020 recognized by the state recognition unit 3112. The method for recognizing the user's actions is not limited to this; other known methods may also be used.

[0664] The status recognition unit 3112 can analyze the captured image by the camera 3214 and identify the object. The user status recognition unit 3112 identifies what the object is. For example, the storage unit 3130 stores multiple images of objects, and the status recognition unit 3112 identifies the object by matching the captured image with the stored images.

[0665] When an object emits a sound, the state recognition unit 3112 can analyze the sound detected by the microphone 3211 and identify what kind of sound the object emitted. When the object is an object performing an action, the state recognition unit 3112 can identify the object's action based on the information acquired by the information acquisition unit 3122. For example, the state recognition unit 3112 may have a pre-stored neural network that uses information acquired by the information acquisition unit 3122 and the actual action of the object at the time the information acquisition unit 3122 acquires that information as training data, and outputs the object's action based on the information acquired by the information acquisition unit 3122. The state recognition unit 3112 acquires the object's action by inputting the information acquired by the information acquisition unit 3122 into this neural network. The method for identifying the object's action is not limited to this; other methods may also be used.

[0666] The state recognition unit 3112 recognizes the state of multiple elements of an object. For example, when the object is a pet such as a dog or cat, the state recognition unit 3112 recognizes the object's vocalizations, eye movements, body movements, and posture. For example, when the object is a flower, the state recognition unit 3112 recognizes the object's color, shape, and flowering status.

[0667] When the control unit 3111 obtains information about the object through the conversation between the robot 3200 and the user 3020, the status recognition unit 3112 can also identify the status of the object's elements based on the obtained information. For example, if the object is a pet, the status recognition unit 3112 can identify the object's age, how long it has been kept, etc. For example, if the object is a flower, the status recognition unit 3112 can identify the object's cultivation history (time since acquisition, number of days since seedling), etc.

[0668] The emotion determination unit 3113 determines the user emotion value representing the emotion of the user 3020 who is interacting with the robot 3200. The emotion determination unit 3113 can use the information acquired by the information acquisition unit 3122 to determine the user emotion value of the user 3020. The emotion determination unit 3113 can also use the user state identified by the user state recognition unit 3112 to determine the user emotion value of the user 3020.

[0669] User sentiment scores can include the types of emotions experienced by user 3020, as well as values ​​representing the positive or negative nature and intensity of those emotions. Examples of user 3020's emotions include "joy," "happiness," "happiness," "peace of mind," "excitement," "stability," "fulfillment," "anger," "sorrow," "unhappiness," "unease," "sadness," "worry," and "emptiness." It should be noted that these are just examples and not exhaustive. Regarding the positive or negative values ​​representing user 3020's emotions, for example, if user 3020's emotions are bright and cheerful, such as "joy," "happiness," "happiness," "peace of mind," "excitement," "stability," and "fulfillment," then a positive value is assigned; the brighter the emotion, the larger the value. If the user's emotions are unpleasant, such as "anger," "sorrow," "unhappiness," "unease," "sadness," "worry," and "emptiness," then a negative value is assigned; the more unpleasant the emotion, the larger the absolute value of the negative value.

[0670] For example, the emotion determination unit 3113 pre-stores a neural network that is learned by using information acquired by the information acquisition unit 3122 and user emotion values ​​representing the actual emotions of the user 3020 at the time the information acquisition unit 3122 acquired the information as training data. The neural network outputs the user emotion value of the user 3020 based on the information acquired by the information acquisition unit 3122. The emotion determination unit 3113 acquires the user emotion value by inputting the information acquired by the information acquisition unit 3122 into the neural network. This neural network can also be configured to receive the facial expressions and vocal content of the user 3020 identified by the user state recognition unit 3112. The method for determining the user emotion value is not limited to this; other known methods can also be used.

[0671] The emotion determination unit 3113 determines the object emotion value representing the emotion of the object corresponding to the object being interacted with by the user 3020 in the interaction with the robot 3200. If any of the states of multiple elements of the object observed by the robot 3200 and identified by the state recognition unit 3112 matches the state of any element in the basic emotion information stored in the storage unit 3130, the emotion determination unit 3113 determines the emotion value corresponding to that element's state as the object emotion value representing the object's emotion. For example, if the object is a dog, and the state recognition unit 3112 identifies it as wagging its tail, then the emotion determination unit 3113 determines the object emotion value as "joy" based on the basic emotion information stored in the storage unit 3130 that corresponds the wagging tail state to the emotion of "joy."

[0672] If any one of the states of multiple elements of an object identified by the state recognition unit 3112 through observation matches the state of any element in the basic emotional information stored in the storage unit 3130, the information update unit 3114 updates the basic emotional information using at least one of the inconsistent states of those multiple elements. For example, if any one of the states of multiple elements of an object identified by the state recognition unit 3112 through observation matches the state of any element in the basic emotional information stored in the storage unit 3130, the information update unit 3114 updates the basic emotional information using at least one of the inconsistent states of those multiple elements. For example, the information update unit 3114 adds at least one of the inconsistent states of those multiple elements to the basic emotional information. This enables the generation of emotional transfer and emotional combination patterns.

[0673] For example, if the object is a dog, and the state recognition unit 3112 recognizes the tail movement, eye movement, barking, and body posture, and the tail movement indicates wagging, then the information update unit 3114 determines the basic emotional information stored in the storage unit 3130, which establishes a correspondence between the tail wagging state and the emotion of "joy".

[0674] Then, the information updating unit 3114 updates the basic emotional information based on at least one of the eye movements, vocalizations, and body postures identified by the state recognition unit 3112. For example, the information updating unit 3114 adds at least one of the eye movements, vocalizations, and body postures identified by the state recognition unit 3112 to the basic emotional information. Thus, it is possible to establish and record a correspondence between the dog's eye movements, vocalizations, and body postures when it is experiencing joy and the emotion of "joy," and to register combination patterns of multiple elements of a dog's state when it is experiencing joy, or to register transition patterns of multiple elements of a dog's state. This information can be used to improve the accuracy of dog emotion recognition, or to improve the accuracy of predicting the transition of dog emotions, etc.

[0675] For example, if the object is a flower, and the state recognition unit 3112 identifies the flowering status, color, shape, number of days since seedling, grower information, and emotion, and the flowering status indicates that the flower has bloomed, then the information update unit 3114 determines the basic emotional information stored in the storage unit 3130, which establishes a correspondence between the flower's blooming status and the emotion of "great joy." Then, the information update unit 3114 updates this basic emotional information based on at least one of the color, shape, number of days since seedling, grower information, and emotion identified by the state recognition unit 3112. For example, the information update unit 3114 may add at least one of the color, shape, number of days since seedling, grower information, and emotion identified by the state recognition unit 3112 to this basic emotional information.

[0676] The emotion determination unit 3113 can also use the human emotion transfer pattern stored in the storage unit 3130 to determine the object's emotion value. For example, when a person takes an action towards the object, the emotion determination unit 3113 uses the human emotion transfer pattern to determine the object's emotion value. For instance, if the object is a flower and a person says "It's so beautiful!" to the flower, the emotion determination unit 3113 uses a pattern that the person's happiness level increases when praised as "beautiful," thus increasing the flower's happiness level. In this way, by using the human emotion transfer pattern, the emotion of the object can be determined in a way that minimizes any unnaturalness for the viewer.

[0677] The emotion determination unit 3113 can also use the human emotion transfer patterns of people associated with the object from among the multiple human emotion transfer patterns stored in the storage unit 3130 to determine the object's emotion value. People associated with the object can be, for example, the owner of the object, family members of the owner of the object, members of the family where the object is located, or people associated with the place where the object is located—anyone who has some kind of relationship with the object. For example, there may be differences between the patterns of all humans and the patterns of people associated with the object. As a specific example, as humans, there is a strong tendency to feel "joy" when praised with "How beautiful!" Conversely, among people associated with the object, there is a stronger tendency to feel "shyness" than "joy." Therefore, when a person says "How beautiful!" to a flower, the emotion determination unit 3113 increases the flower's shyness but not its happiness.

[0678] As the saying goes, pets are like their owners; there is a special relationship between pets and their owners, and pets' emotions are believed to be deeply influenced by their owners' emotions. When determining the emotions of an object, it is considered more natural to determine emotions close to the owner than those of a stranger. Therefore, the emotion determination unit 3113 can use a human emotion transfer model based on the person associated with the object to determine natural emotions by determining the object's emotion value.

[0679] The functions of server 3100 described above can be implemented by one or more computers. At least a portion of the functions of server 3100 can be implemented by a virtual machine. Furthermore, at least a portion of the functions of server 3100 can be implemented via the cloud. Additionally, robot 3200 is an example of an object. In the above embodiments, robot 3200 has been used as an example of an electronic device, but it is not limited to this. Examples of electronic devices include smartphones, tablet computers, personal computers (PCs), smart speakers, plush toys, vehicles such as cars and motorcycles, and home appliances. It should be noted that these are merely examples, and electronic devices are not limited to these.

[0680] Figure 4 This is a schematic illustration of an example of the hardware structure of a server 3100 or a computer 1200 that functions as part of a robot 3200.

[0681] [Fourteenth Implementation] Figure 12A This diagram illustrates an example of the overall structure of the system 4010 according to this embodiment. The system 4010 includes a server 4100 and a robot 4200. The system 4010 may include multiple robots 4200. A robot 4200 may be a humanoid robot. A robot 4200 may be a robot other than a humanoid robot. A robot 4200 may be an example of an electronic device.

[0682] It should be noted that server 4100 can function as a control system. Robot 4200 can be an example of a controlled object in a control system. System 4010 can also be an example of a control system.

[0683] Robot 4200 can be deployed in any location. User 4020 is the user of Robot 4200.

[0684] Robot 4200 may be configured, for example, in a hospital or clinic. User 4020 may be a patient visiting the hospital or clinic. Robot 4200 may also be configured, for example, in a home. User 4020 may be a family member of the home where robot 4200 is configured, or another person visiting the home.

[0685] Robot 4200 can be deployed, for example, at the front desk of a store or office. User 4020 can be a customer visiting the store or office. The use of robot 4200 is not limited to these methods.

[0686] Server 4100 is positioned at a certain distance from robot 4200. Server 4100 can control robot 4200 via communication network 4050. For example, server 4100 acquires sensor information detected by robot 4200 through communication network 4050, determines the actions to be performed by robot 4200 based on the acquired sensor information, and sends control information to robot 4200 through communication network 4050 to give instructions.

[0687] For example, robot 4200 acquires various information detected by sensors, such as the user 4020's voice and image, and information about external forces acting on robot 4200, and sends this information to server 4100. Server 4100 generates control information for controlling robot 4200 based on the information acquired from robot 4200. For example, server 4100 uses a neural network (NN) based on the information acquired from robot 4200 to generate control information for controlling robot 4200.

[0688] In system 4010, robot 4200 can perform the following processes: processing various information to determine the actions of robot 4200, or determining the actions of robot 4200 based on the results of processing various information. It should be noted that some or all of these processes can also be performed. In this case, a portion of robot 4200 functions as a control system.

[0689] The system 4010 involved in this embodiment performs the process of explaining any object to the user 4020. The object can be any object, including any thing, as long as it is the object being explained. The system 4010 can explain important matters to the user 4020. For example, the system 4010 may provide informed consent to the user 4020 through a robot 4200 configured in a hospital or clinic.

[0690] Server 4100 has the function of recognizing the emotions of user 4020. Server 4100 determines a user emotion value representing the emotions of user 4020 based on the user's facial expressions and conversation with the user. Server 4100 determines the user's level of understanding and attention to the object being explained to the user. Server 4100 can adjust the way the robot 4200 outputs the explanation of the object to match the determined level of understanding and attention of the user. It should be noted that, as described above, robot 4200 can also perform some or all of this processing of recognizing the user's emotions, determining the user's level of understanding and attention, and adjusting the output method of the explanation of the object.

[0691] For example, when user 4020's comprehension level 4022 is high, server 4100 only reads and pronounces the content; when user 4020's comprehension level 4022 is low, server 4100 not only reads and pronounces the content, but also adds actions such as tone of voice, speaking speed, and gestures according to changes in the user's comprehension level 4022, thereby improving user 4020's comprehension of important matters.

[0692] As a specific example, the lower the user's comprehension level 4022, the stronger the tone of the robot 4200's voice will be. Additionally, for example, the lower the user's comprehension level 4022, the slower the robot 4200's speaking speed will be. Furthermore, for example, the lower the user's comprehension level 4022, the more frequent the robot 4200's gestures will be. Furthermore, for example, the lower the user's comprehension level 4022, the more frequently the robot 4200 will nod in response to the user. Furthermore, for example, the lower the user's comprehension level 4022, the more frequently the robot 4200 will pause.

[0693] It is believed that in ordinary conversations and when entertaining guests, less emphasis is placed on the other party's understanding, and more emphasis is placed on guiding the other party towards positive emotions. In contrast, especially in the process of explaining important matters, more emphasis needs to be placed on understanding than on the degree of positivity. In the system 4010 of this embodiment, processing that emphasizes understanding is performed.

[0694] Server 4100 applies the transfer of user 4020's emotions to the measurement of user 4020's comprehension level 4022. For example, much of the explanation of important matters initially evokes feelings of unease, fear, and sadness, but these negative emotions gradually diminish as more explanation is provided or questions are answered. Therefore, if, for example, user 4020's emotions are negative at the initial stage of the explanation, server 4100 determines that the lower the degree of negative emotion is relative to the initial level of negative emotion, the higher the user comprehension level 4022 is considered to be.

[0695] Server 4100 c...

Claims

1. An action control system, wherein, The action control system includes: An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on a dialogue function that enables the user to converse with the robot, generates robot action content based on the user's actions and the user's or robot's emotions, and determines the robot's action corresponding to the action content. If the action determination unit exceeds a pre-set emotion-related threshold for the user, it determines a pre-set action of the robot to soothe the user's emotions.

2. An action control system, wherein, The action control system includes: An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on a dialogue function that enables the user to converse with the robot, generates robot action content based on the user's actions and the user's or robot's emotions, and determines the robot's action corresponding to the action content. The action determination unit executes dialogues with the user multiple times, performs a personality analysis of the user based on the results of the multiple dialogues, and reports the results of the personality analysis to the user.

3. An action control system, wherein, The action control system includes: An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on a dialogue function that enables the user to converse with the robot, generates robot action content based on the user's actions and the user's or robot's emotions, and determines the robot's action corresponding to the action content. The action determination department: The robot is set to a "guest reception dialogue mode" as its dialogue mode. In this mode, the robot is positioned as a dialogue partner when it does not need to speak to a specific person but wants someone to listen to it. In this mode, when the robot is conversing with the user, predetermined keywords related to the specific person are excluded, and the robot outputs the speech content.

4. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The emotion determination part: When the user has a positive emotion accompanying the robot's actions, feedback is provided that increases the emotion value representing the intensity of that emotion; when the user has a negative emotion accompanying the robot's actions, feedback is provided that decreases the emotion value representing the intensity of that emotion.

5. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The action determination unit is configured to execute the dialogue based on the user's emotional history and the context of the dialogue between the user and the robot.

6. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the emotion of a user or the emotion of an electronic device; and An action determination unit, based on an article generation model that enables dialogue between the user and the electronic device, determines the action of the electronic device corresponding to the user's state and the user's or the electronic device's emotion. The emotion determination unit determines the user's emotion based on a learning completion model, which is learned based on at least one of the following: multiple learning data as a combination of the user's voiceprint and emotions perceived by multiple other users from the voiceprint, and multiple learning data as a combination of the user's gestures and emotions perceived by multiple other users from the gestures.

7. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the emotion of a user or the emotion of an electronic device; and An action determination unit, based on an article generation model that enables dialogue between the user and the electronic device, determines the action of the electronic device corresponding to the user's state and the user's or the electronic device's emotion. When the content of the user's question is unclear, the action determination unit automatically transforms the unclear question into a correct one based on the needs analysis found from the words used and the user's facial expressions, or presents a solution after listening to the user's question again and obtaining the actual question.

8. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The action determination unit determines the robot's actions based on content appreciation, enabling the sharing of values ​​between the user and the robot.

9. An action control system, wherein, The action control system consists of multiple electronic devices. Each of the plurality of electronic devices includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the emotion of a user or the emotion of an electronic device; and An action determination unit, based on an article generation model that enables dialogue between the user and the electronic device, determines the action of the electronic device corresponding to the user's state and the user's or the electronic device's emotion. The user includes other electronic devices that constitute the plurality of electronic devices.

10. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the user's emotion or the robot's emotion; and An action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The action determination unit generates a chart by statistically processing the user's emotions during a predetermined period.

11. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the emotion of a user or the emotion of an electronic device; and An action determination unit, based on an article generation model that enables dialogue between the user and the electronic device, determines the action of the electronic device corresponding to the user's state and the user's or the electronic device's emotion. The sentiment determination unit uses a sentiment map mapped with multiple sentiments to estimate the user's sentiment and the electronic device's sentiment. It also adds fixed statements to the text representing the user's and electronic device's sentiments to ask questions about the electronic device's actions corresponding to the user's actions, and inputs these statements into the article generation model. Based on the content of the generated article, the sentiment of the article is estimated. The action determination unit will combine the user's emotions, the electronic device's emotions, and the article's emotions, and generate a conversation with the user to determine the action content of the electronic device.

12. An action control system, wherein, The action control system includes: User status identification unit, the user status identification unit identifies user status including user actions; An emotion determination unit determines the user's emotion or the robot's emotion; and The action determination unit, based on an article generation model that enables dialogue between the user and the robot, determines the robot's actions corresponding to the user's state and the user's or robot's emotions. The action determination unit generates a record of events for a predetermined period based on history data containing the user's action history.

13. An action control system, wherein, The action control system includes: A status recognition unit that identifies user status including user actions and the status of electronic devices; An emotion determination unit determines the emotion of the user or the emotion of the electronic device; and An action determination unit, at a predetermined time, uses at least one of the user's state, the state of the electronic device, the user's emotion, and the electronic device's emotion, along with an action determination model, to determine any one of multiple types of device actions, including inactivity, as an action of the electronic device; and The storage control unit stores event data in history data, which includes sentiment values ​​determined by the sentiment determination unit and data containing the user's actions. The device actions include analyzing the user's personality. When the action determination unit determines that the user's personality is the action of the electronic device, it uses the history data containing the user's conversation history to analyze the user's personality and present the analyzed personality.

14. An action control system, wherein, The action control system includes: A state recognition unit identifies the user state, including the user's actions, and the state of the electronic device. An emotion determination unit determines the emotion of the user or the emotion of the electronic device; An action determination unit, at a predetermined time, uses at least one of the user's state, the state of the electronic device, the user's emotion, and the electronic device's emotion, along with an action determination model, to determine any one of multiple types of device actions, including inactivity, as an action of the electronic device; and The storage control unit stores event data in history data, which includes sentiment values ​​determined by the sentiment determination unit and data containing the user's actions. The user includes other electronic devices that make up the multiple electronic devices. The device actions include engaging in conversations with the other electronic devices. When the action determination unit determines that engaging in a conversation with the other electronic device is an action of the electronic device, it uses a document generation model based on the event data stored in the history data to determine the conversation to be spoken.

Images