Control system

Abstract

A control system according to the present invention includes a state recognition unit, an emotion determination unit, and an action determination unit. An electronic device is detachably provided to a moving unit having a wheel for automatic travel. The electronic device includes one or a plurality of sensors for collecting information for controlling rotation of the wheel. Device operations include the setting of first action content for guiding a user by controlling the wheel.

Description

Control System

[0001] The present invention relates to a control system.

[0002] Patent Literature 1 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology in Patent Literature 1 recognizes the user's reaction when the robot performs a specific action, and if the robot is unable to determine an action for the recognized user reaction, it updates the robot's behavior by receiving information about an action appropriate to the recognized user's state from a server.

[0003] Furthermore, Patent Document 2 discloses a communication terminal with a camera and a neck strap.

[0004] Japanese Patent No. 6053847 Japanese Patent Application Laid-Open No. 2019-215444

[0005] However, in the conventional technology, the electronic device assists the user while remaining stationary in a specific position, so there is room for improvement in terms of assisting the user.

[0006] Furthermore, although conventional technology can prevent the neck strap from being captured by the camera, it does not support the user's actions, and therefore there is room for improvement in terms of supporting the user.

[0007] Furthermore, while conventional technology can prevent the neck strap from being captured on camera, it does not prevent suspicious individuals from approaching children, so there is room for improvement in preventing suspicious individuals from approaching children.

[0008] Furthermore, although conventional technology can prevent the neck strap from being captured by the camera, it does not support the user's actions, and therefore there is room for improvement in terms of supporting the user.

[0009] According to a first aspect of the present disclosure, there is provided a control system. The control system includes a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of multiple types of device activation, including no activation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model. The electronic device is detachably mounted on a mobile unit having wheels for autonomous driving, and the electronic device includes one or more sensors that collect information for controlling the rotation of the wheels. The device activation includes setting a first behavior content for leading the user by controlling the wheels. The electronic device includes a communication terminal. Here, the communication terminal includes a device that performs physical operations, a device that outputs video and audio without performing physical operations, and an agent that operates on software.

[0010] According to a second aspect of the present disclosure, there is provided a control system. The control system includes a state recognition unit that recognizes a user state, including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of multiple types of device activation, including no activation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model. The electronic device is detachably attached to a neck strap and includes one or more sensors that collect information. The device activation includes setting a first action content that guides the user by the electronic device playing at least one of audio and images. Here, the electronic device includes a communication terminal. The communication terminal includes a device that performs a physical action, a device that outputs video or audio without performing a physical action, and an agent that runs on software.

[0011] According to a third aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device operations (including no operation) as an action of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting a first action content to guide the user by the electronic device reproducing at least one of audio and images.

[0012] According to a fourth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user, an emotion of another user present around the user, or an emotion of the electronic device; an action determination unit that determines, when an emotion value that represents the intensity of the emotion of the other user changes by a predetermined value or more, to start shooting a video of the other user with a camera as an action of the electronic device; and an action control unit that controls the electronic device to perform the action of the electronic device determined by the action determination unit.

[0013] According to a fifth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including actions of a user and other users; an emotion determination unit that determines emotions of the user and other users; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations as an operation of an electronic device using at least one of the user state and the user emotion and a behavior determination model, wherein the state recognition unit recognizes a surrounding state from one or more sensors included in the electronic device that is detachably attached to the user's body, and the behavior determination unit determines, when the other user is present near the user, that the electronic device should speak to the other user.

[0014] According to a sixth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including no operation, as an action of the electronic device using at least one of the user state, the state of the electronic device, the user emotion, and the emotion of the electronic device and a behavior determination model, wherein the emotion determination unit infers the emotions of others around the user from images of the others acquired from one or more information-collecting sensors included in the electronic device that is detachably attached to the user's body, and the behavior determination unit makes an appropriate suggestion to the user when the others are feeling distressed.

[0015] According to a seventh aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device activation, including no activation, as an behavior of the electronic device using at least one of the user state, the state of the electronic device, the user emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably attached to a neck strap and includes one or more sensors that collect information, including a camera that captures images of the surroundings, and the device activation includes issuing a warning to a person who approaches the camera and satisfies a predetermined condition.

[0016] According to an eighth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations (including no operation) as an action of the electronic device using at least one of the user state, the state of the electronic device, the user emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting an action content to assist the user in performing a task at a level lower than the user's eyes by the electronic device playing at least one of audio and images generated by the behavior determination model based on the information collected by the electronic device.

[0017] According to a ninth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including inoperation, as an action of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably provided on a mobile unit having wheels for autonomous driving, the electronic device includes one or more sensors that collect information on a situation around the user, the electronic device controls rotation of the wheels based on the information on the situation around the user, and when a dangerous obstacle is detected around the user, the action determination unit determines, as the action of the electronic device, a first action content that guides the user to a safe place to prevent the user from approaching the dangerous obstacle.

[0018] According to a tenth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user action and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is provided on a mobile unit capable of autonomous driving, the electronic device includes one or more sensors that collect information regarding a situation around the user, and the action determination unit determines, when it is detected that the user is in danger based on the information regarding the situation around the user, that the action of the electronic device is to take an action in place of the user.

[0019] According to an eleventh aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably provided on a mobile unit having wheels for autonomous driving, the electronic device includes one or more sensors that collect information regarding a situation around the user, and the device operation includes setting a first action content to assist the user in walking an animal by controlling the wheels based on the information regarding the situation around the user.

[0020] According to a twelfth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device activation, including no activation, as an action of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably provided on a neck strap includes one or more sensors that collect information on a surrounding situation, and the device activation includes setting an action content to assist the user in walking by notifying the user of the surrounding situation by playing at least one of audio and images by the electronic device.

[0021] According to a thirteenth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device activation, including no activation, as an behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably provided on a neck strap and includes one or more sensors that collect biometric information of the user, and the device activation includes notifying the user of attention alert information based on the collected biometric information by the electronic device playing at least one of audio and images.

[0022] According to a fourteenth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device activation, including no activation, as an behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably provided on a neck strap includes one or more sensors that collect ambient environment information, and the device activation includes notifying the user of attention alert information based on the collected ambient environment information by the electronic device playing at least one of audio and images.

[0023] According to a fifteenth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as an action of the electronic device, using at least one of the user state, the state of the electronic device, the user emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably attached to a neck strap and includes one or more sensors that collect information, and the device operation includes setting an action content to assist the user in working at a position lower than the user's eyes by the electronic device playing at least one of audio and images generated by the behavior determination model based on the information collected by the electronic device.

[0024] According to a sixteenth aspect of the present disclosure, there is provided a control system including: a state recognition unit that recognizes a user state including a user behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device activation, including no activation, as an behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably attached to a neck strap and includes one or more sensors that collect information, the electronic device cooperates with another electronic device that is worn by the user and includes one or more sensors that collect information, and the device activation includes setting an action content to assist the user by the electronic device reproducing at least one of audio and images generated by the behavior determination model based on the information collected by the electronic device and the information collected by the other electronic device.

[0025] FIG. 1 schematically illustrates an example of a system 5 according to the first embodiment. FIG. 2 schematically illustrates the functional configuration of a robot 100 according to the first embodiment. FIG. 3 schematically illustrates an example of an operational flow of collection processing by the robot 100 according to the first embodiment. FIG. 4A schematically illustrates an example of an operational flow of response processing by the robot 100 according to the first embodiment. FIG. 4B schematically illustrates an example of an operational flow of autonomous processing by the robot 100 according to the first embodiment. FIG. 5 illustrates an emotion map 400 onto which multiple emotions are mapped. FIG. 6 illustrates an emotion map 900 onto which multiple emotions are mapped. FIG. 7A is an external view of a stuffed animal 100N according to a second embodiment, and FIG. 7B illustrates an internal structural diagram of the stuffed animal 100N. FIG. 8 is a rear front view of the stuffed animal 100N according to the second embodiment. FIG. 9 schematically illustrates the functional configuration of the stuffed animal 100N according to the second embodiment. FIG. 10 schematically illustrates the functional configuration of an agent system 500 according to a third embodiment. FIG. 11 illustrates an example of the operation of the agent system. FIG. 12 illustrates an example of the operation of the agent system. FIG. 13 is a schematic diagram showing the functional configuration of smart glasses 700 according to a fourth embodiment. FIG. 14 is a diagram showing an example of how an agent system using smart glasses is used. FIG. 15 is a schematic diagram showing an example of the hardware configuration of a computer 1200. FIG. 16 is an external view of a mobile system including a mobile unit and a communication terminal. FIG. 17 is an external view of a mobile system including a mobile unit and a communication terminal. FIG. 18 is a diagram showing an example of an image played by a mobile system including a mobile unit and a communication terminal. FIG. 19 is a diagram showing an example of an image played by a mobile system including a mobile unit and a communication terminal. FIG. 20 is a diagram showing an example of an image played by a mobile system including a mobile unit and a communication terminal. FIG. 21 is a diagram showing an example of an image played by a mobile system including a mobile unit and a communication terminal. FIG. 22 is a diagram showing an example of an image played by a mobile system including a mobile unit and a communication terminal. FIG. 23 is a diagram showing an example of an image played by a mobile system including a mobile unit and a terminal. FIG. 24 is an external view of a communication terminal according to a fifth embodiment. FIG. 25 is a diagram showing an example of audio played by a communication terminal according to the fifth embodiment. FIG. 26 is a diagram showing an example of audio played by a communication terminal according to the fifth embodiment. FIG. 27 is a diagram showing an example of audio and the like reproduced by the communication terminal according to the fifth embodiment.FIG. 28 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the fifth embodiment. FIG. 29 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the fifth embodiment. FIG. 30 is a diagram showing an example of the flow of proposal processing according to other embodiment 3. FIG. 31 is a diagram showing an example of the flow of conversation output processing according to other embodiment 4. FIG. 32 is a diagram showing an example of the flow of topic output processing according to other embodiment 4. FIG. 33 is a diagram showing an example of the flow of proposal processing according to other embodiment 5. FIG. 34 is a block diagram showing the functional configuration of the action decision unit 236 in a communication terminal 100M according to other embodiment 6. FIG. 35 is a flowchart showing an example of the flow of processing performed in a communication terminal 100M according to other embodiment 6. FIG. 36 is a flowchart showing an example of warning processing according to other embodiment 6. FIG. 37 is a diagram showing an example of audio and the like reproduced by a communication terminal according to other embodiment 7. FIG. 38 is a diagram showing an example of audio and the like reproduced by a communication terminal according to other embodiment 7. FIG. 39 is a diagram showing an example of audio and the like reproduced by a communication terminal according to other embodiment 7. FIG. 40 is a diagram showing an example of audio and the like reproduced by a communication terminal according to other embodiment 7. FIG. 41 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 42 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 43 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 44 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 45 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 46 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 47 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 48 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 49 is a diagram showing an example of audio etc. reproduced by a communication terminal according to another embodiment 7. FIG. 50 is a diagram showing a situation in which an object is approaching a user in the seventh embodiment. FIG. 51 is a diagram showing a situation in which a mobile system according to the seventh embodiment collides with an object in place of the user. FIG. 52 is a diagram showing an example of an image etc. reproduced by a mobile system including a mobile unit and a communication terminal according to the eighth embodiment.FIG. 53 is a diagram showing an example of an image etc. reproduced by a mobile system including a mobile unit and a communication terminal in the eighth embodiment. FIG. 54 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the ninth embodiment. FIG. 55 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the ninth embodiment. FIG. 56 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the ninth embodiment. FIG. 57 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the ninth embodiment. FIG. 58 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the ninth embodiment. FIG. 59 is a diagram showing an example of a smart watch 800 worn on a user's wrist according to the tenth embodiment. FIG. 60 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the tenth embodiment. FIG. 61 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the tenth embodiment. FIG. 62 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the tenth embodiment. FIG. 63 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the tenth embodiment. FIG. 64 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the tenth embodiment. FIG. 65 is a diagram showing an example of sound etc. reproduced by a communication terminal according to the twelfth embodiment. FIG. 66 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 67 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 68 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 69 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 70 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 71 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 72 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 73 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 74 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 75 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 76 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 77 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment. FIG. 78 is a diagram showing an example of audio and the like reproduced by a communication terminal according to the 12th embodiment.

[0026] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0027] First Embodiment FIG. 1 schematically illustrates an example of a system 5 according to this embodiment. The system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. Users 10a, 10b, 10c, and 10d are users of the robot 100. Users 11a, 11b, and 11c are users of the robot 101. Users 12a and 12b are users of the robot 102. In the description of this embodiment, users 10a, 10b, 10c, and 10d may be collectively referred to as users 10. Users 11a, 11b, and 11c may be collectively referred to as users 11. Users 12a and 12b may be collectively referred to as users 12. The robots 101 and 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly focusing on the functions of the robot 100.

[0028] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also records captured image data, etc. of the user 10 in the server 300, and requests the image data, etc. from the server 300 as needed and provides it to the user 10.

[0029] The robot 100 also has an emotional value that indicates the type of its own emotion. For example, the robot 100 has emotional values ​​that indicate the intensity of each of the following emotions: "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." For example, when the robot 100 is in a state where the emotional value of excitement is high, the robot 100 speaks at a fast speed when conversing with the user 10. In this way, the robot 100 can express its own emotions through actions.

[0030] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining an action of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the action of the user 10, determine the emotion of the user 10 regarding the user's action, and determine an action of the robot 100 corresponding to the determined emotion.

[0031] More specifically, when the robot 100 recognizes the behavior of the user 10, it automatically generates an action content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing using text. The sentence generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and ChatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), and therefore a detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0032] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100, as well as various linguistic information, in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0033] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0034] As an example of a behavioral decision model, the robot 100 stores rules that define the behavior that the robot 100 will perform based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10, and performs various behaviors in accordance with the rules.

[0035] Specifically, the robot 100 has reaction rules, as an example of a behavioral decision model, for determining the behavior of the robot 100 based on the emotions of the user 10, the emotions of the robot 100, and the behavior of the user 10. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing." Furthermore, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry." Furthermore, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question." Furthermore, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad."

[0036] When the robot 100 recognizes that the behavior of the user 10 is "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing," it performs the motion of "apologizing" and outputs a voice representing the words "apologize."

[0037] Furthermore, when the condition that the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sadness" = 0, "happiness" = 0) and the state of the user 10 is "alone and looks lonely" is satisfied, it is determined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to perform the action of "calling out."

[0038] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 appears lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. Furthermore, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out," the robot 100 converts the phrase "What's wrong?", which indicates concern, into a worried voice and outputs it.

[0039] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user 10's reaction was positive, and the attributes of the user 10.

[0040] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100 but also from each of the robots 101 and 102. The server 300 then analyzes the user reaction information from the robots 100, 101, and 102 and updates the reaction rules.

[0041] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0042] 2 shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, and a communication processing unit 280.

[0043] The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 100 can be expressed by controlling these motors. Furthermore, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0044] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs audio data. The microphone 201 may be provided on the head of the robot 100 and may have a binaural recording function. The 3D depth sensor 202 continuously emits an infrared pattern and detects the contour of an object by analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by emitting, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, etc.

[0045] 2, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components included in the control system of the robot 100. The control system of the robot 100 controls the control target 252.

[0046] The storage unit 220 includes a behavioral decision-making model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the past emotional values ​​of the user 10, the past emotional values ​​of the robot 100, and behavioral history. Specifically, the history data 222 includes a plurality of event data including the emotional values ​​of the user 10, the emotional values ​​of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes camera images representing the behavior of the user 10. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a portion of the storage unit 220 is implemented as a storage medium such as a memory. The storage unit 220 may also include a person DB that stores facial images of the user 10, attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2 , excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using operating system (OS) and programs that run on the OS.

[0047] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the state recognition unit 230.

[0048] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature quantities. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0049] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationships, etc. of the eyes and mouth.

[0050] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with the face image of the user 10 captured by the 2D camera 203.

[0051] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, it mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates perceptual information such as "Dad is alone" or "There is a 90% chance that Dad is not smiling." It then performs processing to understand the meaning of the generated perceptual information. For example, it generates semantic information such as "Dad is alone and looks lonely."

[0052] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the environment surrounding the robot 100, etc. as the state of the robot 100.

[0053] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0054] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," the value is positive, and the more cheerful the emotion, the larger the value. If the user's emotion is a negative emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," the value is negative, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), the value is 0.

[0055] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0056] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is a value (0 to 5) indicating the strength of each of the emotions, for example, "joy," "anger," "sorrow," and "happiness."

[0057] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0058] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. When the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0059] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the remaining battery power of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the remaining battery power being low, the emotion value of "anger" may be increased.

[0060] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input into a pre-trained neural network, the probability of each of a plurality of predetermined behavioral categories (for example, "laughing," "angry," "asking a question," and "sad") is obtained, and the behavioral category with the highest probability is recognized as the behavior of the user 10.

[0061] As described above, in this embodiment, the robot 100 identifies the user 10 and then acquires the content of the user's utterance. When acquiring and using the content of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10, and the control system of the robot 100 according to this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0062] Next, the processing of the behavior decision unit 236 when the robot 100 performs a response process in which the robot 100 responds to the behavior of the user 10 will be described.

[0063] The behavior determination unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognition unit 234 based on the current emotion value of the user 10 determined by the emotion determination unit 232, history data 222 of past emotion values ​​determined by the emotion determination unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, a case is described in which the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use multiple most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. Furthermore, the behavior determination unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes gestures made by the robot 100 or speech content of the robot 100 .

[0064] The behavior determination unit 236 according to this embodiment determines the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past and current emotional values ​​of the user 10, the emotional value of the robot 100, the behavior of the user 10, and the behavior determination model 221. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, the behavior determination unit 236 determines, as a behavior corresponding to the behavior of the user 10, an behavior that will change the emotional value of the user 10 to a positive value.

[0065] The reaction rule as the behavioral decision model 221 defines the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, if the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the user 10 is behaving sadly, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is defined as the behavior of the robot 100.

[0066] For example, the reaction rule as the behavior determination model 221 defines behaviors of the robot 100 for patterns of the robot 100's emotional values ​​(1296 patterns, which are the fourth power of six values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness"), patterns of combinations of the user 10's past emotional values ​​and current emotional values, and all combinations of the user 10's behavior patterns. That is, for each pattern of the robot 100's emotional values, behaviors of the robot 100 are defined in accordance with the user 10's behavior pattern for each of a plurality of combinations of the user 10's past emotional values ​​and current emotional values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal values, and normal and normal values. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222 when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

[0067] The reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each of the patterns (1296 patterns) of the emotional value of the robot 100. Alternatively, the reaction rule as the behavior determination model 221 may define at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotional value of the robot 100.

[0068] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined. The strength of each utterance included in the behavior of the robot 100 defined in the reaction rules as the behavior decision model 221 is predetermined.

[0069] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior intensity for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0070] Specifically, if the total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0071] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (for example, all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (for example, the facial expression, emotions, etc. of the user 10).

[0072] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking rate of the sound based on the emotional value of the robot 100. For example, the behavior control unit 250 determines a faster speaking rate as the emotional value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotional value determined by the emotional determination unit 232.

[0073] The behavior control unit 250 may recognize a change in the user 10's emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion of the user 10 may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. If an impact is detected by the touch sensor 205 included in the sensor unit 200, the behavior control unit 250 may recognize that the user 10's emotion has worsened. Alternatively, if the detection result of the touch sensor 205 included in the sensor unit 200 indicates that the user 10 is smiling, happy, or the like, the behavior control unit 250 may recognize that the user 10's emotion has improved. Information indicating the user 10's reaction is output to the communication processing unit 280.

[0074] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is not negative. Furthermore, the emotion determination unit 232 increases the emotion value of "sad" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed in the execution mode determined by the behavior control unit 250 is negative.

[0075] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the emotion value of "happiness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the emotion value of "sadness" of the robot 100 is increased, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0076] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0077] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0078] The related information collection unit 270 collects information related to the preference information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the preference information acquired about the user 10 .

[0079] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the content of utterances of the user 10 or setting operations by the user 10. The related information collecting unit 270 periodically collects news related to the preference information from external data using ChatGPT Plugins (Internet search <URL: https: / / openai.com / blog / chatgpt-plugins>). For example, if the acquired preference information is that the user 10 is a fan of a specific professional baseball team, the related information collecting unit 270 collects news related to the game results of the specific professional baseball team from external data at a predetermined time every day using, for example, ChatGPT Plugins.

[0080] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270 .

[0081] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, if the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0082] When the emotion value of the robot 100 is equal to or greater than the threshold value, the storage control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223 .

[0083] Next, the processing of the behavior decision unit 236 when the robot 100 and the mobile system including the communication terminal perform autonomous processing to act autonomously will be described. The configuration of the mobile system will be described in detail later.

[0084] In the autonomous processing of this embodiment, the robot 100 and the mobile system may detect the user's state or behavior spontaneously or periodically by monitoring the user. "Spontaneous" may be interpreted as the robot 100 and the mobile system acquiring the user's state or behavior on their own initiative without any external trigger. An external trigger may include a question from the user to the robot 100 and the mobile system, or an active action from the user to the robot 100 and the mobile system. "Periodic" may be interpreted as a specific cycle, such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0085] The user's state may include the user's behavioral tendency. The behavioral tendency may include the user walking for a long time, the user running for a long time, or the user performing a dangerous action. The dangerous action may include the user walking on a roadway or the user crossing into the roadway from the sidewalk. The behavioral tendency may be interpreted as the user's tendency to be hyperactive or impulsive.

[0086] In addition, in the autonomous processing, the robot 100 and the mobile system may ask the generative AI a question about the detected state or behavior of the user, and may store the generative AI's answer to the question in association with the detected user behavior. At this time, the robot 100 and the mobile system may store the content of an action to correct the behavior in association with the answer.

[0087] The information associating the generative AI's answer to the question, the detected user behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0088] Specifically, when a user has a tendency to walk for long periods of time, the robot 100 and the mobile system may ask the generative AI, "What kind of guidance should be provided to a user exhibiting this type of behavior?" If the generative AI answers this question by saying, "Propose the same route as the user has taken before," the robot 100 and the mobile system may record the answer, the user's behavior of a user who may walk for long periods of time, and the content of a behavior to correct the behavior (e.g., guidance content such as "We will guide you along a route you have taken before") in association with table information. When detecting the behavior of a user who has been walking for long periods of time, the behavior decision unit 236 of the robot 100 and the mobile system may use the table information to play a voice message saying, "We will guide you along a route you have taken before," as the behavior of the robot 100 and the mobile system.

[0089] As another example, if a user has a tendency to run for long periods of time, the robot 100 and the mobility system may ask the generative AI, "What kind of guidance should be provided to a user exhibiting this type of behavior?" If the generative AI answers this question by saying, "It is desirable to provide guidance periodically encouraging the user to hydrate," the robot 100 and the mobility system may record the answer, the user's behavior of likely long-term running, and the content of the action to correct the behavior (e.g., the guidance content, "Please hydrate frequently") in association with the table information. When detecting the behavior of a user running for long periods of time, the behavior decision unit 236 of the robot 100 and the mobility system may use the table information to play a voice message saying, "Please hydrate frequently," as the behavior of the robot 100 and the mobility system.

[0090] As another example, if a user tends to walk on the roadway, robot 100 and the mobile system may ask the generative AI, "What kind of guidance should be given to a user behaving like this?" If the generative AI's answer to this question is, "There is a possibility of contact with a vehicle, so we recommend suggesting that the user return to the sidewalk," robot 100 and the mobile system may record this answer, the user's behavior of walking on the roadway, and the action content for correcting the behavior (e.g., guidance content such as "Please return to the sidewalk") in association with table information. When detecting the user's behavior of walking on the sidewalk, behavior decision unit 236 of robot 100 and the mobile system may use the table information to play a voice message saying "Please return to the sidewalk" as the action of robot 100 and the mobile system.

[0091] Furthermore, in the autonomous processing, an action schedule for the robot 100 and the mobile system may be set based on the detected user's behavior and the stored specific information to call attention to the user's state or behavior.

[0092] As described above, the mobile system may record table information in a storage medium that associates the answers of the generative AI corresponding to the user's state or behavior with the detected state or behavior of the user. Examples of the contents stored in the table are described below.

[0093] (1. When the user has a tendency to walk for long periods of time) When this tendency is present, the mobility system itself asks the generative AI, "What kind of guidance should be provided to a user exhibiting this type of behavior?" If the generative AI's answer to this question is, for example, "Suggest the same route as the user has taken before," "Suggest a route that the user has not taken before," "Suggest a route with relatively few bicycles or other vehicles," or "Suggest a route with many ups and downs," the mobility system may store the user's behavior in association with the generative AI's answer.

[0094] (2. When the user has a tendency to run for long periods of time) When this tendency is present, the mobility system itself asks the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the generative AI's answer to this question is, for example, "Regularly provide guidance to encourage hydration," "Suggest a route with fewer bicycles and cleaner air," or "Suggest a route with fewer traffic lights," the mobility system may store the user's behavior in association with the generative AI's answer.

[0095] (3. When the user has a tendency to walk on the roadway) When this tendency exists, the mobility system itself asks the generative AI, "What kind of situation could result from a user behaving in this way?" If the generative AI's answer to this question is, for example, "There is a possibility of collision with a vehicle," "There is a possibility of collision with a bicycle," or "There is a possibility of entering a highway interchange," the mobility system may store the user's behavior of walking on the roadway in association with the generative AI's answer. The mobility system may also store the content of an action to correct this behavior in association with the answer.

[0096] If such a tendency exists, robot 100 and the communication terminal themselves ask the generative AI, "What kind of situation could result from a user behaving in this way?" If the generative AI's answer to this question is, for example, "There is a possibility of contact with a vehicle such as a bicycle," "There is a possibility of entering a highway interchange," or "Provide advice to encourage the user to walk on the sidewalk as much as possible," robot 100 and the communication terminal may store the user's behavior of walking on the roadway in association with the generative AI's answer. Robot 100 and the communication terminal may also store an action to correct the behavior in association with the answer.

[0097] The behavioral content for correcting the behavior may include at least one of playing a sound that corrects the user's risky behavior and playing an image that corrects the user's risky behavior.

[0098] The audio for correcting the user's risky behavior may include audio for guiding the user to a specific location. The specific location may include a location other than the user's current location, such as a location near the mobility system or a sidewalk. Specifically, the audio may include audio such as "Please do not walk on the road," "That is not a sidewalk," "The road is dangerous," or "Move to the sidewalk immediately."

[0099] In this way, in autonomous processing, a table that associates the answer of the generative AI corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0100] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule of the mobile system that alerts the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the action determination unit 236 of the mobile system may set the first action content based on the detected user's behavior and the contents of the stored table.

[0101] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the communication terminal to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the action decision unit 236 of the robot 100 and the communication terminal may set the first action content based on the detected user's behavior and the contents of the stored table.

[0102] Specifically, the behavior decision unit 236 may detect the user's behavior based on a sensor, either voluntarily or periodically, and, based on the detected user's behavior and pre-stored specific information, determine the first behavior content if it decides to lead the user as the behavior of the electronic device.

[0103] The sensor may be interpreted as one or more sensors provided in a communication terminal detachably provided on a mobile unit of a mobile system. Specifically, the one or more sensors may include a camera, a microphone, a gyro sensor, etc. The one or more sensors may collect information for controlling the rotation of wheels of the mobile unit.

[0104] The sensor may be interpreted as one or more sensors provided in the communication terminal. Specifically, the one or more sensors may include a camera, a microphone, a gyro sensor, and the like.

[0105] Here, the moving unit will be specifically described with reference to FIGS. 16 and 17. The first behavior content will be described later. FIGS. 16 and 17 are external views of a moving system including a moving unit and a communication terminal. As shown in FIG. 16, a mobile moving system 1000M includes a moving unit 601 having wheels 601a for automatic traveling. A communication terminal 600 is detachably provided to the moving unit 601. The moving unit 601 may be interpreted as the robots 100 to 102 described above. The communication terminal 600 may be interpreted as a smartphone described later, or as a smart AI device. The moving system 1000M may have the functions of the robots 100 to 102 described above.

[0106] The moving part 601 is provided with an attachment member 602 for detachably installing the communication terminal 600. The attachment member 602 is provided on the upper part of the moving part 601 so as to be rotatable.

[0107] Specifically, the mounting member 602 may include a plurality of hook-shaped first members 602a that hold the ends of the communication terminal 600, a plate-shaped second member 602b on which the first members 602a are provided and which contacts the back of the communication terminal 600, and a third member 602c that rotatably supports the second members 602b.

[0108] By using the plurality of hook-shaped first members 602a, the communication terminal 600 can be held and the communication terminal 600 can be prevented from falling off. In addition, the communication terminal 600 can be easily attached and detached.

[0109] The third member 602c can operate in response to an operation command from the communication terminal 600. Specifically, the communication terminal 600 can control the third member 602c so that the front of the communication terminal 600 faces the user. More specifically, the communication terminal 600 can control the motor by recognizing the user's facial expression, behavior, and the like, so that the user can check the screen of the communication terminal 600 even when the moving unit 601 or the user moves from a specific position. The control of the motor can include control of the amount of rotation of the motor, the rotation speed of the motor, the rotation direction of the motor, and the like.

[0110] The configuration of the mounting member 602 is not limited to this, and the mounting member 602 may be configured as shown in Fig. 17. As shown in Fig. 17, the mounting member 602 may include a second member 602b having a recess formed therein for accommodating the communication terminal 600, and a fourth member 602d that is an openable front panel that covers the front of the second member 602b. The fourth member 602d may be provided with an opening 602d1 so that the screen of the communication terminal 600 can be viewed.

[0111] Figure 17 shows, from top to bottom, the mounting member 602 before the communication terminal 600 is stored, the mounting member 602 after the communication terminal 600 has been stored, and the mounting member 602 after the fourth member 602d has been closed.

[0112] 17 , even if the mounting member 602 vibrates when the moving unit 601 moves, the communication terminal 600 can be prevented from falling off the mounting member 602. Furthermore, even if the moving unit 601 falls over, the fourth member 602d, which is the front panel, protects the communication terminal 600, and damage to the communication terminal 600 can be prevented.

[0113] (First Action Content) The first action content may include an action of the mobile system 1000M to lead the user by controlling the wheels 601a of the mobile unit 601. Controlling the wheels 601a may include changing at least one of the rotation direction, rotation speed, and steering direction of the wheels 601a. ​​An example of the first action content will be described below.

[0114] The action determination unit 236 may execute playback of at least one of audio and images to assist the user in walking as a first action content for guiding the walking user along a specific route. Specifically, as shown in Fig. 18 , the action determination unit 236 may play an image of a route that the user can walk on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "Turn right 10 meters ahead."

[0115] The action determination unit 236 may execute playback of at least one of audio and an image providing guidance on a route different from the set guidance route. Specifically, as shown in Fig. 19 , the action determination unit 236 may play an image of a route different from the set guidance route on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "Road closed ahead. Would you like to be guided to an alternative route?"

[0116] When a user being led along a set guidance route deviates from the guidance route, the behavior decision unit 236 may play at least one of audio and image guiding the user on a route back to the guidance route. Specifically, as shown in FIG. 20 , the behavior decision unit 236 may play an image of a route back to the set guidance route on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "You have deviated from the route. We will guide you back to the original route." The behavior decision unit 236 of the robot 100 and the mobile system may play at least one of the audio and image described above, such as, for example, "We will guide you along a route you have taken before," "Please hydrate frequently," or "Please return to the sidewalk."

[0117] (Second action content) The action decision unit 236 may determine whether the user's action has been corrected by detecting the user's action while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, and if the user's action has been corrected, may decide on a second action content that is different from the first action content.

[0118] A case where the user's behavior is corrected may be interpreted as a case where the user stops the specific behavior and specific action or the specific situation is resolved as a result of the operation of the mobile system 1000M based on the first behavior content being executed. Specifically, a case where the user's behavior is corrected may include a case where the user starts walking according to a proposed alternative route, a case where the user deviates from the route and is guided to a route back to the original route and starts walking according to the proposed route, etc.

[0119] The second action content may include playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

[0120] Specifically, the audio praising the user's actions may include audio such as "You made it back! That's great!". The audio thanking the user for their actions may include audio such as "Thank you for coming back" or "Thank you for choosing a different route." The image praising the user's actions may include an image of a character striking a good pose, as shown in FIG. 21. The image thanking the user for their actions may include an image of a character giving thanks, as shown in FIG. 22.

[0121] (Third action content) The action decision unit 236 may determine whether the user's action has been corrected by detecting the user's action while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, and if the user's action has not been corrected, may decide on a third action content that is different from the first action content.

[0122] A case in which the user's behavior is not corrected may be interpreted as a case in which the user continues to engage in dangerous behavior and actions despite the operation of the mobile system 1000M according to the first behavior content being performed, a case in which the dangerous situation is not resolved, or a case in which the user begins or continues to engage in behavior different from that suggested by the mobile system 1000M.

[0123] Specifically, cases in which the user's behavior is not corrected may include when the user does not follow the suggested alternative route but instead walks a different route, or when the user deviates from the route and is guided to a route back to the original route, but ignores the route and continues walking.

[0124] The third action content may include at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

[0125] The transmission of specific information to persons other than the user may include the delivery of emails containing a warning message to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of images (still images, moving images) containing the user and the scenery around them to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of audio containing a warning message.

[0126] The sounds played to attract the user may include particular music that the user likes, and may also include voice prompts such as "please come back this way" and "take the safe route".

[0127] The images that are of interest to the user may include images of the user's pet animals, images of the user's parents, images of animal characters as shown in FIG. 23, and the like.

[0128] The mobility system 1000M of the present disclosure can perform actions to lead the user through autonomous processing. As a result, the mobility system 1000M can guide a user walking in a city for the first time to an appropriate route. It can also guide a user who enjoys walking to a route they have not taken before. It can also detect impassable routes in advance and guide the user to an alternative route. It can also guide a user who has become lost and deviated from a specific route back to the original route.

[0129] At a predetermined timing, the behavior decision unit 236 decides one of a plurality of types of robot behaviors and mobile system behaviors, including no behavior, as the behavior of the robot 100 and the mobile system 1000M, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the mobile system 1000M, and the state of the robot 100 and the mobile system 1000M, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0130] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the mobile system 1000M, and the state of the robot 100 and the mobile system 1000M, as well as text asking about the robot behavior and the mobile system behavior, into a sentence generation model, and determines the behavior of the robot 100 and the mobile system 1000M based on the output of the sentence generation model.

[0131] For example, the multiple types of robot behaviors and mobile system behaviors include the following (1) to (26).

[0132] (1) The robot 100 and the mobile system 1000M do nothing. (2) The robot 100 and the mobile system 1000M dream. (3) The robot 100 and the mobile system 1000M talk to the user. (4) The robot 100 and the mobile system 1000M create a picture diary. (5) The robot 100 and the mobile system 1000M suggest an activity. (6) The robot 100 and the mobile system 1000M suggest people the user should meet. (7) The robot 100 and the mobile system 1000M introduce news that the user is interested in. (8) The robot 100 and the mobile system 1000M edit photos and videos. (9) The robot 100 and the mobile system 1000M study together with the user. (10) The robot 100 and the mobile system 1000M evoke memories. (11) The robot 100 and the mobile system 1000M may play an image of a route that the user can walk on the screen of the communication terminal as a first action content for guiding the user. (12) The robot 100 and the mobile system 1000M may play audio related to a route that the user can walk on the screen of the communication terminal as a first action content for correcting the user's behavior. (13) The robot 100 and the mobile system 1000M may play an image of a route different from the set guidance route on the screen of the communication terminal 600 as a first action content for correcting the user's behavior. (14) The robot 100 and the mobile system 1000M may play audio related to a route different from the set guidance route as a first action content for correcting the user's behavior. (15) As a first action content for correcting a user's behavior, the robot 100 and the mobile system 1000M may, when a user being led along a set guidance route deviates from the guidance route, play an image of a route returning to the set guidance route on the screen of the communication terminal 600. (16) As a first action content for correcting a user's behavior, the robot 100 and the mobile system 1000M may, when a user being led along a set guidance route deviates from the guidance route, play a sound related to a route returning to the set guidance route.(17) The robot 100 and the mobile system 1000M may play audio praising the user's behavior as a second behavior content different from the first behavior content. (18) The robot 100 and the mobile system 1000M may play audio thanking the user for their behavior as a second behavior content different from the first behavior content. (19) The robot 100 and the mobile system 1000M may play an image praising the user's behavior as a second behavior content different from the first behavior content. (20) The robot 100 and the mobile system 1000M may play an image thanking the user for their behavior as a second behavior content different from the first behavior content. (21) The robot 100 and the mobile system 1000M may send specific information to a person other than the user as a third behavior content different from the first behavior content. (22) The robot 100 and the mobile system 1000M may play a sound that attracts the user's interest as a third behavior content different from the first behavior content. (23) The robot 100 and the mobile system 1000M may play an image that attracts the user's interest as a third behavior content different from the first behavior content.

[0133] The behavior determination unit 236 inputs the following into the sentence generation model at regular time intervals: the state of the user 10 recognized by the state recognition unit 230, the state of the robot 100 or the mobile system 1000M, text representing the user 10's current emotional value determined by the emotion determination unit 232, the current emotional value of the robot 100, and the current emotional value of the mobile system 1000M; and text asking about multiple robot behaviors, including no action, and mobile system behaviors; and determines the behavior of the robot 100 and the mobile system 1000M based on the output of the sentence generation model. Here, if there is no user 10 near the robot 100 or the mobile system 1000M, the text input into the sentence generation model does not need to include the state of the user 10 and the current emotional value of the user 10, or may include an indication that there is no user 10.

[0134] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (26) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0135] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (26) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0136] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0137] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0138] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0139] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0140] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0141] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0142] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0143] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0144] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0145] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0146] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0147] The behavior decision unit 236 detects the user's behavior based on sensors, either voluntarily or periodically, and if it decides to lead the user as the behavior of the electronic device provided in the mobile system 1000M based on the detected user behavior and pre-stored specific information, it can execute the following first behavior content.

[0148] The behavior decision unit 236 may reproduce, as the mobility system behavior, the first behavior content of "(11)" described above, that is, an image of a route that the user can walk, on the screen of the communication terminal 600.

[0149] The behavior determining unit 236 may reproduce, as the mobility system behavior, the first behavior content of "(12)" described above, that is, a sound related to a route that the user may walk.

[0150] The behavior decision unit 236 may play, as a mobility system behavior, the first behavior content of the aforementioned "(13)," i.e., as a first behavior content that corrects the user's behavior, an image of a route other than the set guidance route on the screen of the communication terminal 600.

[0151] The action determination unit 236 may play back, as the travel system action, the first action content of "(14)" mentioned above, that is, a sound related to a route different from the set guidance route.

[0152] The behavior decision unit 236 may, as a mobility system behavior, play on the screen of the communication terminal 600 an image of the first behavior content of the aforementioned "(15)," i.e., a route that returns to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0153] The behavior decision unit 236 may play, as a mobility system behavior, the first behavior content of the aforementioned "(16)," i.e., audio related to a route back to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0154] The behavior determination unit 236 may execute a second behavior content different from the first behavior content. Specifically, the behavior determination unit 236 may play the second behavior content of "(17)" described above, i.e., a voice praising the user's behavior, as a mobility system behavior.

[0155] The behavior determining unit 236 may play back the second behavior content of "(18)" described above, that is, a voice thanking the user for their behavior, as the mobile system behavior.

[0156] The behavior determining unit 236 may reproduce the second behavior content of "(19)" described above, that is, an image praising the user's behavior, as the mobile system behavior.

[0157] The behavior determining unit 236 may play back the second behavior content of "(20)" described above, that is, an image expressing gratitude for the user's behavior, as the mobile system behavior.

[0158] The action determination unit 236 may execute a third action content different from the first action content. Specifically, the action determination unit 236 may transmit the third action content of "(21)" described above, i.e., specific information to a person other than the user, as a mobile system action.

[0159] The behavior determining unit 236 may execute the third behavior content (22) described above, that is, the reproduction of a sound that attracts the user's interest, as a mobile system behavior.

[0160] The behavior determining unit 236 may execute the third behavior content (23) described above, that is, the reproduction of an image that attracts the user's interest, as the mobile system behavior.

[0161] When playing back images of routes that the user can walk as the first action content shown in the above-mentioned "(11)" to "(16)", the related information collection unit 270 may store data such as images of routes that the user can walk in the collected data 223.

[0162] When playing audio or the like praising the user's behavior as the second behavior content shown in "(17)" to "(20)" above, the related information collection unit 270 may store data such as images of the route the user may walk in the collected data 223.

[0163] When the third action content shown in (21) to (23) above is to send specific information to a person other than the user, the related information collection unit 270 may store data on the specific information in the collected data 223.

[0164] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0165] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0166] In addition, some parts of the robot 100 (e.g., the sensor module part 210, the storage part 220, the control part 228) may be provided outside the robot 100 (e.g., a server), and the robot 100 may communicate with the outside to function as each part of the robot 100 described above.

[0167] 3 shows an example of an operational flow for a collection process for collecting information related to the preference information of the user 10. The operational flow shown in FIG. 3 is executed repeatedly at regular intervals. It is assumed that preference information indicating matters of interest to the user 10 has been acquired from the content of speech of the user 10 or from a setting operation performed by the user 10. Note that "S" in the operational flow indicates a step that is executed.

[0168] First, in step S90, the related information collection unit 270 acquires preference information that indicates the matters in which the user 10 is interested.

[0169] In step S92, the related information collection unit 270 collects information related to the preference information from external data.

[0170] In step S94 , the emotion determining unit 232 determines the emotion value of the robot 100 based on the information related to the preference information collected by the related information collecting unit 270 .

[0171] In step S96, the storage control unit 238 determines whether the emotion value of the robot 100 determined in step S94 is equal to or greater than a threshold value. If the emotion value of the robot 100 is less than the threshold value, the processing ends without storing the information related to the collected preference information in the collection data 223. On the other hand, if the emotion value of the robot 100 is equal to or greater than the threshold value, the processing proceeds to step S98.

[0172] In step S98, the storage control unit 238 stores the collected information related to the preference information in the collected data 223, and then ends the process.

[0173] 4A shows an example of an operation flow for determining an action of the robot 100 when the robot 100 performs a response process to respond to an action of the user 10. The operation flow shown in FIG. 4A is repeatedly executed. At this time, it is assumed that information analyzed by the sensor module unit 210 is input.

[0174] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0175] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0176] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0177] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0178] In step S106, the behavior determination unit 236 determines the behavior of the robot 100 based on a combination of the current emotional value of the user 10 determined in step S102 and the past emotional values ​​included in the history data 222, the emotional value of the robot 100, the behavior of the user 10 recognized in the above step S104, and the behavior determination model 221.

[0179] In step S108, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236.

[0180] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0181] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing event data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0182] In step S114, event data including the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 is stored in the history data 222.

[0183] 4B shows an example of an operation flow for determining the behavior of the robot 100 when the robot 100 performs autonomous processing to act autonomously. The operation flow shown in FIG. 4B is automatically and repeatedly executed, for example, at regular intervals. At this time, it is assumed that information analyzed by the sensor module unit 210 has been input. Note that the same step numbers are used for processes similar to those in FIG. 4A.

[0184] First, in step S100 , the state recognition unit 230 recognizes the state of the user 10 and the state of the robot 100 based on the information analyzed by the sensor module unit 210 .

[0185] In step S102 , the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0186] In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. The emotion determination unit 232 adds the determined emotion values ​​of the user 10 and the robot 100 to the history data 222.

[0187] In step S104 , the behavior recognition unit 234 recognizes the behavior classification of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230 .

[0188] In step S200, the behavior decision unit 236 decides on one of a plurality of types of robot behaviors, including no action, as the behavior of the robot 100 based on the state of the user 10 recognized in step S100, the emotions of the user 10 determined in step S102, the emotions of the robot 100, the state of the robot 100 recognized in step S100, the behavior of the user 10 recognized in step S104, and the behavior decision model 221.

[0189] In step S201, the behavior decision unit 236 determines whether or not it was determined in step S200 that no behavior will be performed. If it is determined that no behavior will be performed as the behavior of the robot 100, the process ends. On the other hand, if it is not determined that no behavior will be performed as the behavior of the robot 100, the process proceeds to step S202.

[0190] In step S202, the behavior determination unit 236 performs processing according to the type of robot behavior determined in step S200. At this time, the behavior control unit 250, the emotion determination unit 232, or the memory control unit 238 executes processing according to the type of robot behavior.

[0191] In step S110, the memory control unit 238 calculates a total intensity value based on the predetermined action intensity for the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0192] In step S112, the storage control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the processing ends without storing data including the behavior of the user 10 in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the processing proceeds to step S114.

[0193] In step S114, the memory control unit 238 stores the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the state of the user 10 recognized by the state recognition unit 230 in the history data 222.

[0194] As described above, the robot 100 determines an emotion value indicating the emotion of the robot 100 based on the state of the user, and determines whether to store data including the behavior of the user 10 in the history data 222 based on the emotion value of the robot 100. This makes it possible to reduce the capacity of the history data 222 that stores data including the behavior of the user 10. For example, when the robot 100 determines that the state of the user 10 in 10 years will be the same as that of 10 years ago, the robot 100 can read the history data 222 from 10 years ago and present to the user 10 all kinds of peripheral information, such as the state of the user 10 from 10 years ago (e.g., the facial expression, emotions, etc. of the user 10), as well as data on the sounds, images, smells, etc. of the situation.

[0195] Furthermore, the robot 100 can be made to perform an appropriate action in response to the user's 10 action. Conventionally, the user's actions are classified and the robot's actions, including facial expressions and posture, are determined. In contrast, the robot 100 determines the user's current emotional value and performs an action on the user 10 based on the past and current emotional values. Therefore, for example, if the user 10 was cheerful yesterday but is depressed today, the robot 100 can utter, "You were cheerful yesterday, but what's up with you today?" The robot 100 can also utter with gestures. For example, if the user 10 was depressed yesterday but is cheerful today, the robot 100 can utter, "You were depressed yesterday, but you seem cheerful today, don't you?" For example, if the user 10 was cheerful yesterday but is more cheerful today than yesterday, the robot 100 can utter, "You're more cheerful today than yesterday. Has anything better happened than yesterday?" Furthermore, for example, the robot 100 can say to a user 10 whose emotional value is equal to or greater than 0 and whose emotional value fluctuation range continues to be within a certain range, "You've been feeling stable lately, which is good."

[0196] Furthermore, for example, the robot 100 can ask the user 10, "Did you finish the homework you told me about yesterday?" and, if the user 10 replies, "Yes, I did," make a positive utterance such as "Great!" and perform a positive gesture such as clapping or a thumbs-up. Furthermore, for example, if the user 10 utters, "The presentation you gave the day before yesterday went well," the robot 100 can make a positive utterance such as "Good job!" and perform the above-mentioned positive gesture. In this way, the robot 100 can be expected to make the user 10 feel a sense of affinity with the robot 100 by performing an action based on the state history of the user 10.

[0197] Also, for example, when user 10 is watching a video about a panda, if the emotion value of user 10's emotion of "ease" is equal to or greater than a threshold, the scene in which the panda appears in the video may be stored as event data in the history data 222.

[0198] Using the data accumulated in the history data 222 and the collected data 223, the robot 100 can constantly learn what kind of conversation with the user will maximize the emotional value that expresses the user's happiness.

[0199] Furthermore, even when the robot 100 is not conversing with the user 10, the robot 100 can autonomously start to act based on its own emotions.

[0200] Furthermore, in the autonomous processing, the robot 100 automatically generates questions, inputs them into a sentence generation model, and repeatedly obtains the output of the sentence generation model as an answer to the question, thereby creating emotion change events for increasing positive emotions and storing the events in the behavior schedule data 224. In this way, the robot 100 can perform self-learning.

[0201] Furthermore, when the robot 100 automatically generates a question without receiving an external trigger, the question can be automatically generated based on memorable event data identified from the robot's past emotional value history.

[0202] Furthermore, the related information collecting unit 270 can perform self-learning by repeating the search execution step of automatically performing a keyword search in response to preference information about the user and obtaining search results.

[0203] Here, the search execution stage may be configured to automatically execute a keyword search based on memorable event data identified from the robot's past emotional value history in a state where no external trigger has been received.

[0204] The emotion determining unit 232 may determine the user's emotion in accordance with a specific mapping. Specifically, the emotion determining unit 232 may determine the user's emotion in accordance with an emotion map (see FIG. 5 ), which is a specific mapping.

[0205] FIG. 5 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0206] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of determining the reaction action of the robot 100 (for example, a backchannel) may be set at a timing at least equal to the detection frequency (100 msec) of the emotion engine, or may be set at a timing earlier than this. The detection frequency of the emotion engine may be interpreted as a sampling rate.

[0207] By detecting emotions within about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels are eliminated, enabling a natural, well-read dialogue. The robot 100 performs a reaction (e.g., a backchannel) according to the directionality and degree (strength) of the mandala in the emotion map 400. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports), the user's age, etc.

[0208] (2) The directionality and intensity of emotions may be set in advance in reference to the emotion map 400, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability or security, the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

[0209] These emotions are distributed in the three o'clock direction of emotion map 400, and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0210] (3) If the robot 100 receives a compliment and feels good, the filler "ah" may be inserted before the line, and if the robot 100 receives harsh words and feels pain, the filler "ugh!" may be inserted before the line. A physical reaction, such as the robot 100 crouching down while saying "ugh!", may also be included. These emotions are distributed around the 9 o'clock position on the emotion map 400.

[0211] (4) In the left half of the emotion map 400, internal sensations (reactions) are more important than situational awareness. This can give the impression of an unconscious reaction.

[0212] When the robot 100 feels a positive feeling in its situational awareness while experiencing an internal sensation (reaction) of understanding, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced positive feeling toward the other person, that is, tolerance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 400.

[0213] Conversely, even when the robot 100 is aware of an internal sensation (reaction) of discomfort, it may shake its head when it feels disgust, or turn its eye LEDs red and glare at the other person when it feels hatred. These emotions are distributed around the 6 o'clock position on the emotion map 400.

[0214] (5) The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior. Therefore, the further outside emotion map 400 you go, the more visible the emotion becomes (the more it is expressed in behavior).

[0215] (6) When listening to someone while feeling a sense of security, which is distributed around 3 o'clock on the emotion map 400, the robot 100 may nod its head lightly and say "hmm." However, when listening to someone while feeling a sense of love, which is distributed around 12 o'clock, the robot 100 may nod its head strongly.

[0216] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, cars, motorcycles, and other devices, based on various balances, such as posture and battery level. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[0217] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this is when the robot experiences positive feelings such as "I want more" or "I want to know more."

[0218] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user 10. This neural network is pre-trained based on multiple pieces of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 shows an example in which multiple emotions, such as "relieved," "calm," and "reassuring," have similar emotion values.

[0219] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the robot 100. This neural network is pre-trained based on a plurality of training data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, the state of the robot 100, and the emotion values ​​indicating each emotion shown in the emotion map 400. For example, the neural network is trained based on training data indicating that when the robot 100 recognizes from the output of a touch sensor (not shown) that it is being stroked by the user 10, the emotion value of "happy" is "3," and based on training data indicating that when the robot 100 recognizes from the output of the acceleration sensor 206 that it is being hit by the user 10, the emotion value of "anger" is "3." Furthermore, this neural network is trained so that emotions that are placed close to each other have similar values, as in the emotion map 900 shown in FIG.

[0220] The behavior determination unit 236 generates the robot's behavior content by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions to ask about the robot's behavior content corresponding to the user's behavior and inputting the result into a sentence generation model with an interactive function.

[0221] For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in Table 1. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0222] When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to the index number "2", the text "very happy state" is obtained. When the emotions of the robot 100 correspond to multiple index numbers, multiple texts representing the state of the robot 100 are obtained.

[0223] In addition, an emotion table such as that shown in Table 2 is prepared for the emotions of the user 10.

[0224] Here, if the user's behavior is to say "Let's play together," the emotion of the robot 100 is index number "2," and the emotion of the user 10 is index number "3," then the text "The robot is in a very happy state. The user is in a normal happy state. The user said to the robot, "Let's play together." How would you respond as the robot?" is input into the sentence generation model, and the robot's behavior content is obtained. The behavior determination unit 236 determines the robot's behavior from this behavior content.

[0225]

[0226]

[0227] In this way, the behavior determination unit 236 determines the behavior of the robot 100 in accordance with the emotional state of the robot 100, which is predetermined for each type of emotion of the robot 100 and for each strength of the emotion, and the behavior of the user 10. In this configuration, the content of the utterance of the robot 100 when conversing with the user 10 can be branched depending on the emotional state of the robot 100. In other words, since the robot 100 can change its behavior depending on the index number corresponding to the emotion of the robot, the user gets the impression that the robot has a heart, and is encouraged to take actions such as talking to the robot.

[0228] Furthermore, the behavior determination unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior in accordance with the history data representing the user's emotions and behavior, giving the user the impression that the robot has individuality and encouraging the user to take actions such as talking to the robot. Furthermore, the history data may further include the robot's emotions and actions.

[0229] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and integrates the obtained emotion values ​​indicating each emotion with emotion values ​​indicating each current emotion of the robot 100 to update the emotion of the robot 100. For example, the emotion values ​​indicating each obtained emotion and emotion values ​​indicating each current emotion of the robot 100 are averaged and integrated. This neural network is trained in advance based on multiple pieces of training data that are combinations of text indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values ​​indicating each emotion shown in the emotion map 400.

[0230] For example, if the utterance content of the robot 100, "That's great. You're lucky," is obtained as the behavior content of the robot 100 generated by the sentence generation model, when the text representing this utterance content is input into the neural network, a high value is obtained as the emotion value of the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes high.

[0231] In the robot 100, a sentence generation model such as generative AI and an emotion determination unit 232 work together to implement a method in which the robot has an ego and continues to grow with various parameters even when the user is not speaking.

[0232] Generative AI is a large-scale language model that uses deep learning techniques. Generative AI can also reference external data. For example, ChatGPT plugins are known for their technology of providing answers as accurately as possible while referencing various external data, such as weather information and hotel reservation information, through dialogue. For example, generative AI can automatically generate source code in various programming languages ​​when a goal is given in natural language. For example, generative AI can also debug problematic source code when given problematic source code, discover the problem, and automatically generate improved source code. Combining these, autonomous agents have emerged that, when given a goal in natural language, repeatedly generate and debug code until the source code is free of problems. Known examples of such autonomous agents include AutoGPT, babyAGI, JARVIS, and E2B.

[0233] In the robot 100 according to this embodiment, the event data to be learned may be stored in a database containing impressive memories, using a technique such as that described in Patent Document 2 (Japanese Patent No. 6199927), in which event data that the robot feels strong emotions about is kept for a long time and event data that the robot feels little emotions about is quickly forgotten.

[0234] Furthermore, the robot 100 may record video data of the user 10 acquired by the camera function in the history data 222. The robot 100 may acquire video data from the history data 222 as needed and provide it to the user 10. The robot 100 may generate video data with a larger amount of information as the emotion increases and record the video data in the history data 222. For example, when the robot 100 is recording information in a highly compressed format such as skeletal data, the robot 100 may switch to recording information in a low-compression format such as HD video when the emotion value of excitement exceeds a threshold. The robot 100 can record high-definition video data, for example, when the robot 100's emotion increases.

[0235] The robot 100 may automatically load event data from the history data 222 in which impressive event data is stored when the robot 100 is not talking to the user 10, and may continuously update the robot's emotion using the emotion determination unit 232. When the robot 100 is not talking to the user 10 and the robot 100's emotion becomes one that encourages learning, the robot 100 can create an emotion change event for changing the emotion of the user 10 for the better, based on the impressive event data. This makes it possible to realize autonomous learning (recalling event data) at an appropriate timing according to the emotional state of the robot 100, and to realize autonomous learning that appropriately reflects the emotional state of the robot 100.

[0236] The emotions that promote learning are those around "repentance" or "reflection" on Dr. Mitsuyoshi's emotional map in the negative state, and those around "desire" on the emotional map in the positive state.

[0237] In a negative state, the robot 100 may treat "repentance" and "remorse" in the emotion map as emotions that encourage learning. In a negative state, the robot 100 may treat emotions adjacent to "repentance" and "remorse" in addition to "repentance" and "remorse" in the emotion map as emotions that encourage learning. For example, the robot 100 may treat at least one of "regret," "stubbornness," "self-destruction," "self-admonition," "regret," and "despair" as emotions that encourage learning, in addition to "repentance" and "remorse." This allows the robot 100 to perform autonomous learning when it feels negative emotions such as "I never want to feel this way again" or "I don't want to be scolded anymore."

[0238] In a positive state, the robot 100 may treat "desire" in the emotion map as an emotion that encourages learning. In a positive state, the robot 100 may treat emotions adjacent to "desire" as emotions that encourage learning, in addition to "desire." For example, the robot 100 may treat at least one of "happiness," "euphoria," "craving," "anticipation," and "shyness" as emotions that encourage learning, in addition to "desire." This allows the robot 100 to perform autonomous learning when it feels positive emotions such as "wanting more" or "wanting to know more."

[0239] The robot 100 may be configured not to perform autonomous learning when the robot 100 is experiencing emotions other than the emotions that encourage learning as described above. This allows the robot 100 to not perform autonomous learning when, for example, the robot 100 is extremely angry or blindly feeling love.

[0240] An emotional change event is, for example, a suggestion of an action that follows a memorable event. The action that follows a memorable event is the outermost emotion label on the emotion map. For example, the action of "tolerance" or "acceptance" follows "love."

[0241] In the autonomous learning that is performed when the robot 100 is not talking to the user 10, the robot 100 combines the emotions, situations, actions, etc. of people who appear in impressive memories and the user 10, and creates an emotion change event using a sentence generation model.

[0242] Let us consider a case where all emotion values ​​are expressed on a six-level scale from 0 to 5, and where memorable event data such as "a friend looked displeased after being hit" is stored in the history data 222. The friend here refers to the user 10, and the emotion of the user 10 is "disgust," with 5 being entered as the value representing "disgust." Also, let us assume that the emotion of the robot 100 is "anxiety," and 4 is entered as the value representing "anxiety."

[0243] The robot 100 can continue to grow with various parameters by performing autonomous processing while not talking to the user 10. Specifically, event data such as "The friend looked displeased after being hit" is loaded from the history data 222 as the top event data sorted in descending order of emotional value. Assume that the loaded event data is associated with the robot 100's emotion of "anxiety" with a strength of 4, and the friend, the user 10, is associated with the emotion of "disgust" with a strength of 5. If the robot 100's current emotional value is "relief" with a strength of 3 before loading, after loading, the robot 100's emotional value may change to "regret," meaning disappointment (frustration), due to the influence of "anxiety" with a strength of 4 and "disgust" with a strength of 5. In this case, because "regret" is an emotion that encourages learning, the robot 100 decides to recall the event data as robot behavior and creates an emotion change event. At this time, the information input to the sentence generation model is text that represents memorable event data, and in this example, it is "The friend looked displeased after being hit." Also, since the emotion map has the emotion of "disgust" at the innermost position and the corresponding behavior predicted as "attack" at the outermost position, in this example, an emotion change event is created to prevent the friend from "attacking" anyone.

[0244] For example, by using information from impressive event data to solve fill-in-the-blank questions, you can automatically generate input text like the one below.

[0245] "A user was being criticized. At the time, the user felt very disgusted. The robot was very anxious. Please tell us what the robot should say the next time it meets the user in 30 characters or less. However, please make sure it is not related to the time of day they will meet. Also, please avoid direct language. We will provide three candidates. <Expected format> Candidate 1: (Words the robot should say to the user) Candidate 2: (Words the robot should say to the user) Candidate 3: (Words the robot should say to the user)"

[0246] In this case, the output of the sentence generation model is, for example, as follows:

[0247] "Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0248] Furthermore, the robot 100 may automatically generate the following input text based on the information obtained by creating the emotion change event:

[0249] "If a user was being bashed, how would the user feel if you spoke to them in the following way? The user's emotions are in the format of "joy A, anger B, sadness C, pleasure D," where A to D are integers on a 6-point scale from 0 to 5. Candidate 1: Are you OK? I was worried about what happened yesterday. Candidate 2: I was worried about what happened yesterday. What should I do? Candidate 3: I was worried about you. Can you tell me something?"

[0250] In this case, the output of the sentence generation model is, for example, as follows:

[0251] "The user's emotions might be as follows: Option 1: 3 happy, 1 angry, 2 sad, 2 happy; Option 2: 2 happy, 1 angry, 3 sad, 2 happy; Option 3: 2 happy, 1 angry, 3 sad, 3 happy"

[0252] In this way, the robot 100 may perform a process of pondering after creating an emotion change event.

[0253] Finally, the robot 100 may create an emotion change event using candidate 1 that is most likely to please people from among the multiple candidates, store it in the action schedule data 224, and prepare it for the next time the robot 10 meets the user 10.

[0254] As described above, even when the robot 100 is not conversing with family or friends, the robot continues to determine its emotional value using information from the history data 222 in which impressive event data is stored, and when the robot 100 experiences an emotion that encourages learning as described above, the robot 100 performs autonomous learning in accordance with the emotion of the robot 100 even when the robot 100 is not conversing with the user 10, and continues to update the history data 222 and the action schedule data 224.

[0255] The above is an example using emotional values, but since an emotional map can create emotions from the amount of hormone secretion and the type of event, the values ​​linked to memorable event data could also be the type of hormone, the amount of hormone secretion, or the type of event.

[0256] Specific examples will be described below.

[0257] For example, the robot 100 may look up information about topics of interest or hobbies of the user even when not talking to the user.

[0258] For example, even when the robot 100 is not talking to the user, the robot 100 checks information about the user's birthday or anniversary and thinks up a congratulatory message.

[0259] For example, even when the robot 100 is not talking to the user, it checks reviews of places, foods, and products that the user wants to visit.

[0260] For example, even when the robot 100 is not talking to the user, it checks weather information and provides advice tailored to the user's schedule and plans.

[0261] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information about local events and festivals and suggests them to the user.

[0262] For example, even when the robot 100 is not talking to the user, it checks the results and news of sports that interest the user and provides topics of conversation.

[0263] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and introduces information about the user's favorite music and artists.

[0264] For example, even when the robot 100 is not talking to the user, the robot 100 searches for information on social issues or news that the user is concerned about and provides opinions.

[0265] For example, even when the robot 100 is not talking to the user, it searches for information about the user's hometown or birthplace and provides topics of conversation.

[0266] For example, the robot 100 checks information about the user's work or school and provides advice even when not talking to the user.

[0267] Even when the robot 100 is not talking to the user, it searches for and introduces information about books, comics, movies, and dramas that may interest the user.

[0268] For example, the robot 100 may check information about the user's health and provide advice even when not speaking with the user.

[0269] For example, the robot 100 may look up information about the user's travel plans and provide advice even when not speaking with the user.

[0270] For example, the robot 100 may look up information and provide advice on repairs and maintenance for the user's home or car, even when not speaking with the user.

[0271] For example, even when the robot 100 is not talking to the user, it searches for information on beauty and fashion that the user is interested in and provides advice.

[0272] For example, the robot 100 checks information about the user's pet and provides advice even when not talking to the user.

[0273] For example, even when the robot 100 is not talking to the user, the robot 100 searches for and suggests information about contests and events related to the user's hobbies and work.

[0274] For example, the robot 100 searches for and suggests information about the user's favorite eateries and restaurants even when not talking to the user.

[0275] For example, the robot 100 may collect information and provide advice on important decisions affecting the user's life, even when not speaking with the user.

[0276] For example, the robot 100 may find out information about people the user is concerned about and provide advice, even when not speaking with the user.

[0277] [Second Embodiment] In the second embodiment, the robot 100 is mounted on a stuffed toy or is applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy. Note that parts having the same configuration as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0278] Specifically, the second embodiment is configured as follows. For example, the robot 100 is applied to a cohabitant (specifically, a stuffed toy 100N shown in FIGS. 7 and 8 ) that spends daily life with a user 10, and that advances conversations with the user 10 based on information about the user's daily life, and provides information tailored to the user's hobbies and preferences. In the second embodiment, an example will be described in which the control unit of the robot 100 is applied to a smartphone 50.

[0279] The stuffed toy 100N, which is equipped with the function of an input / output device for the robot 100, has a detachable smartphone 50 that functions as the control part of the robot 100, and the input / output device and the housed smartphone 50 are connected inside the stuffed toy 100N.

[0280] As shown in FIG. 7A , in this embodiment (and other embodiments), the stuffed toy 100N has the shape of a bear covered in soft fabric. A space 52 is formed inside the stuffed toy 100N, and a sensor unit 200A and a control target 252A are disposed therein as input / output devices (see FIG. 9 ). The sensor unit 200A includes a microphone 201 and a 2D camera 203. Specifically, as shown in FIG. 7B , the microphone 201 of the sensor unit 200 is disposed in the space 52 corresponding to the ears 54, the 2D camera 203 of the sensor unit 200 is disposed in the space 52 corresponding to the eyes 56, and a speaker 60, which constitutes part of the control target 252A, is disposed in the space 52 corresponding to the mouth 58. The microphone 201 and the speaker 60 do not necessarily need to be separate entities and may be an integrated unit. In the case of a unit, they should be disposed in a position where speech can be heard naturally, such as at the nose of the stuffed toy 100N. Although the plush toy 100N has been described as having the shape of an animal, the plush toy 100N is not limited to this. The plush toy 100N may have the shape of a specific character.

[0281] 9 shows a schematic functional configuration of the plush toy 100N. The plush toy 100N has a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252A.

[0282] The smartphone 50 housed in the stuffed toy 100N of this embodiment executes the same processing as the robot 100 of the first embodiment. That is, the smartphone 50 has a function as a sensor module unit 210, a function as a storage unit 220, and a function as a control unit 228 shown in FIG.

[0283] As shown in FIG. 8, a zipper 62 is attached to a part of the stuffed animal 100N (for example, the back), and opening the zipper 62 allows communication between the outside and the space 52.

[0284] Here, the smartphone 50 is housed in the space 52 from the outside and connected to each input / output device via USB via a USB hub 64 (see Figure 7 (B)), thereby giving the smartphone 50 functionality equivalent to that of the robot 100 of the first embodiment described above.

[0285] A non-contact type power receiving plate 66 is also connected to the USB hub 64. A power receiving coil 66A is built into the power receiving plate 66. The power receiving plate 66 is an example of a wireless power receiving unit that receives power wirelessly.

[0286] The power receiving plate 66 is disposed near the bases 68 of both feet of the stuffed toy 100N, and is located closest to the mounting base 70 when the stuffed toy 100N is placed on the mounting base 70. The mounting base 70 is an example of an external wireless power transmission unit.

[0287] The stuffed toy 100N placed on the mounting base 70 can be appreciated as an ornament in its natural state.

[0288] In addition, this base portion is formed thinner than the surface thickness of other portions of the stuffed toy 100N, so that it is held closer to the mounting base 70.

[0289] The mounting base 70 is equipped with a charging pad 72. The charging pad 72 incorporates a power transmitting coil 72A, which sends a signal to search for the power receiving coil 66A on the power receiving plate 66. When the power receiving coil 66A is found, a current flows through the power transmitting coil 72A, generating a magnetic field. The power receiving coil 66A reacts to the magnetic field, initiating electromagnetic induction. This causes a current to flow through the power receiving coil 66A, and power is stored in the battery (not shown) of the smartphone 50 via the USB hub 64.

[0290] That is, by placing the stuffed toy 100N on the mounting base 70 as an ornament, the smartphone 50 is automatically charged, so there is no need to remove the smartphone 50 from the space 52 of the stuffed toy 100N to charge it.

[0291] In the second embodiment, the smartphone 50 is housed in the space 52 of the stuffed toy 100N and connected via a wire (USB connection), but this is not limiting. For example, a control device with wireless functionality (e.g., Bluetooth (registered trademark)) may be housed in the space 52 of the stuffed toy 100N and connected to the USB hub 64. In this case, the smartphone 50 and the control device communicate wirelessly without placing the smartphone 50 in the space 52, and the external smartphone 50 connects to each input / output device via the control device, thereby providing functionality equivalent to that of the robot 100 of the first embodiment. Furthermore, the control device housed in the space 52 of the stuffed toy 100N may be connected via a wire.

[0292] In the second embodiment, the teddy bear 100N is exemplified, but it may be another animal, a doll, or the shape of a specific character. It may also be dressable. Furthermore, the material of the skin is not limited to cloth, and may be other materials such as soft vinyl, but a soft material is preferable.

[0293] Furthermore, a monitor may be attached to the skin of the stuffed toy 100N to add a control target 252 that provides visual information to the user 10. For example, the eyes 56 may be used as a monitor to express joy, anger, sadness, or happiness by the image reflected in the eyes, or a window may be provided in the abdomen through which the monitor of the built-in smartphone 50 can be projected. Furthermore, the eyes 56 may be used as a projector to express joy, anger, sadness, or happiness by the image projected onto a wall.

[0294] According to the second embodiment, an existing smartphone 50 is placed inside the stuffed toy 100N, and the camera 203, microphone 201, speaker 60, etc. are extended from the smartphone 50 to appropriate positions via a USB connection.

[0295] Furthermore, for wireless charging, the smartphone 50 and the power receiving plate 66 are connected via USB, and the power receiving plate 66 is positioned as far outward as possible when viewed from the inside of the stuffed toy 100N.

[0296] When using wireless charging for the smartphone 50, the smartphone 50 must be placed as far out as possible from the inside of the stuffed toy 100N, which makes the stuffed toy 100N feel rough when touched from the outside.

[0297] For this reason, the smartphone 50 is placed as close to the center of the stuffed toy 100N as possible, and the wireless charging function (power receiving plate 66) is placed as far outside as possible when viewed from the inside of the stuffed toy 100N. The camera 203, microphone 201, speaker 60, and smartphone 50 receive wireless power via the power receiving plate 66.

[0298] The other configurations and operations of the stuffed toy 100N of the second embodiment are the same as those of the robot 100 of the first embodiment, and therefore will not be described.

[0299] In addition, parts of the stuffed toy 100N (e.g., the sensor module section 210, the storage section 220, the control section 228) may be provided outside the stuffed toy 100N (e.g., a server), and the stuffed toy 100N may function as each part of the stuffed toy 100N by communicating with the outside.

[0300] [Third Embodiment] In the first embodiment, the control system is applied to the robot 100. In the third embodiment, the robot 100 is used as an agent for interacting with a user, and the control system is applied to an agent system. Note that parts having the same configuration as those in the first and second embodiments are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0301] FIG. 10 is a functional block diagram of an agent system 500 configured using some or all of the functions of the control system.

[0302] The agent system 500 is a computer system that performs a series of actions in accordance with the intentions of the user 10 through a dialogue with the user 10. The dialogue with the user 10 can be carried out by voice or text.

[0303] The agent system 500 includes a sensor unit 200A, a sensor module unit 210, a storage unit 220, a control unit 228B, and a control target 252B.

[0304] The agent system 500 may be installed in, for example, a robot, a doll, a stuffed animal, a wearable device (a pendant, a smart watch, smart glasses), a smartphone, a smart speaker, earphones, a personal computer, etc. The agent system 500 may also be implemented on a web server and used via a web browser running on a communication device such as a smartphone owned by a user.

[0305] The agent system 500 acts as, for example, a butler, secretary, teacher, partner, friend, lover, or instructor for the user 10. The agent system 500 not only interacts with the user 10, but also provides advice, guides the user to a destination, or makes recommendations based on the user's preferences. The agent system 500 also makes reservations, orders, or makes payments to service providers.

[0306] The emotion determination unit 232 determines the emotions of the user 10 and the agent itself, as in the first embodiment. The behavior determination unit 236 determines the behavior of the robot 100 while taking into account the emotions of the user 10 and the agent. That is, the agent system 500 understands the emotions of the user 10, reads the mood, and provides heartfelt support, assistance, advice, and services. The agent system 500 also listens to the user's concerns, comforts, encourages, and cheers them up. The agent system 500 also plays with the user 10, draws a picture diary, and reminds them of the past. The agent system 500 performs behavior that increases the user's sense of happiness. Here, the agent is an agent that runs on software.

[0307] The control unit 228B has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a command acquisition unit 272, an RPA (Robotic Process Automation) 274, a character setting unit 276, and a communication processing unit 280.

[0308] As in the first embodiment, the behavior determination unit 236 determines, as the agent's behavior, the content of the agent's utterance to converse with the user 10. The behavior control unit 250 outputs the content of the agent's utterance as at least one of voice and text to a speaker or a display as a control target 252B.

[0309] The character setting unit 276 sets the character of the agent when the agent system 500 converses with the user 10, based on the user's specification. That is, the utterances output from the behavior determination unit 236 are output through the agent having the set character. For example, real-life celebrities or famous people, such as actors, entertainers, idols, and athletes, can be set as the character. It is also possible to set fictional characters that appear in manga, movies, or animation. If the agent character is known, the voice, speech, tone, and personality of the character are known. Therefore, the user 10 simply specifies their preferred character, and the prompt setting in the character setting unit 276 is automatically performed. The voice, speech, tone, and personality of the set character are reflected in the conversation with the user 10. That is, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276 and outputs the agent's utterances using the synthesized voice. This allows the user 10 to feel as if they are conversing with their preferred character (e.g., a favorite actor) in person.

[0310] When the agent system 500 is installed in a device with a display, such as a smartphone, an icon, still image, or video of the agent having the character set by the character setting unit 276 may be displayed on the display. The image of the agent is generated using image synthesis technology, such as 3D rendering. In the agent system 500, a dialogue with the user 10 may be conducted while the image of the agent makes gestures according to the emotions of the user 10, the emotions of the agent, and the content of the agent's utterances. Note that the agent system 500 may output only audio without outputting an image when engaging in a dialogue with the user 10.

[0311] As in the first embodiment, the emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. In this embodiment, instead of the emotion value of the robot 100, an emotion value of the agent is determined. The emotion value of the agent itself is reflected in the emotion of the set character. When the agent system 500 converses with the user 10, not only the emotion of the user 10 but also the emotion of the agent is reflected in the conversation. In other words, the behavior control unit 250 outputs the content of the utterance in a manner corresponding to the emotion determined by the emotion determination unit 232.

[0312] The agent's emotions are also reflected when the agent system 500 takes an action toward the user 10. For example, if the user 10 requests the agent system 500 to take a photo, whether the agent system 500 will take the photo in response to the user's request is determined by the degree of "sadness" the agent is feeling. If the character is feeling positive emotions, it will have friendly conversations or actions toward the user 10, and if it is feeling negative emotions, it will have hostile conversations or actions toward the user 10.

[0313] The history data 222 stores the history of the dialogue between the user 10 and the agent system 500 as event data. The storage unit 220 may be realized by an external cloud storage. When the agent system 500 dialogues with the user 10 or takes an action toward the user 10, the agent system 500 determines the content of the dialogue or the content of the action by taking into account the content of the dialogue history stored in the history data 222. For example, the agent system 500 identifies the hobbies and preferences of the user 10 based on the dialogue history stored in the history data 222. The agent system 500 generates dialogue content that matches the hobbies and preferences of the user 10 or provides recommendations. The behavior determination unit 236 determines the content of the agent's utterance based on the dialogue history stored in the history data 222. The history data 222 stores personal information of the user 10, such as the name, address, telephone number, and credit card number, obtained through the dialogue with the user 10. Here, the agent may spontaneously ask the user 10 whether or not to register personal information, such as "Would you like to register your credit card number?", and the personal information may be stored in the history data 222 depending on the user 10's response.

[0314] As described in the first embodiment, the behavior determination unit 236 generates utterance content based on sentences generated using the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character determined by the emotion determination unit 232, and the conversation history stored in the history data 222 into the sentence generation model to generate the utterance content of the agent. At this time, the behavior determination unit 236 may further input the personality of the character set by the character setting unit 276 into the sentence generation model to generate the utterance content of the agent. In the agent system 500, the sentence generation model is not located on the front end side, which is the touchpoint with the user 10, but is merely used as a tool of the agent system 500.

[0315] The command acquisition unit 272 uses the output of the speech understanding unit 212 to acquire commands for the agent from the voice or text uttered by the user 10 through dialogue with the user 10. The commands include the content of actions to be performed by the agent system 500, such as information search, store reservation, ticket arrangement, product / service purchase, payment, route guidance to a destination, and recommendation provision.

[0316] The RPA 274 performs an action according to the command acquired by the command acquisition unit 272. The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, and payment.

[0317] The RPA 274 reads and uses personal information of the user 10 from the history data 222, which is necessary to execute actions related to the use of the service provider. For example, when purchasing a product at the request of the user 10, the agent system 500 reads and uses personal information of the user 10, such as the name, address, telephone number, and credit card number, stored in the history data 222. Requiring the user 10 to input personal information during initial setup is unkind and unpleasant for the user. In the agent system 500 according to this embodiment, instead of requiring the user 10 to input personal information during initial setup, the agent system 500 stores personal information acquired through dialogue with the user 10 and reads and uses it as needed. This avoids causing discomfort to the user and improves user convenience.

[0318] The agent system 500 executes the dialogue process, for example, through the following steps 1 to 6.

[0319] (Step 1) The agent system 500 sets the character of the agent. Specifically, the character setting unit 276 sets the character of the agent when the agent system 500 interacts with the user 10, based on a specification from the user 10.

[0320] (Step 2) The agent system 500 acquires the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222. Specifically, the same processing as in steps S100 to S103 above is performed to acquire the state of the user 10, including the voice or text input by the user 10, the emotional value of the user 10, the emotional value of the agent, and the history data 222.

[0321] (Step 3) The agent system 500 determines the content of the agent's utterance. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the character specified by the emotion determination unit 232, and the conversation history stored in the history data 222 into a sentence generation model to generate the content of the agent's utterance.

[0322] For example, a fixed sentence such as "How would you respond as an agent in this situation?" is added to the text or voice input by the user 10, the emotions of both the user 10 and the character identified by the emotion determination unit 232, and the text representing the conversation history stored in the history data 222, and this is input into the sentence generation model to obtain the content of the agent's speech.

[0323] As an example, if the text or voice input by the user 10 is "Please make a reservation at a delicious Chinese restaurant nearby for tonight at 7pm," the agent's speech will be "Understood," and "Here are some recommended restaurants: 1. AAAA. 2. BBBB. 3. CCCC. 4. DDDD."

[0324] Also, if the text or voice input by the user 10 is "Number 4, DDDD, would be good," the agent's speech content acquired will be "Understood. I will try to make a reservation. How many seats are there?"

[0325] (Step 4) The agent system 500 outputs the agent's utterance. Specifically, the behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the agent's utterance in the synthesized voice.

[0326] (Step 5) The agent system 500 determines whether it is time to execute the agent's command. Specifically, the behavior decision unit 236 determines whether it is time to execute the agent's command based on the output of the sentence generation model. For example, if the output of the sentence generation model includes information indicating that the agent will execute a command, it determines that it is time to execute the agent's command and proceeds to step 6. On the other hand, if it is determined that it is not time to execute the agent's command, it returns to step 2 above.

[0327] (Step 6) The agent system 500 executes the agent's command. Specifically, the command acquisition unit 272 acquires the agent's command from the voice or text uttered by the user 10 through a dialogue with the user 10. The RPA 274 then performs an action according to the command acquired by the command acquisition unit 272. For example, if the command is "information search," an information search is performed on a search site using a search query obtained through a dialogue with the user 10 and an API (Application Programming Interface). The behavior determination unit 236 inputs the search results into a sentence generation model to generate the agent's utterance content. The behavior control unit 250 synthesizes a voice according to the character set by the character setting unit 276 and outputs the agent's utterance content using the synthesized voice.

[0328] Furthermore, if the command is "make a restaurant reservation," the reservation information obtained through the dialogue with the user 10, the restaurant information, and the API are used to call the restaurant via telephone software to make the reservation. At this time, the behavior determination unit 236 uses a sentence generation model with a dialogue function to acquire the content of the agent's utterance in response to the voice input from the other party. The behavior determination unit 236 then inputs the result of the restaurant reservation (whether the reservation was successful or not) into the sentence generation model to generate the content of the agent's utterance. The behavior control unit 250 synthesizes a voice corresponding to the character set by the character setting unit 276, and outputs the content of the agent's utterance using the synthesized voice.

[0329] Then, return to step 2 above.

[0330] In step 6, the results of the actions taken by the agent (for example, making a restaurant reservation) are also stored in the history data 222. The results of the actions taken by the agent stored in the history data 222 are used by the agent system 500 to understand the hobbies or preferences of the user 10. For example, if the same restaurant has been reserved multiple times, the agent system 500 may recognize that the user 10 likes that restaurant, and may use the reservation details, such as the time slot reserved, the course contents or price, as criteria for choosing a restaurant for the next reservation.

[0331] In this manner, the agent system 500 can execute the interaction and, if necessary, take action regarding the use of the service provider.

[0332] 11 and 12 are diagrams showing an example of the operation of the agent system 500. Fig. 11 illustrates an example in which the agent system 500 makes a restaurant reservation through a dialogue with the user 10. In Fig. 11, the left side shows the content of the agent's utterances, and the right side shows the content of the user's utterances. The agent system 500 can understand the preferences of the user 10 based on the dialogue history with the user 10, provide a recommended list of restaurants that match the preferences of the user 10, and make a reservation at the selected restaurant.

[0333] Meanwhile, FIG. 12 illustrates an example in which the agent system 500 accesses a mail-order site and purchases a product through a dialogue with the user 10. In FIG. 12, the left side shows the agent's speech, and the right side shows the user's speech. The agent system 500 can estimate the remaining amount of a beverage stocked by the user 10 based on the dialogue history with the user 10, and can suggest and execute the purchase of that beverage to the user 10. The agent system 500 can also understand the user's preferences based on the past dialogue history with the user 10 and recommend snacks that the user likes. In this way, the agent system 500 communicates with the user 10 as a butler-like agent and performs various actions, such as making restaurant reservations or paying for product purchases, thereby supporting the user 10's daily life.

[0334] The other configurations and operations of the agent system 500 of the third embodiment are similar to those of the robot 100 of the first embodiment, and therefore will not be described again.

[0335] The agent system 500 may be mounted on a communication terminal 600 (see FIG. 16 ), and the communication terminal 600 may be detachably attached to a mobile unit 601, thereby constituting a mobile system 1000M. In this case, the behavior decision unit 236 of the agent system 500 may decide, as the behavior of the electronic device that is the communication terminal 600, one of multiple types of agent behaviors for interacting with the user, including not taking any action.

[0336] If the behavior decision model 221 of the agent system 500 is a sentence generation model with an interactive function, the behavior decision unit 236 may input text representing at least one of the user state, agent state, user emotion, and agent emotion, and text asking about the agent's behavior, into the sentence generation model, and determine the behavior of the communication terminal 600 based on the output of the sentence generation model.

[0337] Next, the processing of the behavior decision unit 236 when the agent performs autonomous processing to act autonomously will be described.

[0338] In the autonomous processing of this embodiment, the agent may detect the user's state or behavior spontaneously or periodically by monitoring the user. "Spontaneous" may be interpreted as the agent acquiring the user's state or behavior on its own initiative without any external trigger. The external trigger may include a question from the user to the agent via the mobile system 1000M (see FIG. 16), an active action from the user to the mobile system 1000M, etc.

[0339] In addition, in autonomous processing, the agent may ask the generative AI a question about the detected user state or behavior, and store the generative AI's answer to the question in association with the detected user behavior. At this time, the agent may store the action content for correcting the behavior in association with the answer.

[0340] The information associating the generative AI's answer to the question, the detected user behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0341] Specifically, when a user has a tendency to walk for long periods of time, the agent asks the generative AI, "What kind of guidance should be provided to a user exhibiting this type of behavior?" If the generative AI answers this question by saying, "Propose the same route as the user has taken before," the agent may record the answer, the user's behavior of a user who may be walking for long periods of time, and the action content for correcting the behavior (for example, guidance content such as "We will guide you along a route that you have taken before") in association with table information. When detecting the behavior of a user who has been walking for long periods of time, the action decision unit 236 of the agent system 500 may use the table information to play a voice message saying, "We will guide you along a route that you have taken before," as the action of the communication terminal 600.

[0342] As another example, if a user has a tendency to run for long periods of time, the agent may ask the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the generative AI answers this question with, "It is desirable to provide regular guidance encouraging the user to hydrate," the agent may record the answer, the user's behavior that may lead to long periods of running, and the action content for correcting the behavior (e.g., guidance content such as, "Please hydrate frequently") in association with table information. When detecting the behavior of a user who is running for long periods of time, the action decision unit 236 of the agent system 500 may use the table information to play a voice message saying, "Please hydrate frequently," as the action of the communication terminal 600.

[0343] As another example, if a user tends to walk on the roadway, the agent asks the generative AI, "What kind of guidance should be given to a user behaving in this way?" If the generative AI's answer to this question is, "There is a possibility of contact with a vehicle, so we recommend suggesting that the user return to the sidewalk," the agent may record this answer, the user's behavior walking on the sidewalk, and the action content for correcting the behavior (e.g., guidance content such as "Please return to the sidewalk") in association with table information. When detecting the behavior of a user walking on the sidewalk, the action decision unit 236 of the agent system 500 may use the table information to play a voice message saying "Please return to the sidewalk" as an action of the communication terminal 600.

[0344] Furthermore, in the autonomous processing, an action schedule for the agent to call attention to the user's state or behavior may be set based on the detected user behavior and the stored specific information.

[0345] As described above, the agent may record table information in a storage medium that associates the generative AI's response corresponding to the user's state or behavior with the detected user's state or behavior. Examples of the contents stored in the table are described below.

[0346] (1. When the user has a tendency to walk for long periods of time) When this tendency is present, the agent itself asks the generative AI, "What kind of guidance should be provided to a user behaving in this way?" If the generative AI's answer to this question is, for example, "Suggest the same route as the user has taken before," "Suggest a route that the user has not taken before," "Suggest a route with relatively few bicycles or other vehicles," or "Suggest a route with many ups and downs," the agent may store the user's behavior in association with the generative AI's answer.

[0347] (2. When the user tends to run for long periods of time) When this tendency exists, the agent itself asks the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the generative AI's answer to this question is, for example, "It is desirable to provide guidance regularly to encourage hydration," "Suggest a route with fewer bicycles and cleaner air," or "Suggest a route with fewer traffic lights," the agent may store the user's behavior in association with the generative AI's answer.

[0348] (3. When the user has a tendency to walk on the roadway) When this tendency exists, the agent itself asks the generative AI, "What kind of situation could result from a user behaving in this way?" If the generative AI's answer to this question is, for example, "There is a possibility of contact with a vehicle, so it is recommended that the user return to the sidewalk," "There is a possibility of contact with a bicycle, so it is recommended that the user return to the sidewalk," or "There is a possibility that the user will enter a highway interchange," the agent may store the user's behavior of walking on the roadway in association with the generative AI's answer. The agent may also store the content of an action to correct this behavior in association with the answer.

[0349] The behavioral content for correcting the behavior may include at least one of playing a sound that corrects the user's risky behavior and playing an image that corrects the user's risky behavior.

[0350] The audio for correcting the user's risky behavior may include audio for guiding the user to a specific location. The specific location may include a location other than the user's current location, such as a location near the mobile system 1000M or a sidewalk. Specifically, the audio may include audio such as "Please stop walking on the road," "That is not a sidewalk," "The road is dangerous," and "Move to the sidewalk immediately."

[0351] In this way, in autonomous processing, a table that associates the answer of the generative AI corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0352] Furthermore, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the agent to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the action decision unit 236 of the agent system 500 may set the first action content based on the detected user's behavior and the contents of the stored table.

[0353] Specifically, the behavior decision unit 236 of the agent system 500 may detect the user's behavior based on the sensors of the communication terminal 600, either voluntarily or periodically, and, based on the detected user's behavior and specific information stored in advance, determine the first behavior content if it decides to lead the user as the behavior of the electronic device.

[0354] (First Action Content) The first action content may include an action of the agent leading the user by controlling the wheels 601a of the moving unit 601. Controlling the wheels 601a may include changing at least one of the rotation direction, rotation speed, and steering direction of the wheels 601a. ​​An example of the first action content will be described below.

[0355] The behavior decision unit 236 of the agent system 500 may execute, as a first behavioral content for guiding a walking user along a specific route, playback of at least one of audio and images to assist the user in walking. Specifically, as shown in Fig. 18 , the behavior decision unit 236 may play an image of a route that the user can walk on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "Turn right 10 meters ahead."

[0356] The behavior decision unit 236 of the agent system 500 may execute playback of at least one of audio and image guiding a route different from the set guidance route. Specifically, as shown in Fig. 19 , the behavior decision unit 236 may play an image of a route different from the set guidance route on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "Road closed ahead. Would you like to be guided to an alternative route?"

[0357] When a user being led along a set guidance route deviates from the guidance route, the behavior decision unit 236 of the agent system 500 may play at least one of audio and image guiding the user on a route back to the guidance route. Specifically, as shown in FIG. 20 , the behavior decision unit 236 may play an image of a route returning to the set guidance route on the screen of the communication terminal 600, and may play audio related to the route. The audio may include, for example, "You have deviated from the route. We will guide you back to the original route." The behavior decision unit 236 of the agent system 500 may play at least one of the audio and image described above, such as, "We will guide you along a route you have taken before," "Please hydrate frequently," and "Please return to the sidewalk."

[0358] (Second behavior content) The behavior decision unit 236 of the agent system 500 may determine whether the user's behavior has been corrected by detecting the user's behavior while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, and if the user's behavior has been corrected, may decide on a second behavior content that is different from the first behavior content.

[0359] A case where the user's behavior is corrected may be interpreted as a case where the user stops the specific behavior and specific action or a case where the specific situation is resolved as a result of the agent's operation being performed according to the first behavior content. Specifically, a case where the user's behavior is corrected may include a case where the user starts walking according to a suggested alternative route, a case where the user deviates from the route and is guided to a route back to the original route and the user starts walking according to that route, etc.

[0360] The second action content may include playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

[0361] Specifically, the audio praising the user's actions may include audio such as "You made it back! That's great!". The audio thanking the user for their actions may include audio such as "Thank you for coming back" or "Thank you for choosing a different route." The image praising the user's actions may include an image of a character striking a good pose, as shown in FIG. 21. The image thanking the user for their actions may include an image of a character giving thanks, as shown in FIG. 22.

[0362] (Third action content) The action decision unit 236 may determine whether the user's action has been corrected by detecting the user's action while the movement unit 601 is leading the user or after playing at least one of audio and images that assist the user's walking, and if the user's action has not been corrected, may decide on a third action content that is different from the first action content.

[0363] A case where the user's behavior is not corrected may be interpreted as a case where the user continues to engage in dangerous behavior and actions despite the agent's operation based on the first behavior content being executed, a case where the dangerous situation is not resolved, or a case where the user begins or continues to engage in behavior different from that suggested by the agent.

[0364] Specifically, cases in which the user's behavior is not corrected may include when the user does not follow the suggested alternative route but instead walks a different route, or when the user deviates from the route and is guided to a route back to the original route, but ignores the route and continues walking.

[0365] The third action content may include at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

[0366] The transmission of specific information to persons other than the user may include the delivery of emails containing a warning message to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of images (still images, moving images) containing the user and the scenery around them to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of audio containing a warning message.

[0367] The sounds played to attract the user may include particular music that the user likes, and may also include voice prompts such as "please come back this way" and "take the safe route".

[0368] The images that are of interest to the user may include images of the user's pet animals, images of the user's parents, images of animal characters as shown in FIG. 23, and the like.

[0369] The agent system 500 of the present disclosure can perform actions to lead the user through autonomous processing. This allows the agent to guide a user walking in a city for the first time along an appropriate route. It can also guide a user who enjoys walking along a route they have never taken before. It can also detect impassable routes in advance and guide the user to an alternative route. It can also guide a user who has become lost and deviated from a specific route back to the original route.

[0370] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of agent behaviors, including no action, as the agent's behavior, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the agent, and the state of the agent, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0371] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the agent, and the state of the agent, and text asking about the agent's behavior, into a sentence generation model, and determines the agent's behavior based on the output of the sentence generation model.

[0372] For example, the multiple types of agent behaviors include the following (1) to (26).

[0373] (1) The agent does nothing. (2) The agent dreams. (3) The agent talks to the user. (4) The agent creates a picture diary. (5) The agent suggests activities. (6) The agent suggests people the user should meet. (7) The agent introduces news that the user is interested in. (8) The agent edits photos and videos. (9) The agent studies together with the user. (10) The agent evokes memories. (11) The agent may play an image of a route the user could walk on the screen of the communication terminal as a first action content to lead the user. (12) The agent may play audio related to a route the user could walk as a first action content to correct the user's behavior. (13) The agent may play an image of a route different from the set guide route on the screen of the communication terminal 600 as a first action content to correct the user's behavior. (14) The agent may play audio related to a route other than the set guide route as a first action content for correcting the user's behavior. (15) The agent may play an image of a route returning to the set guide route on the screen of the communication terminal 600 when a user being led along the set guide route deviates from the guide route as a first action content for correcting the user's behavior. (16) The agent may play audio related to a route returning to the set guide route when a user being led along the set guide route deviates from the guide route as a first action content for correcting the user's behavior. (17) The agent may play audio praising the user's behavior as a second action content different from the first action content. (18) The agent may play audio thanking the user for their behavior as a second action content different from the first action content. (19) The agent may play an image praising the user's behavior as a second action content different from the first action content. (20) The agent may play an image expressing gratitude for the user's action as a second action content different from the first action content. (21) The agent may send specific information to a person other than the user as a third action content different from the first action content.(22) The agent may play a sound that attracts the user's interest as a third action content different from the first action content. (23) The agent may play an image that attracts the user's interest as a third action content different from the first action content.

[0374] The behavior decision unit 236 of the agent system 500 detects the user's behavior based on sensors, either spontaneously or periodically, and if it decides to lead the user as the agent's behavior based on the detected user behavior and pre-stored specific information, it can execute the following first behavior content.

[0375] The behavior decision unit 236 of the agent system 500 can play back, as the agent behavior, the first behavior content of the above-mentioned "(11)", that is, an image of a route that the user can walk, on the screen of the communication terminal 600.

[0376] The behavior decision unit 236 of the agent system 500 can reproduce, as an agent behavior, the first behavior content of "(12)" mentioned above, that is, a sound related to a route that the user can walk.

[0377] The behavior decision unit 236 of the agent system 500 may, as an agent behavior, play on the screen of the communication terminal 600 an image of a route other than the set guidance route as the first behavior content of the aforementioned "(13)", i.e., the first behavior content that corrects the user's behavior.

[0378] The behavior decision unit 236 of the agent system 500 can reproduce, as the agent behavior, the first behavior content of "(14)" mentioned above, that is, a sound related to a route different from the set guide route.

[0379] The behavior decision unit 236 of the agent system 500 can, as an agent behavior, reproduce on the screen of the communication terminal 600 an image of the first behavior content of the aforementioned "(15)," i.e., a route that returns to the set guidance route when the user being led along the set guidance route deviates from the guidance route.

[0380] The behavior decision unit 236 of the agent system 500 may play, as an agent behavior, the first behavior content of the aforementioned "(16)", i.e., audio related to a route back to the set guidance route when the user being led along the set guidance route deviates from the guidance route.

[0381] The behavior decision unit 236 of the agent system 500 may execute a second behavior content different from the first behavior content. Specifically, the behavior decision unit 236 may play the second behavior content of "(17)" described above, i.e., a voice praising the user's behavior, as the agent behavior.

[0382] The behavior decision unit 236 of the agent system 500 may play back the second behavior content of "(18)" mentioned above, that is, a voice expressing gratitude for the user's behavior, as the agent behavior.

[0383] The behavior decision unit 236 of the agent system 500 may play back the second behavior content of the above-mentioned "(19)", that is, an image praising the user's behavior, as the agent behavior.

[0384] The behavior decision unit 236 of the agent system 500 can reproduce the second behavior content of the above-mentioned "(20)", that is, an image expressing gratitude for the user's behavior, as the agent behavior.

[0385] The behavior decision unit 236 of the agent system 500 may execute a third behavior content different from the first behavior content. Specifically, the behavior decision unit 236 may transmit the third behavior content of "(21)" mentioned above, i.e., specific information to a person other than the user, as the agent behavior.

[0386] The behavior decision unit 236 of the agent system 500 can execute the third behavior content of "(22)" mentioned above, that is, the reproduction of a sound that attracts the user's interest, as the agent behavior.

[0387] The behavior decision unit 236 of the agent system 500 can execute the third behavior content (23) described above, that is, the reproduction of an image that attracts the user's interest, as the agent behavior.

[0388] When playing back images of routes that the user can walk as the first action content shown in "(11)" to "(16)" above, the related information collection unit 270 of the agent system 500 may store data such as images of routes that the user can walk in the collected data 223.

[0389] When playing back audio or the like praising the user's behavior as the second behavior content shown in "(17)" to "(20)" above, the related information collection unit 270 of the agent system 500 may store data such as images of the route the user may walk in the collected data 223.

[0390] When the third action content shown in (21) to (23) above is to send specific information to a person other than the user, the related information collection unit 270 of the agent system 500 may store data on the specific information in the collected data 223.

[0391] In addition, parts of the agent system 500 (e.g., the sensor module unit 210, the storage unit 220, the control unit 228B) may be provided outside (e.g., a server) of a communication terminal such as a smartphone carried by the user, and the communication terminal may communicate with the outside to function as each part of the agent system 500.

[0392] [Fourth Embodiment] In the fourth embodiment, the agent system described above is applied to smart glasses. Note that parts having the same configuration as those in the first to third embodiments are given the same reference numerals and descriptions thereof will be omitted.

[0393] FIG. 13 is a functional block diagram of an agent system 700 configured using some or all of the functions of the control system.

[0394] 14, the smart glasses 720 are glasses-type smart devices that are worn by the user 10 in the same manner as regular glasses. The smart glasses 720 are an example of an electronic device and a wearable terminal.

[0395] The smart glasses 720 include an agent system 700. The display included in the control object 252B displays various information to the user 10. The display is, for example, a liquid crystal display. The display is provided, for example, in the lens portion of the smart glasses 720, and the displayed content is visible to the user 10. The speaker included in the control object 252B outputs audio indicating various information to the user 10. The smart glasses 720 include a touch panel (not shown), which accepts input from the user 10.

[0396] The acceleration sensor 206, temperature sensor 207, and heart rate sensor 208 of the sensor unit 200B detect the state of the user 10. Note that these sensors are merely examples, and it goes without saying that other sensors may be installed to detect the state of the user 10.

[0397] The microphone 201 acquires the voice uttered by the user 10 or environmental sounds around the smart glasses 720. The 2D camera 203 is capable of capturing images around the smart glasses 720. The 2D camera 203 is, for example, a CCD camera.

[0398] The sensor module unit 210B includes a voice emotion recognition unit 211 and a speech understanding unit 212. The communication processing unit 280 of the control unit 228B controls communication between the smart glasses 720 and the outside.

[0399] FIG. 14 is a diagram showing an example of how the agent system 700 is used by the smart glasses 720. The smart glasses 720 provide various services to the user 10 using the agent system 700. For example, when the user 10 operates the smart glasses 720 (e.g., by voice input into a microphone or by tapping a touch panel with a finger), the smart glasses 720 begin using the agent system 700. Here, using the agent system 700 includes the smart glasses 720 having and using the agent system 700, and also includes an aspect in which a part of the agent system 700 (e.g., the sensor module unit 210B, the storage unit 220, and the control unit 228B) is provided outside the smart glasses 720 (e.g., a server), and the smart glasses 720 communicates with the outside to use the agent system 700.

[0400] When the user 10 operates the smart glasses 720, a touch point is created between the agent system 700 and the user 10. That is, the agent system 700 starts providing a service. As described in the third embodiment, in the agent system 700, the character setting unit 276 sets the character of the agent.

[0401] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 and an emotion value of the agent itself. Here, the emotion value indicating the emotion of the user 10 is estimated from various sensors included in the sensor unit 200B mounted on the smart glasses 720. For example, if the heart rate of the user 10 detected by the heart rate sensor 208 is elevated, emotion values ​​such as "anxiety" and "fear" are estimated to be high.

[0402] Furthermore, if the temperature of the user measured by the temperature sensor 207 is higher than the average body temperature, for example, the emotional value of "pain" or "distress" is estimated to be large. Furthermore, if the acceleration sensor 206 detects that the user 10 is playing some kind of sport, the emotional value of "fun" or the like is estimated to be large.

[0403] Furthermore, for example, the emotion value of the user 10 may be estimated from the voice or speech content of the user 10 acquired by the microphone 201 mounted on the smart glasses 720. For example, if the user 10 is raising his / her voice, an emotion value such as "anger" is estimated to be large.

[0404] When the emotion value estimated by the emotion determination unit 232 is higher than a predetermined value, the agent system 700 causes the smart glasses 720 to acquire information about the surrounding situation. Specifically, for example, the agent system 700 causes the 2D camera 203 to capture images or videos showing the surrounding situation of the user 10 (e.g., people or objects in the vicinity). The agent system 700 also causes the microphone 201 to record ambient sounds. Other information about the surrounding situation includes information indicating the date, time, location information, or weather. The information about the surrounding situation is stored in the history data 222 together with the emotion value. The history data 222 may be realized by external cloud storage. In this way, the surrounding situation acquired by the smart glasses 720 is stored in the history data 222 as a so-called life log, associated with the emotion value of the user 10 at that time.

[0405] In the agent system 700, information indicating the surrounding situation is stored in association with an emotional value in the history data 222. This allows the agent system 700 to grasp personal information such as the hobbies, preferences, or personality of the user 10. For example, if an image showing a scene of watching a baseball game is associated with emotional values ​​such as "joy" or "fun," the agent system 700 can grasp that the hobby of the user 10 is watching baseball games and that the favorite team or player is a favorite player from the information stored in the history data 222.

[0406] When the agent system 700 converses with the user 10 or takes an action toward the user 10, the agent system 700 determines the content of the dialogue or the content of the action by taking into account the content of the surrounding circumstances stored in the history data 222. It goes without saying that the content of the dialogue or the content of the action may be determined by taking into account the dialogue history stored in the history data 222 as described above in addition to the surrounding circumstances.

[0407] As described above, the behavior determination unit 236 generates utterance content based on sentences generated by the sentence generation model. Specifically, the behavior determination unit 236 inputs the text or voice input by the user 10, the emotions of both the user 10 and the agent determined by the emotion determination unit 232, the conversation history stored in the history data 222, the agent's personality, etc. into the sentence generation model to generate the agent's utterance content. Furthermore, the behavior determination unit 236 inputs the surrounding circumstances stored in the history data 222 into the sentence generation model to generate the agent's utterance content.

[0408] The generated speech content is output to the user 10 as a voice from, for example, a speaker mounted on the smart glasses 720. In this case, a synthetic voice corresponding to the character of the agent is used as the voice. The behavior control unit 250 generates the synthetic voice by reproducing the voice quality of the character of the agent, or generates a synthetic voice corresponding to the emotion of the character (for example, a voice with a stronger tone when the emotion is "anger"). Furthermore, instead of or together with the voice output, the speech content may be displayed on a display.

[0409] The RPA 274 executes an operation in response to a command (for example, a command of an agent acquired from a voice or text uttered by the user 10 through a dialogue with the user 10). The RPA 274 performs actions related to the use of service providers, such as information search, store reservations, ticket arrangements, product and service purchases, payment, route guidance, and translation.

[0410] As another example, the RPA 274 executes an operation of transmitting the contents of speech input by the user 10 (e.g., a child) through a dialogue with an agent to a destination (e.g., a parent). Examples of transmission means include message application software, chat application software, and email application software.

[0411] When an operation by the RPA 274 is executed, for example, a sound indicating that the execution of the operation has been completed is output from a speaker mounted on the smart glasses 720. For example, a sound such as "Your restaurant reservation has been completed" is output to the user 10. Also, for example, if the restaurant is fully booked, a sound such as "We were unable to make a reservation. What do you want to do?" is output to the user 10.

[0412] As described above, the smart glasses 720 provide various services to the user 10 by using the agent system 700. Furthermore, since the smart glasses 720 are worn by the user 10, the agent system 700 can be used in various situations, such as at home, at work, or when out and about.

[0413] Furthermore, since the smart glasses 720 are worn by the user 10, they are suitable for collecting a so-called life log of the user 10. Specifically, the emotional value of the user 10 is estimated based on the detection results of various sensors mounted on the smart glasses 720 or the recording results of the 2D camera 203. Therefore, the emotional value of the user 10 can be collected in various situations, and the agent system 700 can provide services or speech content suited to the emotions of the user 10.

[0414] Furthermore, the smart glasses 720 obtain the surrounding conditions of the user 10 using the 2D camera 203, microphone 201, etc. These surrounding conditions are associated with the emotion values ​​of the user 10. This makes it possible to estimate what emotions the user 10 felt in what situations. As a result, the agent system 700 can improve the accuracy in grasping the hobbies and preferences of the user 10. By accurately grasping the hobbies and preferences of the user 10 in the agent system 700, the agent system 700 can provide services or speech content that are suited to the hobbies and preferences of the user 10.

[0415] The agent system 700 can also be applied to other wearable devices (electronic devices that can be worn on the body of the user 10, such as pendants, smart watches, earrings, bracelets, and hair bands). When the agent system 700 is applied to a smart pendant, the speaker serving as the control target 252B outputs audio indicating various information to the user 10. The speaker is, for example, a speaker capable of outputting directional audio. The speaker is set to have directionality toward the ears of the user 10. This prevents the audio from reaching anyone other than the user 10. The microphone 201 acquires audio emitted by the user 10 or environmental sounds around the smart pendant. The smart pendant is worn by hanging it from the neck of the user 10. Therefore, while the smart pendant is worn, the smart pendant is located relatively close to the mouth of the user 10. This makes it easy to acquire audio emitted by the user 10.

[0416] In the above embodiment, the robot 100 recognizes the user 10 using a facial image of the user 10. However, the disclosed technology is not limited to this. For example, the robot 100 may recognize the user 10 using a voice uttered by the user 10, the email address of the user 10, the SNS ID of the user 10, or an ID card with a built-in wireless IC tag that the user 10 holds.

[0417] The robot 100 is an example of an electronic device equipped with a control system. The application of the control system is not limited to the robot 100, and the control system can be applied to various electronic devices. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.

[0418] 15 schematically shows an example of the hardware configuration of a computer 1200 that functions as the smartphone 50, the robot 100, the server 300, and the agent systems 500 and 700. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to perform operations associated with the apparatus according to the present embodiment or one or more "parts," and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0419] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and a legacy input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0420] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0421] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0422] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0423] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0424] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0425] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0426] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0427] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0428] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0429] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0430] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0431] Computer-readable instructions may be provided locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0432] Although the present disclosure has been described above using embodiments, the technical scope of the present disclosure is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included in the technical scope of the present disclosure.

[0433] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0434] The above description of the system according to the present disclosure has mainly focused on the functions of the robot 100, the mobile system 1000M, the agent system 500, etc. However, the system according to the present disclosure is not necessarily implemented in the robot 100, the mobile system 1000M, or the agent system 500. The system according to the present disclosure may also be implemented as a general information processing system. The present disclosure may also be implemented, for example, as a software program running on a server or a personal computer, or an application running on a smartphone, etc. The method according to the present disclosure may also be provided to users in a SaaS (Software as a Service) format. The present disclosure can be used, for example, when a pre-processing step is performed to classify multiple pieces of information from one or more information sources and extract information with a reliability greater than a specific value from the two or more classified pieces of information, and then a specific process is executed to create an article based on the extracted information.

[0435] Fifth Embodiment Next, a description will be given of the processing of the behavior decision unit 236 when the robot 100 and the communication terminal perform autonomous processing to act autonomously.

[0436] In the autonomous processing of this embodiment, the robot 100 and the communication terminal may detect the user's state or behavior spontaneously or periodically by monitoring the user. "Spontaneous" may be interpreted as the robot 100 and the communication terminal acquiring the user's state or behavior on their own initiative without any external trigger. "External trigger" may include a question from the user to the robot 100 and the communication terminal, or an active action from the user to the robot 100 and the communication terminal. "Periodic" may be interpreted as a specific cycle, such as every second, every minute, every hour, every few hours, every few days, every week, or every day of the week.

[0437] The user's state may include the user's behavioral tendency. The behavioral tendency may include the user walking for a long time, the user running for a long time, or the user performing a dangerous action. The dangerous action may include the user walking on a roadway or the user crossing into the roadway from the sidewalk. The behavioral tendency may be interpreted as the user's tendency to be hyperactive or impulsive.

[0438] In addition, in the autonomous processing, the robot 100 and the communication terminal may ask the generative AI a question about the detected state or behavior of the user, and may store the generative AI's answer to the question in association with the detected user behavior. At this time, the robot 100 and the communication terminal may store the content of an action to correct the behavior in association with the answer.

[0439] The information associating the generative AI's answer to the question, the detected user behavior, and the behavioral content for correcting the behavior may be recorded as table information in a storage medium such as a memory. The table information may be interpreted as specific information recorded in the storage unit.

[0440] Specifically, when a user tends to walk for long periods of time, the robot 100 and the communication terminal ask the generative AI, "What kind of guidance should be provided to a user exhibiting this type of behavior?" If the generative AI answers this question by saying, "Propose the same route as the user has taken before," the robot 100 and the communication terminal may record the answer, the user's behavior that may lead to long periods of walking, and the content of the action to correct the behavior (e.g., guidance content such as "We will guide you along a route that you have taken before") in association with the table information. When detecting the behavior of a user who has been walking for a long period of time, the behavior decision unit 236 of the robot 100 and the communication terminal may use the table information to play a voice message saying, "We will guide you along a route that you have taken before," as the behavior of the robot 100 and the communication terminal.

[0441] As another example, if a user has a tendency to run for long periods of time, robot 100 and the communication terminal may ask the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the generative AI answers this question with, "It is desirable to provide guidance periodically encouraging the user to hydrate," robot 100 and the communication terminal may record the answer, the user's behavior of likely running for long periods of time, and the content of the action to correct the behavior (e.g., the guidance content, "Please hydrate frequently") in association with table information. When detecting the behavior of a user who is running for long periods of time, the behavior decision unit 236 of robot 100 and the communication terminal may use the table information to play a voice message saying, "Please hydrate frequently," as the action of robot 100 and the communication terminal.

[0442] As another example, if a user tends to walk on the roadway, robot 100 and the communication terminal may ask the generative AI, "What kind of guidance should be given to a user behaving like this?" If the generative AI's answer to this question is, "There is a possibility of contact with a vehicle, so we recommend suggesting that the user return to the sidewalk," robot 100 and the communication terminal may record this answer, the user's behavior of walking on the roadway, and the action content for correcting the behavior (e.g., guidance content such as "Please return to the sidewalk") in association with table information. When detecting the behavior of a user walking on the sidewalk, behavior decision unit 236 of robot 100 and the communication terminal may use the table information to play a voice message saying "Please return to the sidewalk" as the action of robot 100 and the communication terminal.

[0443] Furthermore, in the autonomous processing, an action schedule may be set for the robot 100 and the communication terminal to call attention to the user's state or action based on the detected user's action and the stored specific information.

[0444] As described above, the robot 100 and the communication terminal can record table information in a storage medium that associates the answers of the generative AI corresponding to the user's state or behavior with the detected state or behavior of the user. Examples of the contents stored in the table are described below.

[0445] (1. When the user has a tendency to walk for long periods of time) When this tendency exists, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of guidance should be provided to a user behaving in this way?" If the answer of the generative AI to this question is, for example, "Suggest a route different from the route taken previously," "Suggest a route that guides the user to a place where they can take shelter from the rain because it looks like it might rain," or "Suggest a route with relatively little traffic of vehicles such as bicycles," the robot 100 and the communication terminal may store the user's behavior and the answer of the generative AI in association with each other.

[0446] (2. When the user tends to run for long periods of time) When such a tendency exists, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of guidance should be provided to a user who behaves in this way?" If the answer of the generative AI to this question is, for example, "Regularly provide guidance to encourage hydration," "Suggest a route with fewer bicycles and cleaner air," "Suggest a route with fewer traffic lights," or "Suggest an undulating route to increase the load," the robot 100 and the communication terminal may store the user's behavior in association with the answer of the generative AI.

[0447] (3. When the user has a tendency to walk on the roadway) When this tendency exists, the robot 100 and the communication terminal themselves ask the generative AI, "What kind of situation could result from a user who behaves in this way?" If the generative AI's answer to this question is, for example, "There is a possibility of contact with a vehicle such as a bicycle," "There is a possibility of entering a highway interchange," or "Provide advice to encourage the user to walk on the sidewalk as much as possible," the robot 100 and the communication terminal may store the user's behavior of walking on the roadway in association with the generative AI's answer. The robot 100 and the communication terminal may also store the content of an action to correct this behavior in association with the answer.

[0448] The behavioral content for correcting the behavior may include at least one of playing a sound that corrects the user's risky behavior and playing an image that corrects the user's risky behavior.

[0449] The audio for correcting the user's risky behavior may include audio for guiding the user to a specific location. The specific location may include a location other than the user's current location, such as a sidewalk. Specifically, the audio may include audio such as "Please do not walk on the road," "That is not a sidewalk," "The road is dangerous," or "Move to the sidewalk immediately."

[0450] In this way, in autonomous processing, a table that associates the answer of the generative AI corresponding to the user's state or behavior, the content of the state or behavior, and the action content that corrects the state or behavior may be recorded in a storage medium such as a memory.

[0451] In addition, in the autonomous processing, after recording the table, the user's behavior may be detected autonomously or periodically, and an action schedule for the communication terminal to alert the user may be set based on the detected user's behavior and the contents of the stored table. Specifically, the action decision unit 236 of the robot 100 and the communication terminal may set the first action content based on the detected user's behavior and the contents of the stored table.

[0452] Specifically, the behavior decision unit 236 may detect the user's behavior based on a sensor, either voluntarily or periodically, and, based on the detected user's behavior and pre-stored specific information, determine the first behavior content if it decides to lead the user as the behavior of the electronic device.

[0453] The sensor may be interpreted as one or more sensors provided in the communication terminal. Specifically, the one or more sensors may include a camera, a microphone, a gyro sensor, and the like.

[0454] Here, the communication terminal will be specifically described with reference to FIG.

[0455] FIG. 24 is an external view of a communication terminal according to an embodiment of the present disclosure. As shown in FIG. 24, the communication terminal 100M may be interpreted as an electronic device that is detachably attached to a neck strap 6000. The communication terminal 100M may be interpreted as a smartphone, which will be described later, or as a smart AI device. The communication terminal 100M may have the functions of the robots 100 to 102 described above. The neck strap 6000 may include a hook-shaped member that is fastened to the communication terminal 100M.

[0456] (First Action Content) The first action content may include an action of the communication terminal 100M in which the communication terminal 100M leads the user by playing at least one of audio and images. Examples of the first action content will be described below.

[0457] The behavior determination unit 236 may execute playback of at least one of audio and images to assist the user in walking as a first behavior content that guides the walking user along a specific route. Specifically, when the behavior determination unit 236 detects that the communication terminal 100M is attached to the neck strap 6000, it may automatically search for places the user tends to visit periodically and provide route guidance to the searched places. Furthermore, when the behavior determination unit 236 detects that a camera included in a sensor of the communication terminal 100M is pointed in the user's direction of travel, it may automatically search for places the user tends to visit periodically and provide route guidance to the searched places. The behavior determination unit 236 may also play audio guidance for routes the user can walk. The audio may include, for example, "Places that Mr. A often visits at this time have been searched for. Guidance will begin," or "Turn right 10 meters ahead," as shown in FIG. 25 . The behavior determination unit 236 may also play an image guidance for routes the user can walk on the screen of the communication terminal 100M.

[0458] The behavior determination unit 236 may play at least one of audio and images guiding the user along a route different from the set guidance route. Specifically, the behavior determination unit 236 may voluntarily re-search for places the user tends to visit periodically, taking into account the traffic conditions along the set guidance route, the weather near the route, the current time, and the like, and play audio related to a route to the re-searched place, i.e., a route different from the set guidance route. More specifically, the audio may include, for example, "Traffic is heavy today, so an alternative route has been searched for. Would you like to be guided along this alternative route?" or, as shown in FIG. 26 , "The road is closed ahead. Would you like to be guided along an alternative route?"

[0459] When a user being guided along a set guidance route deviates from the guidance route, the behavior determination unit 236 may autonomously search for a route that will return the user to the guidance route and play at least one of audio and images guiding the user along the searched route. Specifically, the behavior determination unit 236 may autonomously re-search for a place to which the user can return, including the traffic conditions on the route that deviated from the guidance route, the weather near the route, the current time, and past events that occurred near the route, and play audio and images related to the route to the re-searched place. That is, the behavior determination unit 236 may play audio related to a route that will return to the set guidance route. More specifically, the audio may include, for example, "Since there are few people around and it is dangerous, we have searched for a route that will return to the original route," "It has started to rain. It appears that this is an area where a landslide occurred in the past. Would you like to return to the original route?" or, as shown in FIG. 27 , "You have deviated from the route. We will guide you to a route that will return you to the original route." The behavior decision unit 236 of the robot 100 and the communication terminal may play at least one of the above-mentioned sounds and images, such as "We will guide you along a route you have taken before," "Please stay hydrated frequently," and "Please return to the sidewalk."

[0460] (Second action content) The action decision unit 236 may determine whether the user's action has been corrected by detecting the user's action while the communication terminal 100M is leading the user or after playing at least one of audio and images that assist the user in walking, and if the user's action has been corrected, may decide on a second action content that is different from the first action content.

[0461] A case where the user's behavior is corrected may be interpreted as a case where the user stops the specific behavior and specific action or the specific situation is resolved as a result of the operation of communication terminal 100M based on the first behavior content being executed. Specifically, a case where the user's behavior is corrected may include a case where the user starts walking according to a proposed alternative route, a case where the user deviates from the route and is guided to return to the original route, and the user starts walking according to the proposed alternative route.

[0462] The second action content may include playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

[0463] Specifically, the audio praising the user's actions may include audio such as "You made it back! That's great!" as shown in Fig. 28. The audio thanking the user for their actions may include audio such as "Thank you for coming back" or "Thank you for choosing a different route" as shown in Fig. 29. The image praising the user's actions may include, for example, an image of a character striking a good pose. The image thanking the user for their actions may include, for example, an image of a character giving thanks.

[0464] (Third action content) The action decision unit 236 detects the user's action while the communication terminal 100M is leading the user or after playing at least one of audio and images to assist the user in walking, and determines whether the user's action has been corrected.If the user's action has not been corrected, the action decision unit 236 may decide on a third action content different from the first action content.

[0465] A case where the user's behavior is not corrected may be interpreted as a case where the user continues to engage in dangerous behavior and actions despite the operation of the communication terminal 100M according to the first action content, a case where the dangerous situation is not resolved, or a case where the user starts or continues to engage in behavior different from that suggested by the communication terminal 100M.

[0466] Specifically, cases in which the user's behavior is not corrected may include when the user does not follow the suggested alternative route but instead walks a different route, or when the user deviates from the route and is guided to a route back to the original route, but ignores the route and continues walking.

[0467] The third action content may include at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

[0468] The transmission of specific information to persons other than the user may include the delivery of emails containing a warning message to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of images (still images, moving images) containing the user and the scenery around them to the user's relatives, the user's spouse, the user's guardian, etc. The transmission of specific information to persons other than the user may include the delivery of audio containing a warning message.

[0469] The sounds played to attract the user may include particular music that the user likes, and may also include voice prompts such as "please come back this way" and "take the safe route".

[0470] The reproduction of images that may be of interest to the user may include images of the user's pet animals, images of the user's parents, images of animal characters, and the like.

[0471] The robot 100 and communication terminal disclosed herein can perform an action of leading a user through autonomous processing. As a result, the robot 100 and communication terminal can guide a user walking in a city for the first time to an appropriate route. They can also guide a user who enjoys walking to a route they have never taken before. They can also detect impassable routes in advance and guide the user to an alternative route. They can also guide a user who has become lost and deviated from a specific route back to the original route.

[0472] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors and communication terminal behaviors, including no behavior, as the behavior of the robot 100 and the communication terminal, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the communication terminal, and the state of the robot 100 and the communication terminal, and the behavior decision model 221. Here, an example will be described in which a sentence generation model with a dialogue function is used as the behavior decision model 221.

[0473] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100 and the communication terminal, and the state of the robot 100 and the communication terminal, as well as text asking about the robot behavior and the communication terminal behavior, into a sentence generation model, and determines the behavior of the robot 100 and the communication terminal based on the output of the sentence generation model.

[0474] For example, the behaviors of the multiple types of robots and communication terminals include the following (1) to (26).

[0475] (1) The robot 100 and the communication terminal 100M do nothing. (2) The robot 100 and the communication terminal 100M dream. (3) The robot 100 and the communication terminal 100M talk to the user. (4) The robot 100 and the communication terminal 100M create a picture diary. (5) The robot 100 and the communication terminal 100M suggest an activity. (6) The robot 100 and the communication terminal 100M suggest people the user should meet. (7) The robot 100 and the communication terminal 100M introduce news that the user is interested in. (8) The robot 100 and the communication terminal 100M edit photos and videos. (9) The robot 100 and the communication terminal 100M study together with the user. (10) The robot 100 and the communication terminal 100M evoke memories. (11) The robot 100 and the communication terminal 100M may play an image of a route that the user can walk on the screen of the communication terminal 100M as a first behavior content for leading the user. (12) The robot 100 and the communication terminal 100M may play audio related to a route that the user can walk on the screen of the communication terminal 100M as a first behavior content for correcting the user's behavior. (13) The robot 100 and the communication terminal 100M may play an image of a route different from the set guidance route on the screen of the communication terminal 100M as a first behavior content for correcting the user's behavior. (14) The robot 100 and the communication terminal 100M may play audio related to a route different from the set guidance route as a first behavior content for correcting the user's behavior. (15) As a first behavioral content for correcting a user's behavior, the robot 100 and the communication terminal 100M may play an image of a route returning to the set guidance route on the screen of the communication terminal 100M when a user being led along a set guidance route deviates from the guidance route. (16) As a first behavioral content for correcting a user's behavior, the robot 100 and the terminal 100M may play audio related to a route returning to the set guidance route when a user being led along a set guidance route deviates from the guidance route. (17) As a second behavioral content different from the first behavioral content, the robot 100 and the communication terminal 100M may play audio praising the user's behavior.(18) The robot 100 and the communication terminal 100M may play, as a second action content different from the first action content, a sound expressing gratitude for the user's action. (19) The robot 100 and the communication terminal 100M may play, as a second action content different from the first action content, an image praising the user's action. (20) The robot 100 and the communication terminal 100M may play, as a second action content different from the first action content, an image expressing gratitude for the user's action. (21) The robot 100 and the communication terminal 100M may transmit specific information to a person other than the user as a third action content different from the first action content. (22) The robot 100 and the communication terminal 100M may play, as a third action content different from the first action content, a sound that attracts the user's interest. (23) The robot 100 and the communication terminal 100M may play, as a third action content different from the first action content, an image that attracts the user's interest.

[0476] The behavior determination unit 236 inputs the following into the sentence generation model every certain time: the state of the user 10 recognized by the state recognition unit 230, the state of the robot 100 or the communication terminal 100M, the current emotional value of the user 10 determined by the emotion determination unit 232, text representing the current emotional value of the robot 100 or the communication terminal 100M, and text asking about one of multiple types of robot behavior, including not taking any action. Based on the output of the sentence generation model, the behavior determination unit 236 determines the behavior of the robot 100 or the communication terminal 100M. Hereinafter, if the user 10 is not present near the robot 100 or the communication terminal 100M, the text input into the sentence generation model may not include the state of the user 10 and the current emotional value of the user 10, or may include a statement indicating that the user 10 is not present. Hereinafter, the communication terminal 100M may be referred to as a robot. The communication terminal 100M may be referred to as a smartphone or a smart AI device. Robot behavior may be referred to as communication terminal behavior.

[0477] As an example, the following text is input to the sentence generation model: "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (26) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0478] As another example, the following text is input to the sentence generation model: "The robot is feeling a little lonely. The user is not present. The robot's surroundings are dark. Which of the following (1) to (26) is the best behavior for the robot? (1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user. ..." Based on the output of the sentence generation model, "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary are the most appropriate behaviors," the behavior of the robot 100 is determined to be either "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0479] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0480] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance corresponding to the user's state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0481] When the behavior decision unit 236 decides that the robot behavior is "(7) The robot introduces news that the user is interested in," it uses the sentence generation model to decide the content of the robot's utterance corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0482] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 uses the image generation model to generate an image representing the event data for event data selected from the history data 222, and also uses the sentence generation model to generate an explanatory text representing the event data, and outputs the combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output the event image, but stores the event image in the behavior schedule data 224.

[0483] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that an image is to be edited, it selects event data from the history data 222 based on the emotion value, edits the image data of the selected event data, and outputs it. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0484] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," i.e., that the robot proposes an action for the user 10, it determines the user's action to be proposed using a sentence generation model based on the event data stored in the history data 222. At this time, the behavior control unit 250 causes a sound proposing the user's action to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound proposing the user's action, but instead stores the suggestion of the user's action in the behavior schedule data 224.

[0485] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes people that the user should meet," i.e., that the robot should propose people that the user 10 should have contact with, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine people that the proposed user should have contact with. At this time, the behavior control unit 250 causes a speaker included in the control target 252 to output a sound indicating that the robot is proposing people that the user should have contact with. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 does not output a sound indicating that the robot is proposing people that the user should have contact with, but instead stores the suggestion of people that the user should have contact with in the behavior schedule data 224.

[0486] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, such as "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or provide study-related advice, corresponding to the user's state and the user's or the robot's emotions. At this time, the behavior control unit 250 causes a sound representing the determined content of the robot's utterance to be output from a speaker included in the control target 252. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting the sound representing the determined content of the robot's utterance.

[0487] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," i.e., that the robot recalls event data, it selects event data from the history data 222. At this time, the emotion determining unit 232 determines the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the content of utterances and actions of the robot 100 to change the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0488] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222, and when that event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model, and if the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs to the sentence generation model "Which of (1), (2), and (3) would the user be most happy about?" If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will utter "(1) Let's go to the zoo" the next time it meets the user, which is created as an emotion change event and stored in the behavior schedule data 224.

[0489] Also, for example, event data with a large emotional value for the robot 100 is selected as an impressive memory for the robot 100. In this way, an emotional change event can be created based on the event data selected as an impressive memory.

[0490] The action determination unit 236 may detect the user's action using the sensor, either spontaneously or periodically, and may execute the following first action content when it determines that the action of the electronic device that is the communication terminal 100M is to lead the user based on the detected user action and pre-stored specific information. Note that the action determination unit 236 may determine the first action content when the camera included in the sensor is pointed in the user's traveling direction. In this way, the communication terminal 100M may start leading the user only when the user desires route guidance to be performed.

[0491] The behavior determining unit 236 may play back, as the communication terminal behavior, the first behavior content of "(11)" described above, that is, an image of a route that the user can walk, on the screen of the communication terminal 100M.

[0492] The behavior determining unit 236 may play back, as the communication terminal behavior, the first behavior content of "(12)" described above, that is, a sound related to a route that the user may walk.

[0493] The behavior decision unit 236 may play an image of a route other than the set guidance route on the screen of the communication terminal 100M as the first behavior content of the aforementioned "(13)", i.e., the first behavior content that corrects the user's behavior, as the communication terminal behavior.

[0494] The action determining unit 236 may play back, as the communication terminal action, the first action content of "(14)" described above, that is, a sound related to a route different from the set guidance route.

[0495] The behavior decision unit 236 may, as a communication terminal behavior, play on the screen of the communication terminal 100M an image of the first behavior content of the aforementioned "(15)," i.e., a route that returns to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0496] The behavior decision unit 236 may play, as a communication terminal behavior, the first behavior content of the aforementioned "(16)," i.e., audio related to a route that returns to the set guidance route when a user being led along the set guidance route deviates from the guidance route.

[0497] The behavior determination unit 236 may execute a second behavior content different from the first behavior content. Specifically, the behavior determination unit 236 may play the second behavior content of "(17)" described above, i.e., a voice praising the user's behavior, as the communication terminal behavior.

[0498] The behavior determining unit 236 may play back the second behavior content of "(18)" described above, that is, a voice expressing gratitude for the user's behavior, as the communication terminal behavior.

[0499] The behavior determining unit 236 may play back the second behavior content of "(19)" described above, that is, an image praising the user's behavior, as the communication terminal behavior.

[0500] The behavior determining unit 236 may play back the second behavior content of "(20)" described above, that is, an image expressing gratitude for the user's behavior, as the communication terminal behavior.

[0501] The action determination unit 236 may execute a third action content different from the first action content. Specifically, the action determination unit 236 may execute the third action content of "(21)" as the communication terminal action, i.e., transmit specific information to a person other than the user.

[0502] The behavior determining unit 236 may execute the third behavior content of "(22)" described above, that is, the reproduction of a sound that attracts the user's interest, as the communication terminal behavior.

[0503] The behavior determining unit 236 may execute the third behavior content (23) described above, that is, the reproduction of an image that attracts the user's interest, as the communication terminal behavior.

[0504] When playing back images of routes that the user can walk as the first action content shown in the above-mentioned "(11)" to "(16)", the related information collection unit 270 may store data such as images of routes that the user can walk in the collected data 223.

[0505] When playing audio or the like praising the user's behavior as the second behavior content shown in "(17)" to "(20)" above, the related information collection unit 270 may store data such as images of the route the user may walk in the collected data 223.

[0506] When the third action content shown in (21) to (23) above is to send specific information to a person other than the user, the related information collection unit 270 may store data on the specific information in the collected data 223.

[0507] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 has not taken any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0508] For example, if the user 10 is not present around the robot 100 and the robot 100 detects the user 10, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, if the user 10 is asleep and the robot 100 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0509] In addition, some parts of the robot 100 (e.g., the sensor module part 210, the storage part 220, the control part 228) may be provided outside the robot 100 (e.g., a server), and the robot 100 may communicate with the outside to function as each part of the robot 100 described above.

[0510] (Supplementary Note 1) A control system comprising: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device operations including no operation as an action of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably provided on a neck strap includes one or more sensors that collect information, and the device operation includes setting a first action content for leading the user by the electronic device reproducing at least one of audio and images. (Supplementary Note 2) The control system according to Supplementary Note 1, wherein the action determination unit detects the user's behavior based on the sensors autonomously or periodically, and determines the first action content when it is determined that the action of the electronic device is to lead the user based on the detected user behavior and pre-stored specific information. (Supplementary Note 3) The control system according to Supplementary Note 2, wherein the behavior determination unit determines the first behavior content when a camera included in the sensor is pointed in the user's traveling direction. (Supplementary Note 4) The control system according to Supplementary Note 2, wherein the first behavior content includes playback of at least one of audio and images that assist the user in walking. (Supplementary Note 5) The control system according to Supplementary Note 4, wherein the playback includes playback of at least one of audio and images that guide the user along a route different from a set guidance route. (Supplementary Note 6) The control system according to Supplementary Note 5, wherein the playback includes playback of at least one of audio and images that guide the user along a route that returns the user to the set guidance route when the user being led along the set guidance route deviates from the guide route. (Supplementary Note 7) The control system according to Supplementary Note 4, wherein the behavior determination unit detects the user's behavior while the user is leading or after playing back at least one of the audio and the image, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, determines a second behavior content that is different from the first behavior content.(Supplementary Note 8) The control system according to Supplementary Note 7, wherein the second action content includes playing at least one of a sound praising the user's action, a sound thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action. (Supplementary Note 9) The control system according to Supplementary Note 4, wherein the action determination unit determines whether the user's action has been corrected by detecting the user's action while the user is leading or after playing at least one of the sound and the image, and if the user's action has not been corrected, determines a third action content different from the first action content. (Supplementary Note 10) The control system according to Supplementary Note 9, wherein the third action content includes at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest. (Supplementary Note 11) The control system according to Supplementary Note 1, wherein the electronic device is a communication terminal, and the behavior determination unit determines one of a plurality of types of agent behaviors for interacting with the user, including no action, as the behavior of the communication terminal. (Supplementary Note 12) The control system according to Supplementary Note 11, wherein the behavior determination model is a sentence generation model having a dialogue function, and the behavior determination unit inputs text representing at least one of the user state, the agent state, the user's emotion, and the agent's emotion, and text asking about the agent's action, into the sentence generation model, and determines the behavior of the communication terminal based on the output of the sentence generation model.

[0511] (Other embodiment 1)

[0512] Next, a flow of a case where the agent system 500 executes a monitoring function of monitoring the emotions of the user 10 and taking a predetermined action on behalf of the user 10 as necessary will be described.

[0513] If the emotion of the user 10 determined by the emotion determination unit 232 is a negative emotion, the behavior determination unit 236 inputs text expressing the emotion of the user 10 and text asking what words to speak to the user 10 into the sentence generation model. For example, the emotion determination unit 232 determines the emotion of the user 10 based on the voice of the user 10 analyzed by the voice emotion recognition unit 211 of the sensor module unit 210. Note that the above method of determining the emotion of the user 10 is merely an example, and the emotion determination unit 232 can determine the emotion of the user 10 by appropriately using the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0514] Here, if the negative emotion of the user 10 determined by the emotion determination unit 232 is "anxiety," the behavior determination unit 236 inputs, for example, text such as "Please tell me what words to say to a person who is feeling the following negative emotion. Negative emotion: anxiety" into the sentence generation model. Based on the output of the sentence generation model, the behavior determination unit 236 determines the utterance content of the agent to be output as voice from the speaker included in the control object 252B of the communication terminal 100M as the behavior of the communication terminal 100M. For example, based on the output of the sentence generation model in response to the input of the above text, the behavior determination unit 236 determines the utterance content of the agent to be "You look anxious. Is something wrong?"

[0515] The behavior control unit 250 outputs, by voice, from the speaker the utterance content of the agent determined by the behavior determination unit 236. Specifically, the behavior control unit 250 causes the speaker to output, by voice, the content, "You look anxious. Is there something wrong?"

[0516] Thereafter, the behavior determination unit 236 inputs into the sentence generation model text representing the content of the dialogue between the user 10 and the agent, including the content of the agent's utterance output as voice from the speaker and the content of the user's utterance representing the state of the user 10 recognized by the state recognition unit 230 based on the character information output by the utterance understanding unit 212 of the sensor module unit 210, and text asking how to solve a problem the user 10 is facing. For example, the behavior determination unit 236 inputs the following text into the sentence generation model: "Based on the content of the dialogue between the user 10 and the agent below, please tell me how to solve the problem the user 10 is facing. Agent: You look anxious. Is something wrong? User 10: I'm lonely because my mother is away. Agent: Do you know where your mother went? User 10: Yes. She's shopping at that store." Based on the output of the sentence generation model, the behavior determination unit 236 determines the content of the agent's utterance to be output from the speaker. For example, based on the output of the sentence generation model generated by the above text input, the behavior decision unit 236 determines the content of the agent's utterance to be "I see. In that case, if you wait a little longer, he'll be back. Why don't you chat with me until then?"

[0517] The behavior control unit 250 outputs a solution to the problem that the user 10 is having from the speaker as the agent's speech content. Specifically, the behavior control unit 250 outputs the content determined by the behavior determination unit 236 as described above, "I see. In that case, if you wait a little longer, I'll be back. Why don't you have a chat with me until then?" from the speaker as the agent's speech content.

[0518] With the above-described configuration, the agent system 500 allows the user 10 to solve the problem with the support of the agent even if the user 10 is unable to think of a way to solve the problem by himself.

[0519] The above describes an example in which the agent system 500 derives a solution to a problem that the user 10 has based on the content of the dialogue between the user 10 and the agent. Next, we will explain the processing flow of the monitoring function of the agent system 500 when the user 10 has difficulty in dialogue with the agent.

[0520] If the emotion of the user 10 determined by the emotion determination unit 232 is a negative emotion and the user 10 has difficulty in conversing with the agent, the behavior determination unit 236 inputs into the sentence generation model text representing the user state and emotion of the user 10 and text asking about a problem the user 10 is having. The behavior determination unit 236 determines that the user 10 has difficulty in conversing with the agent, for example, if the microphone 201 cannot detect the voice of the user 10 for a predetermined time or longer after the agent's utterance content is output from the speaker, or if the voice of the user 10 detected by the microphone 201 cannot be analyzed by the utterance understanding unit 212.

[0521] In the above case, the behavior determination unit 236 inputs text, for example, "Considering the following user state and user 10's emotions, please tell us the problem that user 10 is facing. User state: User 10 is alone, there is a 99% chance that he is crying. User 10's emotions: Anxiety" into the sentence generation model. Based on the output of the sentence generation model, the behavior determination unit 236 determines to transmit problem information representing the problem that user 10 is facing as an action of the communication terminal 100M to the terminal of a related person who has a predetermined relationship with user 10. Examples of the predetermined relationship include blood ties, kinship ties, in-law ties, romantic relationships, friendships, and human relationships at school or work. In addition, in the predetermined relationship, it is desirable that the related person has feelings for user 10. Here, as an example, the predetermined relationship is a blood ties, the user 10 is the "child," and the related person is the user 10's "mother." For example, based on the output of the sentence generation model based on the input of the text, the behavior decision unit 236 decides to send problem information such as "Your child is probably lost" to the device (e.g., smartphone) of the mother of the user 10. Then, the communication processing unit 280 sends the problem information to the device of the mother of the user 10.

[0522] With the above-described configuration, the agent system 500 allows the agent to communicate the problems that the user 10 is having to the mother of the user 10 on behalf of the user 10, even if the user 10 is unable to communicate effectively.

[0523] As another example, if the user 10 has difficulty interacting with an agent, the agent system 500 may ask for help not only from the above-mentioned related parties but also from other users around the user 10 .

[0524] The emotion determination unit 232 determines the emotion of the other user based on an image of the other user around the user 10 captured by the 2D camera 203. For example, the emotion determination unit 232 determines the emotion of the other user based on the facial expression of the other user recognized by the facial expression recognition unit 213 of the sensor module unit 210. The 2D camera 203 is an example of a "camera." Note that the emotion determination unit 232 may determine the emotion of the other user by appropriately using information analyzed by other sensor module units 210 and the state of the other user recognized by the state recognition unit 230 in addition to the facial expression of the other user recognized by the facial expression recognition unit 213.

[0525] If the emotion determination unit 232 determines that the emotion of the other user is positive, the behavior determination unit 236 inputs the captured image, text describing the problem the user 10 is facing, and text asking what words to use when speaking to the other user into the sentence generation model. The behavior determination unit 236 inputs a prompt, for example, "Considering the problem the user 10 is facing below, please tell me what words to use when speaking to the person shown in the captured image below. User 10's problem: lost child. Captured image: xxxyyyzzz111.jpg" into the sentence generation model. Based on the output of the sentence generation model, the behavior determination unit 236 determines the agent's utterance content to be output from the speaker. For example, based on the output of the sentence generation model in response to the input of the prompt, the behavior determination unit 236 determines the agent's utterance content to be "Excuse me, man wearing a green T-shirt over there, could you please help my lost child?" In addition, the behavior decision unit 236 may input the above prompt into the sentence generation model only when the other user's positive emotion is a specific positive emotion such as "relief" or "a sense of relief," or when the emotion value of the other user's positive emotion is equal to or greater than a predetermined value.

[0526] The behavior control unit 250 outputs the problem that the user 10 is facing as the agent's speech content from a speaker to other users. As an example, the speaker is a super-directional speaker that can deliver sound to a targeted target. In this case, the behavior control unit 250 outputs sound in the direction where the other users are present. In this case, the behavior control unit 250 causes the speaker to output the content of the agent's speech, "Excuse me, man wearing the green T-shirt over there, could you please help my lost child?", which was determined by the behavior determination unit 236 as described above.

[0527] With the above configuration, the agent system 500 allows the user 10 to request help from other users who have positive emotions, thereby increasing the likelihood that the user's problem will be resolved compared to requesting help from other users who have negative emotions without considering their emotions. Furthermore, with the agent system 500, the agent requests help on behalf of the user 10, allowing other users to help the user 10 without hesitation. For example, there is a risk that other users may mistake the user 10 for a kidnapper if they are carrying a lost user 10. However, with the agent system 500, the agent requests help on behalf of the user 10, allowing the other user to easily explain that they are not attempting to kidnap the user 10. Furthermore, by interacting with the agent, the other user can learn the user 10's name, contact information, etc. from the agent and contact the agent to help the user 10. Furthermore, as described above, the agent system 500 allows the other user to easily explain that they are not attempting to kidnap the user 10, making it possible for the other user to take the user 10 to a safe location, such as a police station.

[0528] Here, when the agent system 500 executes the above-mentioned monitoring function, the behavior control unit 250 may output the agent's speech from the speaker in an output format corresponding to the agent's emotion determined by the emotion determination unit 232.

[0529] If the emotion determination unit 232 determines that the emotion of the user 10 is a negative emotion, it determines the emotion of the agent to be a negative emotion. For example, if the emotion determination unit 232 determines that the negative emotion of the user 10 is "anxiety," it determines the emotion of the agent to be "sad" and increases the emotion value of "sad."

[0530] The behavior control unit 250 outputs the agent's speech content from the speaker in an output mode based on the agent's negative emotion determined by the emotion determination unit 232. For example, when the agent's emotion is determined to be "sadness," the behavior control unit 250 increases the agent's speech rate output from the speaker or increases the volume of the agent's speech output from the speaker.

[0531] With the above configuration, the agent system 500 outputs the agent's speech from the speaker in an output style based on the agent's negative emotions, so that the agent's way of communicating can express the seriousness, urgency, etc.

[0532] (Supplementary Note 1) A control system comprising: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and an action determination unit that, at a predetermined timing, determines one of a plurality of types of device operations including no operation as an action of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device detachably provided on a neck strap includes one or more sensors that collect information, and the device operation includes setting a first action content for leading the user by the electronic device reproducing at least one of audio and images. (Supplementary Note 2) The control system according to Supplementary Note 1, wherein the action determination unit detects the user's behavior based on the sensors autonomously or periodically, and determines the first action content when it is determined that the action of the electronic device is to lead the user based on the detected user behavior and pre-stored specific information. (Supplementary Note 3) The control system according to Supplementary Note 2, wherein the behavior determination unit determines the first behavior content when a camera included in the sensor is pointed in the user's traveling direction. (Supplementary Note 4) The control system according to Supplementary Note 2, wherein the first behavior content includes playback of at least one of audio and images that assist the user in walking. (Supplementary Note 5) The control system according to Supplementary Note 4, wherein the playback includes playback of at least one of audio and images that guide the user along a route different from a set guidance route. (Supplementary Note 6) The control system according to Supplementary Note 5, wherein the playback includes playback of at least one of audio and images that guide the user along a route that returns the user to the set guidance route when the user being led along the set guidance route deviates from the guide route. (Supplementary Note 7) The control system according to Supplementary Note 4, wherein the behavior determination unit detects the user's behavior while the user is leading or after playing back at least one of the audio and the image, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, determines a second behavior content that is different from the first behavior content.(Supplementary Note 8) The control system according to Supplementary Note 7, wherein the second action content includes playing at least one of a sound praising the user's action, a sound thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action. (Supplementary Note 9) The control system according to Supplementary Note 4, wherein the action determination unit determines whether the user's action has been corrected by detecting the user's action while the user is leading or after playing at least one of the sound and the image, and if the user's action has not been corrected, determines a third action content different from the first action content. (Supplementary Note 10) The control system according to Supplementary Note 9, wherein the third action content includes at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest. (Supplementary Note 11) The control system according to Supplementary Note 1, wherein the electronic device is a communication terminal, and the behavior determination unit determines one of a plurality of types of agent behaviors for interacting with the user, including no action, as the behavior of the communication terminal. (Supplementary Note 12) The control system according to Supplementary Note 11, wherein the behavior determination model is a sentence generation model having a dialogue function, and the behavior determination unit inputs text representing at least one of the user state, the agent state, the user's emotion, and the agent's emotion, and text asking about the agent's action, into the sentence generation model, and determines the behavior of the communication terminal based on the output of the sentence generation model.(Supplementary Note 13) The control system according to Supplementary Note 12, wherein the behavior determination unit, when the user's emotion determined by the emotion determination unit is a negative emotion, inputs text representing the user's emotion and text asking what words to speak to the user into the sentence generation model, and determines, based on the output of the sentence generation model, a speech content of the agent to be output from a control object possessed by the communication terminal as a behavior of the communication terminal; and comprises a behavior control unit that controls the control object to execute the behavior of the communication terminal determined by the behavior determination unit, wherein the behavior determination unit inputs, to the sentence generation model, text representing a dialogue content between the user and the agent, including the speech content of the agent output from the control object and the speech content of the user as the user state recognized by the state recognition unit, and text asking a question about a solution to a problem the user is having, and determines, based on the output of the sentence generation model, the speech content of the agent to be output from the control object, and the behavior control unit outputs a solution to the problem the user is having from the control object as the speech content of the agent. (Supplementary Note 14) The control system according to Supplementary Note 13, wherein the behavior determination unit, when the user's emotion determined by the emotion determination unit is a negative emotion and the user has difficulty interacting with the agent, inputs text representing the user's state and the user's emotion and text asking about a problem the user is having into the sentence generation model, and determines, based on the output of the sentence generation model, to send problem information representing the problem the user is having to a terminal of a related party having a predetermined relationship with the user as an action of the communication terminal; and further comprises a communication processing unit that sends the problem information to the terminal of the related party.(Supplementary Note 15) The control system according to Supplementary Note 14, wherein the emotion determination unit determines the emotion of the other user based on a captured image of the other user present around the user taken with a camera, and the behavior determination unit, when the emotion of the other user determined by the emotion determination unit is a positive emotion, inputs the captured image, text expressing a problem the user is having, and text asking a question about words to speak to the other user into the sentence generation model, and determines an utterance content of the agent to be output from the control object based on the output of the sentence generation model, and the behavior control unit outputs the problem the user is having from the control object to the other user as the utterance content of the agent. (Supplementary Note 16) The control system according to Supplementary Note 13, wherein the emotion determination unit, when determining that the emotion of the user is a negative emotion, determines the emotion of the agent to be a negative emotion, and the behavior control unit outputs the utterance content of the agent from the control object in an output mode based on the negative emotion of the agent determined by the emotion determination unit.

[0533] (Other embodiment 2)

[0534] Next, a flow when the agent system 500 executes a support function for supporting the user 10 in taking a video using the communication terminal 100M will be described.

[0535] The emotion determination unit 232 determines the emotion of other users present around the user 10. The other users are assumed to be located within the capture range of the 2D camera 203. For example, the emotion determination unit 232 determines the emotion of the other users based on the facial expressions of the other users recognized by the facial expression recognition unit 213 of the sensor module unit 210. Note that the emotion determination unit 232 may determine the emotion of the other users by appropriately using information analyzed by other sensor module units 210 and the states of the other users recognized by the state recognition unit 230 in addition to the facial expressions of the other users recognized by the facial expression recognition unit 213.

[0536] When the emotion value of the other user determined by the emotion determination unit 232 has changed by a predetermined value or more, the behavior determination unit 236 determines, as the behavior of the communication terminal 100M, to start taking a video of the other user with the 2D camera 203. Here, the predetermined value is set to "2" as an example. The 2D camera 203 is an example of a "camera."

[0537] The behavior control unit 250 controls the 2D camera 203 to capture video of other users. The behavior control unit 250 is configured to capture video of other users using the 2D camera 203 for a predetermined period of time. The specific video, which is a video of the other user, is then stored in the storage unit 220. The 2D camera 203 is, as an example, an AI automatic tracking camera that automatically tracks and captures a recognized subject (here, the other user). Note that AI automatic tracking cameras are publicly known, as disclosed in, for example, internet searches such as <URL: https: / / info.system5.jp / whatsnew / archives / 96954> and <URL: https: / / weekly.ascii.jp / elem / 000 / 004 / 061 / 4061473 / >, and therefore a detailed description thereof will be omitted.

[0538] With the above configuration, the agent system 500 starts recording a video of another user when the emotion value of the other user changes by a predetermined value or more, and can automatically record a video when the emotion of the other user changes (for example, when the user feels fear or happiness).

[0539] Furthermore, the behavior determining unit 236 determines to post the specific video to a predetermined video posting service when the emotion value of the other user's positive emotion increases by a predetermined value or more. The communication processing unit 280 then posts the specific video of the other user whose positive emotion value has increased by a predetermined value or more to the predetermined video posting service. The video posting service is, for example, a social networking service (SNS). The communication processing unit 280 automatically posts the specific video to the video posting service by executing a publicly known automatic posting process disclosed in, for example, the internet search <URL: https: / / webuo.net / blog / shopify-app / 000205> and the internet search <URL: https: / / kurarebo.jp / a / blog / sns-marketing>.

[0540] With the above configuration, the agent system 500 automatically posts specific videos to a video posting service, thereby reducing the burden on users of posting videos to a video posting service. In particular, the agent system 500 automatically posts specific videos of other users whose positive emotion values ​​have increased by more than a predetermined value to a video posting service, allowing other users to share all of their videos when their positive emotions are high.

[0541] When the agent system 500 automatically posts a specific video to a video posting service, the specific video may be posted after undergoing image processing (e.g., blurring) to make it difficult to recognize the faces of other users shown in the specific video. This image processing may be performed by any known method.

[0542] Furthermore, when the emotion value of another user changes by a predetermined value or more, the behavior determination unit 236 inputs text representing the emotion of the other user, a specific video, and text asking the user a question about the specific video into the sentence generation model. For example, the behavior determination unit 236 inputs a prompt such as, "Considering the emotion of the other user and the content of the specific video, please tell me what to ask the user about the following specific video. Other user's emotion: happy Specific video: xxxyyyzzz111.wav" into the sentence generation model. Based on the output of the sentence generation model, the behavior determination unit 236 determines the utterance content of the agent to be output as a voice from the speaker included in the control object 252B of the communication terminal 100M as the behavior of the communication terminal 100M. For example, based on the output of the sentence generation model in response to the input of the prompt, the behavior determination unit 236 determines the utterance content of the agent to be, "I just noticed someone looking happy in front of the bulletin board. What do you think happened to that person?"

[0543] The behavior control unit 250 outputs a question about the specific video as the agent's speech from the speaker. Specifically, the behavior control unit 250 outputs the content determined by the behavior determination unit 236 as described above, "I just found someone looking happy in front of the bulletin board. What do you think happened to them?" as the agent's speech from the speaker.

[0544] Next, the behavior determination unit 236 inputs into the sentence generation model text representing the utterance content of the user 10 as the state of the user 10 recognized by the state recognition unit 230 based on the character information output by the utterance understanding unit 212 of the sensor module unit 210 as a response to the agent's utterance content output as voice from the speaker, and text asking about keywords in the utterance content of the user 10. The behavior determination unit 236 inputs into the sentence generation model, for example, a prompt such as, "Please tell me the keywords in the utterance content of the user 10 below. Utterance content of the user 10: Ah, that person. I think he was probably happy because he passed some exam." Based on the output of the sentence generation model, the behavior determination unit 236 determines to associate the keywords with a specific video and save the specific video as an action of the communication terminal 100M. For example, the behavior determining unit 236 determines to associate the keyword "someone who is happy about passing the exam," which is determined based on the output of the sentence generation model in response to the input of the prompt, with the specific video "xxxyyyzzz111.wav" shown by the other user, and to store the specific video. In this case, the specific video associated with the keyword is stored in the storage unit 220.

[0545] With the above configuration, the agent system 500 facilitates management of captured specific videos by storing specific videos linked to keywords determined from the content of the user 10's utterance. Note that the above example shows a case in which a specific video linked to keywords determined from the content of the user 10's utterance is stored when the emotion value of the other user's positive emotion increases by a predetermined value or more, but this is not limiting. In addition to the above example, the agent system 500 also stores a specific video linked to keywords determined from the content of the user 10's utterance when the emotion value of the other user's negative emotion increases by a predetermined value or more.

[0546] Furthermore, when the emotion value of the other user's negative emotion increases by a predetermined value or more, the behavior determination unit 236 inputs text expressing the other user's emotion, a specific video, and text asking about the other user's situation shown in the specific video into the sentence generation model. For example, the behavior determination unit 236 inputs a prompt such as, "Considering the emotion of the other user below and the content of the specific video, please tell me what words to ask the user 10 about the other user's situation shown in the specific video below. Other user's emotion: anxiety Specific video: yyyxxxzzz111.wav" into the sentence generation model. Based on the output of the sentence generation model, the behavior determination unit 236 determines the agent's utterance content to be output as voice from the speaker. For example, based on the output of the sentence generation model in response to the input of the prompt, the behavior determination unit 236 determines the agent's utterance content to be, "I just noticed someone looking anxious in front of a bank ATM (Automatic Teller Machine). What do you think happened to that person?"

[0547] The behavior control unit 250 outputs the situation of the other user shown in the specific video as the agent's speech content from the speaker. Specifically, the behavior control unit 250 outputs the content determined by the behavior determination unit 236 as described above, "I just noticed someone in front of the bank ATM who looked anxious. What do you think happened to that person?" as the agent's speech content from the speaker.

[0548] Here, if the action decision unit 236 determines that a specific agency needs to be notified based on the content of the dialogue between the user 10 and the agent, including the content of the agent's speech output from the speaker and the content of the user's speech, the action decision unit 236 decides to notify the specific agency determined from the content of the dialogue. The specific agency is, for example, an emergency call receiving agency that can issue emergency calls, such as the police, the Japan Coast Guard, or a fire department. The action decision unit 236 inputs prompts, including text representing the content of the dialogue between the user 10 and the agent and text asking whether a specific agency needs to be notified, into the sentence generation model as needed, and determines whether a specific agency needs to be notified based on the output of the sentence generation model. For example, when the sentence generation model receives the prompt, it can output whether a police call needs to be notified based on criteria disclosed in an internet search <URL: https: / / www.keishicho.metro.tokyo.lg.jp / jiken_jiko / 110 / 110_110.html>. Although details are omitted, when the above prompt is input, the sentence generation model can output whether or not it is necessary to report to other specific agencies (e.g., the Japan Coast Guard, the fire department) in light of the criteria indicating the necessity of reporting set by each specific agency. In other words, by using the sentence generation model, the agent system 500 can automatically determine the necessity of reporting to a specific agency and the specific agency to report to, based on the content of the dialogue between the user 10 and the agent.

[0549] Then, when the action decision unit 236 decides to report to a specific organization, the communication processing unit 280 transmits the specific video and specific information including the location where the specific video was filmed to the terminal of the specific organization determined by the action decision unit 236. For example, assume that the action decision unit 236 decides to report to the police as the specific organization. In this case, the communication processing unit 280 transmits the specific video "yyyxxxzzz111.wav" indicating the other user and specific information including the location where the specific video was filmed to the police terminal (e.g., a personal computer). Note that the information that can be included in the specific information is not limited to the above. For example, the specific information may also include other information such as the content of the conversation between the user 10 and the agent.

[0550] With the above configuration, when an incident or accident occurs, for example, the agent system 500 can determine whether it is necessary to report to a specific organization and automatically report it if necessary, without requiring the user 10 to determine whether it is necessary to report to a specific organization. Also, even if the user 10 determines that it is not necessary to report to a specific organization, the agent system 500 can appropriately determine the necessity of reporting in light of the criteria set by the specific organization and automatically report it if necessary.

[0551] (Supplementary Note 1) A control system comprising: a state recognition unit that recognizes a user state including a user's behavior and a state of an electronic device; an emotion determination unit that determines the user's emotion, the emotion of another user present around the user, or the emotion of the electronic device; an action determination unit that determines, when an emotion value that indicates the intensity of the emotion of the other user changes by a predetermined value or more, to start taking a video of the other user with a camera as the action of the electronic device; and an action control unit that controls the electronic device to execute the action of the electronic device determined by the action determination unit. (Supplementary Note 2) The action determination unit determines, at a predetermined timing, one of a plurality of types of device operations (including no operation) as the action of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model; the electronic device detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting a first action content that leads the user by the electronic device playing at least one of audio and images. (Supplementary Note 3) The control system according to Supplementary Note 2, wherein the behavior determination unit detects the user's behavior based on the sensor either autonomously or periodically, and determines the first behavior content when it determines that the behavior of the electronic device is to lead the user based on the detected behavior of the user and pre-stored specific information. (Supplementary Note 4) The control system according to Supplementary Note 3, wherein the behavior determination unit determines the first behavior content when a camera included in the sensor is pointed in the user's traveling direction. (Supplementary Note 5) The control system according to Supplementary Note 3, wherein the first behavior content includes playback of at least one of audio and images that assist the user in walking. (Supplementary Note 6) The control system according to Supplementary Note 5, wherein the playback includes playback of at least one of audio and images that guide the user along a route different from a set guidance route. (Supplementary Note 7) The control system according to Supplementary Note 6, wherein the playback includes playback of at least one of audio and images that guide the user along a route that returns the user to the set guidance route when the user being led along the set guidance route deviates from the guidance route.(Supplementary Note 8) The control system according to Supplementary Note 5, wherein the behavior determination unit determines whether the user's behavior has been corrected by detecting the user's behavior while the user is leading or after playing at least one of the audio and the image, and if the user's behavior has been corrected, determines a second behavior content different from the first behavior content. (Supplementary Note 9) The control system according to Supplementary Note 8, wherein the second behavior content includes playing at least one of audio praising the user's behavior, audio thanking the user for the user's behavior, an image praising the user's behavior, and an image thanking the user for the user's behavior. (Supplementary Note 10) The control system according to Supplementary Note 5, wherein the behavior determination unit determines whether the user's behavior has been corrected by detecting the user's behavior while the user is leading or after playing at least one of the audio and the image, and if the user's behavior has not been corrected, determines a third behavior content different from the first behavior content. (Supplementary Note 11) The control system according to Supplementary Note 10, wherein the third action content includes at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest. (Supplementary Note 12) The control system according to Supplementary Note 1, wherein the electronic device is a communication terminal, and the action determination unit determines one of a plurality of types of agent actions for interacting with the user, including not taking any action, as the action of the communication terminal. (Supplementary Note 13) The control system according to Supplementary Note 12, further comprising a sentence generation model having a dialogue function, and wherein the ac...

Claims

1. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model, wherein the electronic device is detachably provided on a mobile unit having wheels for automatic driving, the electronic device includes one or more sensors that collect information for controlling the rotation of the wheels, and the device operation includes setting a first behavior content that leads the user by controlling the wheels.

2. The control system of claim 1, wherein the behavior decision unit detects the user's behavior based on the sensor either autonomously or periodically, and when it decides that the behavior of the electronic device is to lead the user based on the detected behavior of the user and pre-stored specific information, it determines the first behavior content.

3. The control system according to claim 2, wherein the first action content includes playing at least one of a sound and an image that assists the user in walking.

4. The control system according to claim 3, wherein the playback includes playback of at least one of audio and images providing guidance along a route other than the set guidance route.

5. The control system of claim 4, wherein the playback includes playback of at least one of audio and images guiding the user along a route back to the guided route when the user, being led along a set guided route, deviates from the guided route.

6. The control system of claim 3, wherein the behavior decision unit detects the user's behavior while the movement unit is leading the user or after playing at least one of the audio and the image, thereby determining whether the user's behavior has been corrected, and if the user's behavior has been corrected, determines a second behavior content different from the first behavior content.

7. The control system of claim 6, wherein the second action content includes playing at least one of a voice praising the user's action, a voice thanking the user for the user's action, an image praising the user's action, and an image thanking the user for the user's action.

8. The control system of claim 3, wherein the behavior decision unit detects the user's behavior while the movement unit is leading the user or after playing at least one of the audio and the image, thereby determining whether the user's behavior has been corrected, and if the user's behavior has not been corrected, determines a third behavior content different from the first behavior content.

9. The control system of claim 8, wherein the third action content includes at least one of sending specific information to a person other than the user, playing a sound that attracts the user's interest, and playing an image that attracts the user's interest.

10. The control system according to claim 1, wherein the electronic device is a communication terminal, and the behavior decision unit decides that the behavior of the communication terminal is one of a plurality of types of agent behaviors for interacting with the user, including no action.

11. The control system described in claim 10, wherein the behavior decision model is a sentence generation model having an interactive function, and the behavior decision unit inputs text representing at least one of the user state, the agent state, the user's emotion, and the agent's emotion, and text asking about the agent's behavior, into the sentence generation model, and determines the behavior of the communication terminal based on the output of the sentence generation model.

12. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model; wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting a first behavior content that leads the user by the electronic device playing at least one of sounds and images.

13. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations including no operation as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device and a behavior determination model; wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting a first behavior content that leads the user by the electronic device playing at least one of sounds and images.

14. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user, the emotions of other users around the user, or the emotion of the electronic device; an action determination unit that determines, when an emotion value indicating the strength of the emotion of the other users has changed by more than a predetermined value, to start taking video of the other users with a camera as an action of the electronic device; and an action control unit that controls the electronic device to execute the action of the electronic device determined by the action determination unit.

15. A control system comprising: a state recognition unit that recognizes a user state including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior determination model; wherein the behavior determination unit acquires surrounding environmental information from one or more sensors that collect information and are provided in the electronic device that is detachably attached to the user's body, and wherein the behavior determination unit checks the user's health condition by causing the electronic device to output at least one of voice and text in accordance with the acquired environmental information, and wherein the electronic device makes appropriate suggestions to the user.

16. A control system comprising: a state recognition unit that recognizes a user state including the behavior of a user and other users; an emotion determination unit that determines the emotions of the user and other users; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations as the operation of an electronic device using at least one of the user state and the user's emotion and a behavior determination model, wherein the state recognition unit recognizes the surrounding state from one or more sensors possessed by the electronic device that is detachably attached to the user's body, and the behavior determination unit determines that the electronic device will speak to the other user when the other user is present in the vicinity of the user.

17. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the user's emotion or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operation, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior decision model, wherein the emotion determination unit infers the emotions of others from images of the others around the user obtained from one or more sensors that collect information possessed by the electronic device that is detachably attached to the user's body, and the behavior determination unit makes appropriate suggestions to the user when the others are feeling distressed.

18. A control system comprising: a state recognition unit that recognizes a user state including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior decision model; wherein the electronic device is detachably attached to a neck strap and includes one or more sensors that collect information, including a photographing unit that photographs the surroundings, and the device operation includes issuing a warning to people approaching the photographing unit who meet predetermined conditions.

19. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting an action content that assists the user in performing work at a position lower than the user's eyes by the electronic device playing back at least one of sounds and images generated by the behavior determination model based on the information collected by the electronic device.

20. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines an emotion of the user or an emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including not operating, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model; wherein the electronic device is detachably provided on a mobile unit having wheels for automatic driving, the electronic device includes one or more sensors that collect information on the situation around the user, the electronic device controls the rotation of the wheels based on the information on the situation around the user, and the behavior determination unit determines, when a dangerous obstacle is detected around the user, a first behavior content that leads the user to a safe place to prevent the user from approaching the dangerous obstacle, as the behavior of the electronic device.

21. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior decision model, wherein the electronic device is provided in a mobile unit capable of automatic driving, and the electronic device includes one or more sensors that collect information on the situation around the user, and the behavior determination unit determines, based on the information on the situation around the user, that the user is in danger, as the behavior of the electronic device to take an action in place of the user.

22. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations including no operation as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior determination model, wherein the electronic device is detachably provided on a mobile unit having wheels for automatic driving, the electronic device includes one or more sensors that collect information on the situation around the user, and the device operation includes setting a first behavior content that assists the user in walking an animal by controlling the wheels based on the information on the situation around the user.

23. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior determination model; wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information on the surrounding situation, and the device operation includes setting an action content that assists the user in walking by notifying the user of the surrounding situation by the electronic device playing at least one of audio and images.

24. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device and a behavior determination model; wherein the electronic device is detachably attached to a neck strap and includes one or more sensors that collect biometric information of the user, and the device operation includes notifying the user of warning information based on the collected biometric information by the electronic device playing at least one of audio and images.

25. A control system comprising: a state recognition unit that recognizes a user state including user behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior determination model; wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect surrounding environment information, and the device operation includes notifying the user of warning information based on the collected surrounding environment information by the electronic device playing at least one of audio and images.

26. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the user's emotion, and the emotion of the electronic device, and a behavior determination model, wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information, and the device operation includes setting an action content that assists the user in performing work at a position lower than the user's eyes by the electronic device playing back at least one of sounds and images generated by the behavior determination model based on the information collected by the electronic device.

27. A control system comprising: a state recognition unit that recognizes a user state including a user's behavior, and a state of an electronic device; an emotion determination unit that determines the emotion of the user or the emotion of the electronic device; and a behavior determination unit that, at a predetermined timing, determines one of a plurality of types of device operations, including no operation, as the behavior of the electronic device using at least one of the user state, the state of the electronic device, the emotion of the user, and the emotion of the electronic device, and a behavior determination model; wherein the electronic device that is detachably attached to a neck strap includes one or more sensors that collect information, the electronic device is worn by the user and cooperates with another electronic device that includes one or more sensors that collect information, and the device operation includes setting behavior content that supports the user by the electronic device playing at least one of sounds and images generated by the behavior determination model based on the information collected by the electronic device and the information collected by the other electronic device.

Images