electronic machinery

Abstract

To take appropriate actions.SOLUTION: An electronic device comprises a control unit configured to: recognize an action of a user using a vehicle; determine an own action in response to the recognized action of the user in order to avoid dangerous driving; and control a control target on the basis of the determined own action.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The disclosed embodiments relate to electronic devices.

Background Art

[0002] Conventionally, a technique for determining appropriate actions of a robot with respect to a user's state has been disclosed (see, for example, Patent Document 1). Patent Document 1 discloses that when a robot recognizes a user's reaction when performing a specific action and cannot determine the robot's action with respect to the recognized user's reaction, the robot's action is updated by receiving information on an action suitable for the recognized user's state from a server.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, there is room for improvement in performing appropriate actions with respect to a user's actions.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an electronic device capable of performing appropriate actions.

Means for Solving the Problems

[0006] An electronic device according to an aspect of an embodiment recognizes an action of a user who uses a vehicle, determines its own action so as to be able to avoid dangerous driving corresponding to the recognized action of the user, and controls a control target based on the determined own action. The electronic device includes a control unit. [[ID=4)]

Effects of the Invention

[0007] According to one embodiment, appropriate actions can be taken. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an example of the control system according to this embodiment. [Figure 2] Figure 2 is a schematic diagram showing the functional configuration of the robot. [Figure 3] Figure 3 is a schematic diagram illustrating an example of a motion flow related to actions that determine a robot's behavior. [Figure 4] Figure 4 shows an example of robot placement in a vehicle. [Figure 5] Figure 5 is a schematic diagram showing the functional configuration of a vehicle that enables automated driving control by a robot. [Figure 6] Figure 6 is a schematic diagram showing an example of the hardware configuration of a computer that functions as both a robot and a server. [Figure 7] Figure 7 shows an emotion map in which multiple emotions are mapped. [Figure 8] Figure 8 shows another example of an emotion map. [Figure 9] Figure 9 shows an example of an emotion table. [Figure 10] Figure 10 shows an example of an emotion table. [Modes for carrying out the invention]

[0009] The present invention will be described below through embodiments, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention. In this embodiment, a "robot" will be used as an example of an electronic device. However, the electronic device may be a stuffed animal, a portable terminal device such as a smartphone, an input device such as a smart speaker, etc.

[0010] Figure 1 is a schematic diagram showing an example of a control system 1 according to this embodiment. As shown in Figure 1, the control system 1 comprises a plurality of robots 100, a coordinating device 400, and a server 300. Each of the plurality of robots 100 is managed by a user.

[0011] Robot 100 engages in conversation with the user and provides the user with video. In this process, robot 100 collaborates with a server 300 or the like, which can communicate via the communication network 20, to engage in conversation with the user and provide the user with video and other information. For example, robot 100 not only learns appropriate conversations on its own, but also learns to conduct more appropriate conversations with the user in cooperation with server 300. Furthermore, robot 100 has server 300 record video data of the user that it has captured, and requests video data from server 300 as needed to provide it to the user.

[0012] Furthermore, robot 100 possesses emotion values ​​that represent the type of emotion it is experiencing. For example, robot 100 has emotion values ​​that represent the intensity of each of the following emotions: joy, anger, sadness, pleasure, pleasure, displeasure, relief, anxiety, sadness, excitement, worry, relief, fulfillment, emptiness, and neutrality. For example, when robot 100 is conversing with a user while its excitement emotion value is high, it will speak at a fast pace. In this way, robot 100 can express its emotions through its actions.

[0013] Furthermore, the robot 100 may be configured to determine its actions in response to the user's emotions by matching a text generation model with an emotion engine. Specifically, the robot 100 may be configured to recognize the user's actions, determine the user's emotions in response to those actions, and determine its actions in response to the determined emotions.

[0014] More specifically, when the robot 100 recognizes the user's actions, it automatically generates the content of the actions that the robot 100 should take with respect to the user's actions using a preset text generation model. The text generation model may be interpreted as an algorithm and operation for automatic dialogue processing by characters. Since the text generation model is publicly known as disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-081444 and chatGPT (Internet search <URL: https: / / openai.com / blog / chatgpt>), a detailed description thereof is omitted. Such a text generation model is composed of a large language model (LLM: Large Language Model). As described above, in this embodiment, by combining the large language model and the emotion engine, it is possible to reflect the emotions of the user and the robot 100 and various language information in the actions of the robot 100. That is, according to this embodiment, a synergistic effect can be obtained by combining the text generation model and the emotion engine.

[0015] In addition, the robot 100 has a function of recognizing the user's actions. The robot 100 recognizes the user's actions by analyzing the user's face image acquired by the camera function and the user's voice acquired by the microphone function. The robot 100 determines the actions to be executed by the robot 100 based on the recognized user's actions and the like.

[0016] The robot 100 stores rules that define the actions to be executed by the robot 100 based on the user's emotions, the robot 100's emotions, and the user's actions, and performs various actions according to the rules.

[0017] Specifically, the robot 100 has reaction rules for determining the actions of the robot 100 based on the user's emotions, the emotions of the robot 100, and the user's actions. For example, in the reaction rules, when the user's action is "laugh", the action of "laugh" is defined as the action of the robot 100. Also, in the reaction rules, when the user's action is "angry", the action of "apologize" is defined as the action of the robot 100. Further, in the reaction rules, when the user's action is "ask a question", the action of "answer" is defined as the action of the robot 100. In the reaction rules, when the user's action is "be sad", the action of "talk to" is defined as the action of the robot 100.

[0018] Based on the reaction rules, when the robot 100 recognizes that the user's action is "angry", the robot 100 selects the action of "apologize" defined in the reaction rules as the action to be executed. For example, when the robot 100 selects the action of "apologize", it performs the "apologize" motion and outputs a voice representing the words "apologize".

[0019] Also, when the emotion of the robot 100 is "normal" (i.e., "joy" = 0, "anger" = 0, "sorrow" = 0, "happiness" = 0) and the condition that the user's state is "alone and seems lonely" is satisfied, it is defined that the emotional change content of the robot 100 becomes "worried" and the action of "talk to" can be executed.

[0020] Based on the reaction rules, when the current emotion of the robot 100 is "normal" and the robot 100 recognizes that the user is in a state of being alone and seems lonely, the robot 100 increases the emotional value of "sorrow". Also, the robot 100 selects the action of "talk to" defined in the reaction rules as the action to be executed for the user. For example, when the robot 100 selects the action of "talk to", it converts the words "what's wrong?" expressing worry into a worried voice and outputs it.

[0021] Furthermore, the robot 100 transmits user response information to the server 300, indicating that a positive reaction was received from the user as a result of this action. This user response information includes, for example, the user's action of "getting angry," the robot 100's action of "apologizing," the fact that the user's reaction was positive, and the user's attributes.

[0022] The server 300 stores user response information received from each robot 100. The server 300 then analyzes the user response information from each robot 100 and updates the response rules.

[0023] Robot 100 receives updated response rules from server 300 by querying server 300 for the updated response rules. Robot 100 incorporates the updated response rules into the response rules it stores. This allows robot 100 to incorporate response rules acquired by other robots into its own response rules. When response rules are updated, they may be automatically sent from server 300 to robot 100.

[0024] Furthermore, the robot 100 can perform actions in cooperation with the linked devices 400. The linked devices 400 include, for example, karaoke machines, wine cellars, refrigerators, terminal devices (PCs (Personal Computers), smartphones, tablets, etc.), washing machines, vehicles 500, cameras, toilet facilities, electric toothbrushes, televisions, displays, furniture (closets, etc.), medicine cabinets, musical instruments, lighting equipment, and exercise toys (unicycles, etc.). These linked devices 400 are connected to the robot 100 via the communication network 20 so as to be able to communicate with it, and send and receive information between them. With this configuration, the linked devices 400 can control themselves and converse with users, etc., according to the instructions of the robot 100.

[0025] This disclosure describes an example of performing various actions for a user through the cooperation of a vehicle 500, which is a linked device 400, and a robot 100. The vehicle 500 may be a four-wheeled automobile, a motorcycle, a personal mobility device, or the like.

[0026] Specifically, robot 100 is placed inside, for example, vehicle 500, and determines whether the user is driving dangerously based on conversations with the user and the user's driving behavior. If robot 100 determines that the user is driving dangerously, it increases its own emotional values ​​such as "discomfort," "anxiety," and "worry," and then, based on corresponding response rules, takes actions to calm or heighten the user's emotions in order to prevent dangerous driving.

[0027] For example, robot 100 determines whether the vehicle is speeding based on the user's speech, accelerator pedal position, vehicle speed, etc. If robot 100 determines that the vehicle is speeding, it will perform actions to calm the user by saying things like, "You're going too fast," "Let's slow down," or "We have time, so let's go slowly."

[0028] Furthermore, for example, when entering an intersection, robot 100 will determine whether or not it is likely to run a red light based on factors such as the timing of its speech when the light is about to turn red, the accelerator pedal position, and the vehicle speed. If robot 100 determines that it is likely to run a red light, it will take actions to calm the user by saying things like, "It's impossible, let's stop," "It's dangerous," or "Calm down."

[0029] Furthermore, for example, robot 100 determines whether a situation is likely to become aggressive driving based on the user's utterances regarding the vehicle that cut in, the distance between vehicles, and the frequency of braking. If robot 100 determines that a situation is likely to become aggressive driving, it will take actions to calm the user's emotions by uttering phrases such as, "It's understandable to be upset when someone cuts you off, but stay calm," or "Maintain a safe distance."

[0030] Furthermore, for example, robot 100 determines whether the user is in a state of distracted driving with reduced concentration and attention, based on the user's absentminded speech, distractions, driving while looking away, and situations where the scenery is unchanging. If robot 100 determines that the user is in a state of distracted driving, it will take actions to improve the user's emotional state by saying things like, "Let's concentrate on driving," or "Don't look away."

[0031] Furthermore, for example, robot 100 determines whether the user is likely to fall asleep at the wheel based on the user's verbal utterances such as "I'm tired" or "I'm sleepy," as well as yawning, prolonged silence, the degree of vehicle 500's swaying based on steering angle, and the frequency of braking. If robot 100 determines that the user is likely to fall asleep at the wheel, it will perform actions to enhance the user's emotions by uttering phrases such as "Let's sing a song," or "By the way, I have an interesting story for you. Would you like to hear it?" At this time, it may intentionally make jarring noises to wake the user up.

[0032] Furthermore, if the robot 100 determines that the driver is likely to fall asleep at the wheel, it may take action to guide the driver to take a break by uttering a phrase such as "Let's take a break." At this time, the robot 100 may search for information such as nearby facilities via the communication network 20 and, along with uttering a phrase such as "It looks like there's a relaxing cafe nearby. I'll guide you there," navigate the driver to a suitable place to take a break.

[0033] Furthermore, for example, robot 100 determines whether the situation is likely to lead to dangerous driving due to poor health, based on the user's verbal statements such as "I feel sluggish" or "I have a headache," prolonged periods of silence, the degree of swaying of vehicle 500 based on steering angle, and the frequency of braking. If robot 100 determines that the situation is likely to lead to dangerous driving due to poor health, it will take action to help the user avoid danger by uttering a statement such as "You seem unwell. Let's stop for a moment." At this time, robot 100 may also search for information such as nearby medical facilities via the communication network 20 and, along with uttering a statement such as "There seems to be a hospital nearby. I will guide you there," provide navigation to a suitable location for medical treatment.

[0034] Furthermore, the robot 100 may be configured to cooperate with the vehicle control system 600 (see Figure 5) of the vehicle 500, enabling automatic driving control in which the robot 100 operates the vehicle 500 on behalf of the user. In this case, the robot 100 may perform actions such as moving to a suitable location for rest or medical consultation, or making an emergency stop in a safe location, through its own automatic driving control.

[0035] Furthermore, if the user follows the robot 100's instructions to avoid dangerous driving and continues to drive safely, the robot 100 will become happy, so to speak, by increasing its emotional values ​​such as "joy," "pleasantness," "security," "relief," and "fulfillment." On the other hand, if the user repeatedly engages in dangerous driving, the robot 100 will increase its negative emotional values ​​such as "anger" and "displeasure," and take action such as becoming genuinely angry. In this case, if the robot 100 is capable of driving the vehicle 500 on behalf of the user, as described above, the robot 100 may forcibly disable the user's driving operation if its emotional values ​​such as "anger" and "displeasure" exceed a predetermined threshold, and the robot 100 may take over driving the vehicle 500.

[0036] Furthermore, the robot 100 may automatically learn various situations that could lead to dangerous driving, and thereafter, whenever it encounters such a situation, it may take action to warn the driver to avoid dangerous driving.

[0037] Thus, in this disclosure, the robot 100 can perform actions in cooperation with the vehicle 500 to enable the user driving the vehicle 500 to take actions such as avoiding dangerous driving. In other words, the robot 100 according to this disclosure can perform appropriate actions for the user.

[0038] Figure 2 is a schematic diagram showing the functional configuration of robot 100. Robot 100 is composed of a control unit having a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, an action recognition unit 234, an action determination unit 236, a memory control unit 238, an action control unit 250, a controlled object 252, and a communication processing unit 280.

[0039] The controlled object 252 includes a display device, a speaker, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 100 are controlled by controlling the motors in the arms, hands, and feet. Some of the robot 100's emotions can be expressed by controlling these motors. The robot 100's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes. For example, the display device is located on the robot 100's chest. The robot 100's facial expressions can also be expressed by controlling the display on the display device. The display device may also display the content of the conversation with the user as text. The posture, gestures, and facial expressions of the robot 100 are examples of the robot 100's attitude.

[0040] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, an acceleration sensor 205, a thermosensor 206, and a touch sensor 207. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 is mounted on the head of the robot 100 and may have a function for binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating it with an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by an infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures images using visible light and generates visible light video information. The contour of an object may be detected from the video information generated by the 2D camera 203. The distance sensor 204 detects the distance to an object by irradiating it with, for example, a laser or ultrasound. The acceleration sensor 205 is, for example, a gyro sensor and detects the acceleration of the robot 100. The thermosensor 206 detects the temperature around the robot 100. The touch sensor 207 is a sensor that detects user touch operations and is placed, for example, on the head and hands of the robot 100. The sensor unit 200 may also include other sensors such as a clock and motor feedback sensors.

[0041] Note that, among the components of the robot 100 shown in Figure 2, the components other than the controlled object 252 and the sensor unit 200 are examples of components of the behavior control system of the robot 100. The behavior control system of the robot 100 controls the controlled object 252.

[0042] The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user's past emotional values ​​and behavioral history. This emotional value and behavioral history is recorded for each user, for example, by associating it with the user's identification information. At least a portion of the storage unit 220 is implemented by a storage medium such as memory. It may also include a person database that stores the user's facial image, user attribute information, etc. Of the components of the robot 100 shown in Figure 2, the functions of the components other than the controlled object 252, the sensor unit 200, and the storage unit 220 can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations by the basic software (OS) and programs that run on the OS.

[0043] The sensor module 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. The sensor module 210 receives information detected by the sensor unit 200. The sensor module 210 analyzes the information detected by the sensor unit 200 and outputs the analysis results to the user state recognition unit 230.

[0044] The voice emotion recognition unit 211 of the sensor module 210 analyzes the user's voice detected by the microphone 201 and recognizes the user's emotions. For example, the voice emotion recognition unit 211 extracts feature quantities such as frequency components of the voice and recognizes the user's emotions based on the extracted feature quantities. The speech understanding unit 212 analyzes the user's voice detected by the microphone 201 and outputs text information representing the content of the user's speech. For example, if the user is speaking, the speech understanding unit 212 determines the degree of match with the phrase of the speech.

[0045] The facial expression recognition unit 213 recognizes the user's facial expressions and emotions from the user's image captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the user's facial expressions and emotions based on the shape and positional relationship of the eyes and mouth. For example, the facial expression recognition unit 213 recognizes the user's facial expressions and emotions while the user is driving the vehicle 500.

[0046] The face recognition unit 214 recognizes the user's face. The face recognition unit 214 recognizes the user by matching the face images stored in the person database (not shown) with the user's face image captured by the 2D camera 203.

[0047] The user state recognition unit 230 recognizes the user's state based on the information analyzed by the sensor module unit 210. For example, it primarily performs perceptual processing using the analysis results from the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as the user is likely to exceed the speed limit, run a red light, is driving distractedly, appears sleepy, or appears to be unwell, and processes the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as the user is likely to commit a speeding violation.

[0048] The emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user state are input into a pre-trained neural network to obtain an emotion value indicating the user's emotion.

[0049] Here, the emotion value, which indicates the user's emotion, is a value that indicates whether the user's emotion is positive or negative. For example, if the user's emotion is a positive emotion accompanied by pleasure or comfort, such as "joy," "pleasure," "pleasantness," "relief," "excitement," "relief," and "fulfillment," it will show a positive value, and the more positive the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sadness," "discomfort," "anxiety," "grief," "worry," and "emptiness," it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the user's emotion is none of the above ("neutral"), it will show a value of 0.

[0050] Furthermore, the emotion determination unit 232 determines an emotion value indicating the robot 100's emotions based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230.

[0051] Robot 100's emotional value includes emotional values ​​for each of several emotional categories, such as values ​​(0-5) indicating the intensity of "joy," "anger," "sadness," and "happiness."

[0052] Specifically, the emotion determination unit 232 determines an emotion value indicating the robot 100's emotions according to a rule for updating the robot 100's emotion value, which is determined in association with the information analyzed by the sensor module unit 210 and the user state recognized by the user state recognition unit 230.

[0053] For example, if the user state recognition unit 230 recognizes that the user appears lonely, the emotion determination unit 232 increases the "sadness" emotion value of the robot 100. Also, if the user state recognition unit 230 recognizes that the user has smiled, the emotion determination unit 232 increases the "joy" emotion value of the robot 100.

[0054] Furthermore, the emotion determination unit 232 may determine an emotion value indicating the robot 100's emotions by further considering the state of the robot 100. For example, the emotion value of "sadness" of the robot 100 may be increased when the robot 100's battery level is low or when the surrounding environment of the robot 100 is completely dark. In addition, if the user continues to talk to the robot despite the low battery level, the emotion value of "anger" may be increased.

[0055] The behavior recognition unit 234 recognizes user behavior based on information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the behavior recognition unit 234 inputs the information analyzed by the sensor module unit 210 and the recognized user state into a pre-trained neural network, obtains the probability of each of several predetermined behavior classifications (e.g., "laughing," "getting angry," "asking a question," "being sad"), and recognizes the behavior classification with the highest probability as the user's behavior. For example, the behavior recognition unit 234 recognizes user behaviors such as "concentrating on driving," "looking away," "driving while distracted," and "dozing off" in vehicle 500.

[0056] As described above, in this embodiment, the robot 100 identifies the user and acquires the user's speech content. However, when acquiring and using said speech content, the robot obtains the necessary consent from the user in accordance with the law, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the user's personal information and privacy.

[0057] The action decision unit 236 determines an action corresponding to the user's action recognized by the action recognition unit 234, based on the user's current emotion value determined by the emotion decision unit 232, the history data 222 of past emotion values ​​determined by the emotion decision unit 232 before the user's current emotion value was determined, and the emotion value of the robot 100. In this embodiment, the action decision unit 236 uses the most recent emotion value included in the history data 222 as the user's past emotion value, but the disclosed technology is not limited to this embodiment. For example, the action decision unit 236 may use multiple most recent emotion values ​​as the user's past emotion value, or it may use an emotion value from a unit period, such as one day ago. Furthermore, the action decision unit 236 may determine an action corresponding to the user's action by further considering not only the robot 100's current emotion value but also the history of the robot 100's past emotion values. The action determined by the action decision unit 236 includes gestures performed by the robot 100 or the content of speech uttered by the robot 100.

[0058] In this embodiment, the action decision unit 236 determines the action of the robot 100 based on a combination of the user's past and current emotional values, the emotional value of the robot 100, the user's action, and the reaction rule 221, as an action corresponding to the user's action. For example, if the user's past emotional value is positive and the current emotional value is negative, the action decision unit 236 determines an action to change the user's emotional value to positive, as an action corresponding to the user's action.

[0059] Response rule 221 defines the actions of robot 100 in response to a combination of the user's past and current emotional values, robot 100's emotional value, and the user's actions. For example, if the user's past emotional value is positive, the current emotional value is negative, and the user's action is sad, then the rule defines a combination of gestures and speech content for robot 100 to ask encouraging questions while using gestures.

[0060] For example, the response rule 221 defines the robot 100's actions for all combinations of the robot's emotional value patterns (1296 patterns, which are the 6 values ​​from "0" to "5" for "joy," "anger," "sadness," and "happiness" raised to the power of 4), the combinations of the user's past and current emotional values, and the user's behavior patterns. In other words, for each emotional value pattern of the robot 100, the robot 100's actions are defined according to the user's behavior patterns for each of the multiple combinations of the user's past and current emotional values, such as negative and negative, negative and positive, positive and negative, positive and positive, negative and neutral, and neutral and neutral. The action determination unit 236 may also transition to an operation mode in which it determines the robot 100's actions using history data 222 when the user makes an utterance that intends to continue a conversation from a past topic, such as "I want to talk about that topic we talked about before."

[0061] Furthermore, response rule 221 may specify at least one of a gesture and a statement as an action for robot 100 for each of the 1296 patterns of robot 100's emotional value. Alternatively, response rule 221 may specify at least one of a gesture and a statement as an action for robot 100 for each of the groups of patterns of robot 100's emotional value.

[0062] Each gesture included in the actions of robot 100 as defined in response rule 221 has a predetermined intensity. Each utterance included in the actions of robot 100 as defined in response rule 221 has a predetermined intensity.

[0063] For example, response rule 221 defines the actions of robot 100 in response to behavioral patterns such as when the user is concentrating on driving, looking away, driving while distracted, or dozing off. Furthermore, response rule 221 also defines the actions of robot 100 in response to user utterances in each of these cases (e.g., "I'm tired," "I'm sleepy," "I'm lethargic," "I have a headache," etc.).

[0064] The memory control unit 238 decides whether or not to store data including the user's actions in the history data 222, based on the predetermined intensity of the action determined by the action decision unit 236 and the emotion value of the robot 100 determined by the emotion decision unit 232.

[0065] Specifically, if the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gestures included in the actions determined by the action decision unit 236, and the predetermined intensity for the speech content included in the actions determined by the action decision unit 236 is greater than or equal to a threshold, it is decided to store the data including the user's actions in the history data 222.

[0066] When the memory control unit 238 decides to store data including user actions in the history data 222, it stores the actions determined by the action determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time prior (for example, all surrounding information such as sound, images, smells, etc.), and the user's state recognized by the user state recognition unit 230 (for example, the user's facial expressions, emotions, etc.) in the history data 222.

[0067] The behavior control unit 250 controls the controlled object 252 based on the action determined by the action decision unit 236. For example, if the action decision unit 236 determines an action that includes speaking, the behavior control unit 250 causes the speaker included in the controlled object 252 to output sound. At this time, the behavior control unit 250 may determine the speech output speed based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speech output speed the greater the emotion value of the robot 100. In this way, the behavior control unit 250 determines the execution form of the action determined by the action decision unit 236 based on the emotion value determined by the emotion decision unit 232. For example, in situations such as speeding, running a red light, or aggressive driving by the user, the behavior control unit 250 causes the robot 100 to make a statement to calm the user's emotions. Also, for example, in situations such as distracted driving, drowsy driving, or poor physical condition by the user, the behavior control unit 250 causes the robot 100 to make a statement to upset the user's emotions.

[0068] The action control unit 250 may recognize changes in the user's emotions in response to the action decided by the action decision unit 236. For example, it may recognize changes in emotions based on the user's voice or facial expressions. In addition, it may recognize changes in the user's emotions based on the detection of an impact by the touch sensor 207 included in the sensor unit 200. If an impact is detected by the touch sensor 207, it may recognize that the user's emotions have worsened, or if the detection result of the touch sensor 207 indicates that the user is laughing or happy, it may recognize that the user's emotions have improved. Information indicating the user's reaction is output to the communication processing unit 280.

[0069] Furthermore, after the action control unit 250 executes the action determined by the action decision unit 236 in an execution mode determined according to the robot 100's emotions, the emotion decision unit 232 further changes the robot 100's emotion value based on the user's reaction to the execution of the action. Specifically, the emotion decision unit 232 increases the robot 100's "joy" emotion value if the user's reaction to the action performed by the action decision unit 236 in an execution mode determined by the action control unit 250 was not unfavorable. Also, the emotion decision unit 232 increases the robot 100's "sadness" emotion value if the user's reaction to the action performed by the action decided by the action decision unit 236 in an execution mode determined by the action control unit 250 was unfavorable.

[0070] Furthermore, the behavior control unit 250 expresses the emotions of the robot 100 based on the determined emotion values ​​of the robot 100. For example, if the behavior control unit 250 increases the "joy" emotion value of the robot 100, it controls the controlled object 252 to make the robot 100 perform joyful gestures. Also, if the behavior control unit 250 increases the "sadness" emotion value of the robot 100, it controls the controlled object 252 so that the robot 100 assumes a dejected posture.

[0071] Furthermore, if the user continues to drive dangerously, the behavior control unit 250 increases the emotional values ​​of the robot 100, such as "anger" and "displeasure," causing the robot 100 to take actions that indicate genuine anger. For example, if the robot 100 is capable of driving the vehicle 500 on behalf of the user, as described above, the behavior control unit 250 forcibly disables the user's driving operation and performs automatic driving control of the vehicle 500, treating the vehicle 500 as part of the controlled object 252, so that the robot 100 drives the vehicle 500. Similarly, if the behavior control unit 250 determines that the user needs to rest, see a doctor, or make an emergency stop to a safe place due to poor health, etc., it performs automatic driving control of the vehicle 500, treating the vehicle 500 as part of the controlled object 252, so that the robot 100 drives the vehicle 500. An example of a configuration in which the vehicle 500 is part of the controlled object 252 will be described later using Figures 4 and 5.

[0072] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user response information to the server 300. The communication processing unit 280 also receives updated response rules from the server 300. When the communication processing unit 280 receives updated response rules from the server 300, it updates response rule 221. The communication processing unit 280 can send and receive information with the cooperating device 400.

[0073] The server 300 communicates with each robot 100, receives user response information transmitted from the robot 100, and updates the response rules based on the response rules that include actions for which a positive response was obtained.

[0074] Figure 3 is a schematic diagram showing an example of an action flow related to the actions that determine the behavior of the robot 100. The action flow shown in Figure 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module 210 is input. In the action flow, "S" represents the step that is executed.

[0075] First, in step S101, the user state recognition unit 230 recognizes the user's state based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as the user is likely to exceed the speed limit, run a red light, is driving distractedly, appears sleepy, or appears to be unwell, and processes the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as the user is likely to commit a speeding violation.

[0076] In step S102, the emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230.

[0077] In step S103, the emotion determination unit 232 determines an emotion value indicating the robot 100's emotions based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined user emotion value to the history data 222.

[0078] In step S104, the behavior recognition unit 234 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the behavior recognition unit 234 recognizes user behaviors in the vehicle 500 such as "concentrating on driving," "looking away," "driving while distracted," and "dozing off."

[0079] In step S105, the action decision unit 236 determines the action of the robot 100 based on the combination of the user's current emotion value and past emotion values ​​included in the history data 222 determined in step S102, the emotion value of the robot 100, the user's actions recognized by the action recognition unit 234, and the reaction rule 221.

[0080] In step S106, the behavior control unit 250 controls the controlled object 252 based on the action determined by the action decision unit 236. For example, in situations such as speeding, running a red light, or aggressive driving by the user, the behavior control unit 250 causes the robot 100 to make statements to calm the user's emotions. Also, for example, in situations such as distracted driving, drowsy driving, or poor physical condition by the user, the behavior control unit 250 causes the robot 100 to make statements to upset the user. Furthermore, if the behavior control unit 250 determines that automatic driving control of the vehicle 500 is necessary because the user repeatedly engages in dangerous driving or the user is in poor physical condition, it executes automatic driving control of the vehicle 500 so that the robot 100 operates the vehicle 500 as part of the controlled object 252.

[0081] In step S107, the memory control unit 238 calculates a total intensity value based on the predetermined intensity of the action determined by the action decision unit 236 and the emotion value of the robot 100 determined by the emotion decision unit 232.

[0082] In step S108, the memory control unit 238 determines whether the total intensity value is equal to or greater than a threshold. If the total intensity value is less than the threshold, the process is terminated without storing data including the user's actions in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold, the process proceeds to step S109.

[0083] In step S109, the action determined by the action decision unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time prior, and the user's state recognized by the user state recognition unit 230 are stored in the history data 222.

[0084] Next, examples of the arrangement of the robot 100 in the vehicle 500, and an example of the functional configuration of the vehicle 500 that enables automatic driving control by the robot 100 will be explained using Figures 4 and 5. Figure 4 is a diagram showing an example of the arrangement of the robot 100 in the vehicle 500. Figure 5 is a diagram schematically showing the functional configuration of the vehicle 500 that enables automatic driving control by the robot 100.

[0085] When the robot 100 is mounted on the vehicle 500, the robot 100 is provided as a small robot and, as shown in Figure 4, is positioned, for example, near the dashboard or instrument panel inside the vehicle.

[0086] Furthermore, the robot 100 does not have to be a small, separate unit, but may be installed near the seats inside the vehicle. Alternatively, the robot 100 may be installed so that it can sit on the seats of the vehicle 500. In addition, the robot 100 does not have to be a humanoid unit, but may be mounted in the vehicle 500 as part of the vehicle's onboard equipment.

[0087] As shown in Figure 5, the robot 100 is connected to the vehicle control system 600 via an in-vehicle network 20a, which is part of the communication network 20. The in-vehicle network 20a is implemented as a communication network compliant with any standard, such as wireless LAN (Local Area Network), wired LAN, CAN (Controller Area Network), LIN (Local Interconnect Network), or FlexRay (registered trademark).

[0088] The vehicle control system 600 includes a memory unit 601, a sensor system 602, a sensor system ECU (Electronic Control Unit) 602a, a drive system 603, a drive system ECU 603a, a body system 604, a body system ECU 604a, an information and communication system 605, and an information and communication system ECU 605a. Each of the sensor system ECU 602a, drive system ECU 603a, body system ECU 604a, and information and communication system ECU 605a may consist of one or more ECUs.

[0089] The memory unit 601 is implemented by, for example, a memory device such as RAM (Random Access Memory) or flash memory, or a disk device such as a hard disk drive or optical disk drive, and in the example shown in Figure 5, it stores the automatic driving control model 601a. ​​The automatic driving control model 601a corresponds to an example of an AI (Artificial Intelligence) model.

[0090] The automated driving control model 601a is a control model for autonomously driving the vehicle 500, and is, for example, a DNN (Deep Neural Network) model that has been trained to enable automated driving control of the vehicle 500 in accordance with the recognition content based on various sensor data from the sensor system 602. The automated driving control model 601a may also be stored in the robot 100's memory.

[0091] Sensor system 602 is a general term for a group of various sensors mounted on the vehicle 500. These sensors include, for example, cameras, millimeter-wave radar, LiDAR (Light Detection And Ranging), GPS (Global Positioning System) sensors, acceleration sensors, accelerator sensors, brake sensors, and steering sensors. Sensor system 602 may also include biometric sensors to enable the vehicle 500 user to determine if they need to take a break, seek medical attention, or make an emergency stop to a safe location. Sensor system 602 is connected to the in-vehicle network 20a via sensor system ECU 602a.

[0092] The drivetrain 603 is a general term for various devices and mechanisms related to the powertrain. The drivetrain 603 includes, for example, the engine, motor, inverter, battery control device, electric power steering, electric parking brake, etc. The drivetrain 603 is connected to the in-vehicle network 20a via the drivetrain ECU 603a and is controlled by the drivetrain ECU 603a in response to user operations or automatic driving control of the robot 100.

[0093] The body system 604 is a general term for various devices and mechanisms involved in the body control of the vehicle 500. The body system 604 includes, for example, air conditioners, power windows, electric mirrors, electric power seats, and automatic sliding doors. The body system 604 is connected to the in-vehicle network 20a via the body system ECU 604a and is controlled by the body system ECU 604a in response to user operations, automatic driving control of the robot 100, etc.

[0094] The information and communication system 605 is a general term for various devices and mechanisms related to information and communication in the vehicle 500. The information and communication system 605 includes, for example, a car navigation system and HMI (Human Machine Interface) such as a display and speaker. The information and communication system 605 is connected to the in-vehicle network 20a via the information and communication system ECU 605a and is controlled by the information and communication system ECU 605a in response to user operations, automatic driving control of the robot 100, etc.

[0095] The robot 100 is connected to such a vehicle control system 600 and, when performing actions to calm or heighten the user's emotions in order to avoid the aforementioned dangerous driving, it determines whether the user is driving dangerously, for example by taking into account sensor data from the sensor system 602.

[0096] At this time, the sensor module 210 of the robot 100 recognizes the user's operating status based on various sensor data from the sensor system 602 via the sensor system ECU 602a.

[0097] Furthermore, if the user continues to drive dangerously, or if the user needs to take a break, seek medical attention, or make an emergency stop in a safe place due to illness, the robot 100 will perform automatic driving control of the vehicle 500, using the vehicle control system 600 as part of the controlled object 252. At this time, the behavior control unit 250 of the robot 100 will use the automatic driving control model 601a to perform automatic driving control of the vehicle 500 according to the recognition content based on various sensor data from the sensor system 602.

[0098] For example, the behavior control unit 250 inputs sensor data from the sensor system 602 to the automatic driving control model 601a, and generates instruction signals for each ECU based on the output values ​​output from the automatic driving control model 601a accordingly. Then, based on the generated instruction signals, the behavior control unit 250 causes the sensor system ECU 602a to control the sensor system 602, the drive system ECU 603a to control the drive system 603, the body system ECU 604a to control the body system 604, and the information and communication system ECU 605a to control the information and communication system 605. In this way, the robot 100 realizes automatic driving control of the vehicle 500.

[0099] As described above, the robot 100 includes a control unit that recognizes the actions of the user using the vehicle 500, determines its own actions to avoid dangerous driving corresponding to the recognized user actions, and controls the target based on its determined actions. As a result, the robot 100 can take appropriate actions to avoid dangerous driving for the user using the vehicle 500.

[0100] Furthermore, when the control unit of robot 100 recognizes the user's actions in response to dangerous driving, it decides to take actions to calm or heighten the user's emotions. This allows robot 100 to calm or heighten the user's emotions in order to avoid dangerous driving.

[0101] Furthermore, the control unit of the robot 100, upon recognizing user behavior corresponding to at least speeding, running a red light, or aggressive driving, determines an action to calm the user's emotions. This allows the robot 100 to avoid dangerous driving by calming the user's emotions in response to user behavior such as speeding, running a red light, or aggressive driving.

[0102] Furthermore, the control unit of the robot 100, upon recognizing user behaviors such as distracted driving, drowsy driving, or poor physical condition, determines an action to enhance the user's emotions. This allows the robot 100 to avoid dangerous driving by enhancing the user's emotions in response to user behaviors such as distracted driving, drowsy driving, or poor physical condition.

[0103] Furthermore, the control unit of the robot 100 performs navigation to guide the vehicle 500 to a suitable location for resting, seeking medical attention, or making an emergency stop, based on the user's actions, if it determines that a situation requiring rest, medical attention, or an emergency stop is necessary. This allows the robot 100 to avoid dangerous driving in situations requiring rest, medical attention, or an emergency stop.

[0104] Furthermore, the control unit of the robot 100 is equipped to perform automatic driving control, which allows it to operate the vehicle 500 on behalf of the user. This enables the robot 100 to avoid dangerous driving situations by operating the vehicle 500 itself.

[0105] Furthermore, the control unit of the robot 100 forcibly disables the user's driving operations and executes automatic driving control if the user repeatedly engages in dangerous driving. This allows the robot 100 to prevent the vehicle 500 from being endangered by the user's repeated dangerous driving.

[0106] Furthermore, the control unit of the robot 100 performs automatic driving control to move the vehicle 500 to a location suitable for resting or receiving medical attention, or to make an emergency stop in a safe location, if it determines from the user's actions that a situation requiring rest, medical attention, or an emergency stop is necessary. This allows the robot 100 to eliminate situations where the user might drive dangerously due to poor health or other reasons, and to prevent danger to the area around the vehicle 500.

[0107] In the above embodiment, the case in which the robot 100 recognizes the user using the user's facial image was described, but the disclosed technology is not limited to this embodiment. For example, the robot 100 may recognize the user using the user's voice, the user's email address, the user's SNS ID, or an ID card with a built-in wireless IC tag that the user possesses.

[0108] Robot 100 is an example of an electronic device equipped with a behavior control system. The behavior control system is not limited to robot 100, and can be applied to various electronic devices. Furthermore, the functions of server 300 may be implemented by one or more computers. At least some of the functions of server 300 may be implemented by a virtual machine. Also, at least some of the functions of server 300 may be implemented in the cloud.

[0109] Figure 6 is a schematic diagram showing an example of the hardware configuration of a computer 1200 that functions as a robot 100 and a server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus according to this embodiment, or to cause the computer 1200 to execute operations associated with the apparatus according to this embodiment or such one or more "parts", and / or to cause the computer 1200 to execute a process or a stage of such process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0110] The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid-state drive, etc. The computer 1200 also includes input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0111] The CPU 1212 operates according to the programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires the image data generated by the CPU 1212 and stores it in the frame buffer provided in RAM 1214 or within itself, so that the image data is displayed on the display device 1218.

[0112] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0113] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 upon activation. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0114] The program is provided on a computer-readable storage medium such as a DVD-ROM or IC card. The program is read from the computer-readable storage medium and installed on a storage device 1224, RAM 1214, or ROM 1230, which are examples of computer-readable storage media, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of the computer 1200.

[0115] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM, or IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area provided on the recording medium.

[0116] Furthermore, the CPU 1212 may read all or necessary parts of a file or database stored on an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0117] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the multiple entries for an entry that matches the specified condition for the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies the predetermined condition.

[0118] The program or software module described above may be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0119] In this embodiment, blocks in the flowchart and block diagram may represent a stage in a process in which an operation is performed or a "part" of a device that has the role of performing an operation. A particular stage and "part" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, exclusive OR, negated AND, negated OR, and other logical operations, flip-flops, registers, and memory elements.

[0120] A computer-readable storage medium may include any tangible device capable of storing instructions to be executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

[0121] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar languages.

[0122] Computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or a programmable circuit, either locally or via a wide area network (WAN) such as a local area network (LAN) or the internet, so that the computer-readable instructions may be executed by the processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, in order to generate means for performing operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

[0123] (Other embodiments) The robot 100 described above may be mounted on a stuffed animal, or it may be applied to a control device wirelessly or via a wired connection to a controlled device (speaker or camera) mounted on the stuffed animal.

[0124] The emotion determination unit 232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 7).

[0125] Figure 7 shows the emotion map 700, in which multiple emotions are mapped. In the emotion map 700, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Furthermore, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. In this way, in the emotion map 700, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0126] (1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects an emotion at approximately 100 msec, the determination of the robot 100's reaction action (e.g., nodding) may be set to a timing similar to the emotion engine's detection frequency (100 msec), at least in frequency, or it may be set to an earlier timing. The emotion engine's detection frequency may be interpreted as the sampling rate.

[0127] By detecting emotions in approximately 100 msec and immediately responding with a corresponding reaction (e.g., nodding), the reaction becomes natural and allows for a conversation that reads the atmosphere. Robot 100 performs reaction actions (such as nodding) according to the direction and degree (strength) of the emotion mandala in emotion map 700. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms and may be changed depending on the situation (e.g., when playing sports) or the user's age.

[0128] (2) By referring to the emotion map 700, the direction and intensity of the emotion may be set in advance, and the movement and intensity of the nodding may be set. For example, if robot 100 is feeling stable or safe, robot 100 will nod and continue listening. If robot 100 is feeling anxious, confused, or suspicious, robot 100 may tilt its head or stop shaking its head.

[0129] These emotions are distributed at the 3 o'clock position on the Emotion Map 700, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 700, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0130] (3) If Robot 100 feels pleasure from being praised, the filler word "Ah" may be added before the line. If Robot 100 feels pain from receiving harsh words, the filler word "Ugh!" may be added before the line. Additionally, physical reactions such as Robot 100 crouching down while saying "Ugh!" may be included. These emotions are distributed around the 9 o'clock position on emotion map 700.

[0131] (4) In the left half of the Emotion Map 700, internal feelings (reactions) are dominant over situational awareness. Therefore, it may give the impression that the reaction was unintentional.

[0132] When Robot 100 experiences an internal feeling (reaction) of understanding and also feels favorable in its perception of the situation, it may nod deeply while looking at the other person and may say "yes, yes." In this way, Robot 100 may generate balanced favorability towards the other person, that is, behaviors such as acceptance and tolerance. Such emotions are distributed around the 12 o'clock position on emotion map 700.

[0133] Conversely, even when Robot 100 experiences an internal sensation (reaction) of discomfort, in terms of situational awareness, it may shake its head when it feels disgusted, or turn its eye LEDs red and glare at the other party when it feels hatred. Such emotions are distributed around the 6 o'clock position on emotion map 700.

[0134] (5) The inside of the Emotion Map 700 represents what is in your mind, and the outside of the Emotion Map 700 represents your actions. Therefore, the further you go from the outside of the Emotion Map 700, the more visible your emotions become (they are expressed in your actions).

[0135] (6) When listening to someone speak while feeling secure, which is distributed around the 3 o'clock position on the emotion map 700, Robot 100 will only nod slightly and say "hmm hmm," but when it comes to love, which is around the 12 o'clock position, it may nod more strongly, such as a deep nod.

[0136] The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized user state into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 700, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of the information analyzed by the sensor module unit 210, the recognized user state, and the emotion values ​​representing each emotion shown in the emotion map 700. Furthermore, this neural network is trained so that emotions placed close together have similar values, as shown in the emotion map 900 in Figure 8. Figure 8 shows another example of an emotion map. In Figure 8, an example is shown where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0137] Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the user's state recognized by the user state recognition unit 230, and the robot 100's state into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 700, and determines the emotion of the robot 100. This neural network is pre-trained based on multiple training data sets, which are combinations of the information analyzed by the sensor module unit 210, the recognized user's state, the robot 100's state, and the emotion values ​​representing each emotion shown in the emotion map 700. For example, the neural network is trained based on training data that indicates that when the output of the touch sensor 207 recognizes that the robot 100 is being petted by the user, the emotion value for "happy" is "3," or training data that indicates that when the output of the acceleration sensor 205 recognizes that the robot 100 is being hit by the user, the emotion value for "angry" is "3." Furthermore, this neural network learns to assign similar values ​​to emotions that are located close together, as shown in the emotion map 900 in Figure 8.

[0138] The action decision unit 236 generates the robot's actions by adding fixed sentences for asking about the robot's actions corresponding to the user's actions to text representing the user's actions, the user's emotions, and the robot's emotions, and inputting these into a text generation model with dialogue capabilities.

[0139] For example, the action decision unit 236 obtains text representing the state of the robot 100 from the emotions of the robot 100 determined by the emotion decision unit 232, using an emotion table as shown in Figure 9. Figure 9 is a diagram showing an example of an emotion table. In this emotion table, each emotion value is assigned an index number for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

[0140] If the emotion of robot 100 determined by the emotion determination unit 232 corresponds to index number "2", the text "very happy state" is obtained. If the emotion of robot 100 corresponds to multiple index numbers, multiple texts representing the state of robot 100 are obtained.

[0141] Furthermore, a sentiment table, as shown in Figure 10, is prepared for the user's emotions. Figure 10 is a diagram showing an example of the sentiment table. Here, if the user's action is "to speak to AAA", the robot 100's emotion is index number "2", and the user's emotion is index number "3", then the following is input to the text generation model: "The robot is in a very happy state. The user is in a normally happy state. The user has spoken to you with "AAA". As a robot, how should you respond?" and the robot's action content is obtained. The action decision unit 236 determines the robot's action from this action content. Note that "AAA" is the name (nickname) that the user has given to the robot 100.

[0142] In this way, because robot 100 can change its behavior according to an index number corresponding to the robot's emotions, users get the impression that robot 100 has a heart, and are encouraged to take actions such as talking to the robot.

[0143] Furthermore, the action decision unit 236 may generate the robot's actions by adding text representing the user's actions, the user's emotions, and the robot's emotions, as well as text representing the contents of the history data 222, and then adding fixed sentences for asking about the robot's actions corresponding to the user's actions, and inputting these into a text generation model with dialogue capabilities. As a result, the robot 100 can change its actions according to the history data representing the user's emotions and actions, so that the user gets the impression that the robot has a personality and is encouraged to take actions such as talking to the robot. In addition, the history data may also include the robot's emotions and actions.

[0144] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0145] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before" or "prior to," and that these can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," and "next," for convenience, this does not mean that it is essential to perform the operations in that order. [Explanation of Symbols]

[0146] 1. Control System 20 Communication Network 100 robots 200 Sensor Unit 201 Mike 202 3D depth sensor 203 2D Camera 204 Distance Sensor 210 Sensor Module Section 211 Voice Emotion Recognition Unit 212 Speech comprehension section 213 Facial expression recognition unit 214 Face Recognition Unit 220 Storage Unit 221 Reaction Rules 222 Historical data 230 User State Recognition Unit 232 Emotion Determination Department 234 Behavior Recognition Department 236 Decision-Making Department 238 Memory Control Unit 250 Action Control Unit 252 Controlled object 280 Communication Processing Unit 300 servers 400 Interconnected Devices 500 vehicles 600 Vehicle Control System

Claims

1. A control unit that recognizes the actions of a user using a vehicle, determines its own actions to avoid dangerous driving corresponding to the recognized user actions, and controls the controlled object based on its own determined actions. Equipped with, The control unit, Using the emotion engine that determines the user's emotions and the user's own emotions, and a text generation model with dialogue capabilities, The recognized user behavior and the text representing the user's emotions and the user's own emotions as determined by the emotion engine are combined with a fixed sentence to ask about the user's actions corresponding to their own actions, and these are input into the text generation model. The user then determines their own actions according to the content output from the text generation model in response to this input. The control unit further, The system is configured to perform automatic driving control, which operates the vehicle on behalf of the user. If the user repeatedly engages in dangerous driving, the system increases the user's "anger" or "displeasure" emotional level, and if this emotional level exceeds a predetermined threshold, the system forcibly disables the user's driving operations and executes the automatic driving control. electronic equipment.

2. The control unit, When the system recognizes the user's actions in response to the dangerous driving described above, it decides to take action to calm or heighten the user's emotions. The electronic device according to claim 1.

3. The control unit, If the system recognizes the user's actions, such as speeding, running a red light, or aggressive driving, it will decide to take action to calm the user's emotions. The electronic device according to claim 2.

4. The control unit, When the system recognizes the user's actions, such as distracted driving, drowsy driving, or poor physical condition, it decides to take actions that will upset the user. The electronic device according to claim 2.

5. The control unit, If, based on the user's actions, it is determined that a situation requiring a rest, medical consultation, or emergency stop is necessary, the system will perform navigation to guide the vehicle to a location suitable for such a rest, medical consultation, or emergency stop. The electronic device according to claim 4.

6. The control unit, If, based on the user's actions, it is determined that a situation requiring a rest, medical consultation, or emergency stop is necessary, the system will execute the automatic driving control to move the vehicle to a location suitable for the rest or medical consultation, or to bring the vehicle to a safe emergency stop. The electronic device according to claim 1.

7. It is mounted on the plush toy, or connected wirelessly or via wire to a controlled device mounted on the plush toy. The electronic device according to claim 1.

Images