Behavior control system

Abstract

To ensure a position as a friendly partner with a user by causing an empathy talking when providing information, in addition to an essential purpose to provide the information through speech.SOLUTION: A behavior control system includes: an input unit that accepts input of sentences; a processing unit that performs specific processing using a sentence generation model on the sentences accepted by the input unit; and an output unit that controls a behavior of a robot to output results of the specific processing. The processing unit identifies emotion transfer points by analyzing a sentence form including word decomposition and phrase connection of the sentences input from the input unit, converts the sentences into speech information for speaking, and assigns an emotional expression including at least one of high / low sound, speed, sound intensity, and inflection to each of the emotion transfer points when speaking based on the converted speech information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a behavior control system. [Background technology]

[0002] Conventionally, in an agent (electronic device such as a robot, smartphone, or smart glasses) that speaks sentences, if the sentence itself is appropriate, it can be said that the agent is fulfilling its role sufficiently.

[0003] Patent document 1 describes a machine translation system including an acquisition step of acquiring pre-translation text information generated by converting first speech data representing input speech spoken in a first language into text information, a determination step of determining whether the pre-translation text information acquired in the acquisition step includes first specific text information stored in a storage unit, and an output step of outputting at least one of second specific text information stored in the storage unit and associated with the first specific text information and second speech data of the second specific text information if it is determined in the determination step that the pre-translation text information includes the first specific text information.

[0004] While Patent Document 1 functions adequately as a machine translation system, for example, when speaking, the pitch, speed, accent, and intonation of the sound are all at a roughly constant level, and there is no concept of empathy with the speech.

[0005] For reference, Patent Document 2 discloses a technology for determining an appropriate robot behavior for a user's state. The conventional technology of Patent Document 2 recognizes the user's reaction when the robot executes a specific action, and if the robot cannot determine an action for the recognized user's reaction, it updates the robot's behavior by receiving information on an action suitable for the recognized user's state from a server. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-120616 A [Patent Document 2] Patent No. 6053847 Summary of the Invention [Problem to be solved by the invention]

[0007] However, for example, when considering an agent (especially an interactive agent such as a robot) speaking as a conversation partner with a lonely user, the robot may be required to be positioned as a so-called partner between the user, but the user may not be satisfied with just the appropriateness of the sentences themselves, and the robot may not be able to become a partner.

[0008] Taking the above facts into consideration, the present invention aims to provide a behavior control system that, in addition to the essential objective of providing information through speech, can establish itself as a friendly partner with the user by having the user empathize and speak when providing the information. [Means for solving the problem]

[0009] According to a first aspect of the present invention, there is provided a behavior control system. The behavior control system includes an input unit that accepts input of a sentence, a processing unit that performs specific processing on the sentence accepted by the input unit using a sentence generation model, and an output unit that controls the behavior of the robot to output the results of the specific processing. The processing unit has a sentence morphology analysis unit that identifies points of empathy by analyzing the sentence morphology including word breakdown and phrase connections of the sentence input from the input unit, an utterance conversion unit that converts the sentence into speech information for speaking, and an emotional expression assignment unit that assigns an emotional expression including at least one of pitch, speed, intensity, and intonation to each of the points of empathy when speaking based on the speech information converted by the utterance conversion unit.

[0010] The behavior control system is characterized in that the processing section further has a sentence-pronunciation database in which sentences and pronunciations of the sentences are associated and stored by the speech conversion section.

[0011] The behavior control system is characterized in that the processing unit further has an emotional expression database in which predetermined emotional expression patterns are stored as emotional expressions for shifting the emotions of the user, who is the target of the speech, to a more positive side.

[0012] The behavior control system is characterized in that the emotional expression database stores at least one of a first emotional expression pattern which automatically assigns an appropriate emotion to an input sentence from a sentence input and uses the input sentence, a second emotional expression pattern which uses an emotion assigned by voice emotion recognition from a voice input and a sentence obtained by voice speech recognition, a third emotional expression pattern which uses an emotion obtained by facial emotion recognition from an image captured by a camera and a voice input and a sentence obtained by voice speech recognition, and a fourth emotional expression pattern obtained by manual input of a sentence and an emotion or a sentence and a person's image.

[0013] The behavior control system is characterized in that the sentence input to the input unit is either a sentence generated based on voice input when the user speaks, a sentence generated by scanning a text sentence on a paper medium, etc., or a sentence generated based on key input from an input device.

[0014] The electronic device may be a robot, where the robot includes a device that performs a physical action, a device that outputs video and audio without performing a physical action, and an agent that operates on software. [Brief description of the drawings]

[0015] [Figure 1] 1 illustrates a schematic diagram of an example of a system 5 according to the present embodiment. [Diagram 2]2 shows a schematic functional configuration of a robot 100 according to the present embodiment. [Diagram 3] 2 shows a schematic functional configuration of a specific processing unit of the robot 100 according to the present embodiment. [Figure 4] 4 is a functional block diagram in which the processing of a specific processing unit shown in FIG. 3 is classified by function. [Diagram 5] 13 is an operational flowchart relating to an operation of a robot performing a specific process (sentence emotion generation control) for empathizing with an utterance for a sentence. [Figure 6] (A) is a setting value chart that quantifies the pitch, speed (tempo), accent, and intonation of the sound when the original sentence is spoken by the robot, and (B) is a level characteristic chart when quantified. [Figure 7] FIG. 11 is a functional block diagram in which the processing of a specific processing unit according to a modified example is classified by function. [Figure 8] 1 illustrates an example of a hardware configuration of a computer 1200. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0017] (Robot Overview)

[0018] As shown in FIG. 1, the system 5 includes a robot 100, a robot 101, a robot 102, and a server 300. A user 10a, a user 10b, a user 10c, and a user 10d are users of the robot 100. A user 11a, a user 11b, and a user 11c are users of the robot 101. A user 12a and a user 12b are users of the robot 102. In the description of this embodiment, the user 10a, the user 10b, the user 10c, and the user 10d may be collectively referred to as the user 10. The user 11a, the user 11b, and the user 11c may be collectively referred to as the user 11. The user 12a and the user 12b may be collectively referred to as the user 12. The robot 101 and the robot 102 have substantially the same functions as the robot 100. Therefore, the system 5 will be described mainly with reference to the functions of the robot 100.

[0019] The robot 100 converses with the user 10 and provides images to the user 10. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via a communication network 20 to converse with the user 10 and provide images, etc. to the user 10. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user 10. The robot 100 also causes the server 300 to record captured image data, etc. of the user 10, and requests the server 300 for the image data, etc. as necessary, and provides the data, etc. to the user 10.

[0020] The robot 100 also has an emotion value that indicates the type of its own emotion. For example, the robot 100 has emotion values ​​that indicate the intensity of each of the emotions of "joy", "anger", "sorrow", "pleasure", "discomfort", "relief", "anxiety", "sadness", "excitement", "concern", "relief", "fulfillment", "emptiness", and "neutral". When the robot 100 converses with the user 10 in a state where the emotion value of excitement is high, for example, the robot 100 utters sounds at a fast speed. In this way, the robot 100 can express its own emotion through actions.

[0021] Furthermore, the robot 100 may be configured to match a sentence generation model using AI (Artificial Intelligence) with an emotion engine, thereby determining the behavior of the robot 100 corresponding to the emotion of the user 10. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 corresponding to the determined emotion.

[0022] More specifically, when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates the behavioral content that the robot 100 should take in response to the behavior of the user 10, using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is described in, for example, JP 2018-081444 A and ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and thus detailed description thereof will be omitted. Such a sentence generation model is configured using a large language model (LLM).

[0023] As described above, in this embodiment, by combining a large-scale language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100. In other words, according to this embodiment, by combining a sentence generation model and an emotion engine, a synergistic effect can be obtained.

[0024] The robot 100 also has a function of recognizing the behavior of the user 10. The robot 100 recognizes the behavior of the user 10 by analyzing a facial image of the user 10 acquired by a camera function and a voice of the user 10 acquired by a microphone function. The robot 100 determines the behavior to be performed by the robot 100 based on the recognized behavior of the user 10, etc.

[0025] The robot 100 stores, as an example of a behavior decision model, rules that define the behaviors that the robot 100 will execute based on the emotions of the user 10, the emotions of the robot 100, and the behaviors of the user 10, and performs various behaviors according to the rules.

[0026] Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the emotion of the user 10, the emotion of the robot 100, and the behavior of the user 10, as an example of a behavior decision model. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the behavior of the user 10 is "laughing". In addition, the reaction rules define the behavior of the robot 100 as "apologizing" when the behavior of the user 10 is "angry". In addition, the reaction rules define the behavior of the robot 100 as "answering" when the behavior of the user 10 is "asking a question". In addition, the reaction rules define the behavior of the robot 100 as "calling out" when the behavior of the user 10 is "sad".

[0027] When the robot 100 recognizes the behavior of the user 10 as "anger" based on the reaction rules, the robot 100 selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", the robot 100 performs the motion of "apologizing" and outputs a voice expressing the word "apologize".

[0028] In addition, when the emotion of the robot 100 is "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, and "happy" = 0) and the condition that the state of the user 10 is "alone and looks lonely" is satisfied, it is defined that the emotion of the robot 100 will change to "worried" and that the robot 100 will be able to execute the action of "calling out."

[0029] When the robot 100 recognizes based on the reaction rule that the current emotion of the robot 100 is "normal" and that the user 10 is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rule as the action to be performed toward the user 10. For example, when the robot 100 selects the action of "calling out", the robot 100 converts the words "What's wrong?", which indicate concern, into a worried voice and outputs it.

[0030] The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user 10. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the reaction of the user 10 was positive, and attributes of the user 10.

[0031] The server 300 stores the user reaction information received from the robot 100. The server 300 receives and stores user reaction information not only from the robot 100, but also from each of the robots 101 and 102. The server 300 analyzes the user reaction information from the robots 100, 101, and 102, and updates the reaction rules.

[0032] The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate the reaction rules acquired by the robot 101, the robot 102, etc. into its own reaction rules.

[0033] 2A shows a schematic functional configuration of the robot 100. The robot 100 has a sensor unit 200, a sensor module unit 210, a storage unit 220, a control unit 228, and a control target 252. The control unit 228 has a state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a related information collection unit 270, a communication processing unit 280, and a specific processing unit 290.

[0034] The control target 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

[0035] The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, a touch sensor 205, and an acceleration sensor 206. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The distance sensor 204 detects the distance to an object by irradiating, for example, a laser or ultrasonic waves. The sensor unit 200 may also include a clock, a gyro sensor, a sensor for motor feedback, and the like.

[0036] 2A, the components of the robot 100 excluding the control target 252 and the sensor unit 200 are examples of components of a behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

[0037] The storage unit 220 includes a behavioral decision model 221, history data 222, collected data 223, and behavioral schedule data 224. The history data 222 includes the past emotional values ​​of the user 10, the past emotional values ​​of the robot 100, and behavioral history, and specifically includes a plurality of event data including the emotional values ​​of the user 10, the emotional values ​​of the robot 100, and the behavior of the user 10. The data including the behavior of the user 10 includes a camera image representing the behavior of the user 10. The emotional values ​​and behavioral history are recorded for each user 10, for example, by being associated with the identification information of the user 10. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the face image of the user 10, attribute information of the user 10, and the like. Note that the functions of the components of the robot 100 shown in FIG. 2A, excluding the control target 252, the sensor unit 200, and the storage unit 220, can be realized by a CPU operating based on a program. For example, the functions of these components can be implemented as CPU operations using an operating system (OS) and a program that runs on the OS.

[0038] The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200, and outputs the analysis result to the state recognition unit 230.

[0039] The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the voice of the user 10 detected by the microphone 201 and recognizes the emotion of the user 10. For example, the voice emotion recognition unit 211 extracts feature amounts such as frequency components of the voice and recognizes the emotion of the user 10 based on the extracted feature amounts. The speech understanding unit 212 analyzes the voice of the user 10 detected by the microphone 201 and outputs text information representing the content of the utterance of the user 10.

[0040] The facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 from the image of the user 10 captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the facial expression and emotion of the user 10 based on the shapes, positional relationship, etc. of the eyes and mouth.

[0041] The face recognition unit 214 recognizes the face of the user 10. The face recognition unit 214 recognizes the user 10 by matching a face image stored in a person DB (not shown) with a face image of the user 10 captured by the 2D camera 203.

[0042] The state recognition unit 230 recognizes the state of the user 10 based on the information analyzed by the sensor module unit 210. For example, the state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, it generates sensory information such as "Daddy is alone" or "There is a 90% chance that Daddy is not smiling." It performs processing to understand the meaning of the generated sensory information. For example, it generates semantic information such as "Daddy is alone and looks lonely."

[0043] The state recognition unit 230 recognizes the state of the robot 100 based on the information detected by the sensor unit 200. For example, the state recognition unit 230 recognizes the remaining battery level of the robot 100, the brightness of the surrounding environment of the robot 100, and the like as the state of the robot 100.

[0044] The emotion determination unit 232 determines an emotion value indicating the emotion of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network to obtain an emotion value indicating the emotion of the user 10.

[0045] Here, the emotion value indicating the emotion of the user 10 is a value indicating the positive or negative emotion of the user. For example, if the emotion of the user is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy", "pleasure", "comfort", "relief", "excitement", "relief", and "fulfillment", the value indicates a positive value, and the more cheerful the emotion, the larger the value. If the emotion of the user is an unpleasant emotion, such as "anger", "sorrow", "discomfort", "anxiety", "sorrow", "worry", and "emptiness", the value indicates a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value. If the emotion of the user is not any of the above ("neutral"), the value indicates 0.

[0046] In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210, the information detected by the sensor unit 200, and the state of the user 10 recognized by the state recognition unit 230.

[0047] The emotion value of the robot 100 includes emotion values ​​for each of a plurality of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of "happiness," "anger," "sorrow," and "pleasure."

[0048] Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 in accordance with a rule for updating the emotion value of the robot 100, which rule is determined in association with the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230.

[0049] For example, when the state recognition unit 230 recognizes that the user 10 looks lonely, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100. Also, when the state recognition unit 230 recognizes that the user 10 is smiling, the emotion determination unit 232 increases the emotion value of "joy" of the robot 100.

[0050] The emotion determining unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is pitch black, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when the user 10 continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.

[0051] The behavior recognition unit 234 recognizes the behavior of the user 10 based on the information analyzed by the sensor module unit 210 and the state of the user 10 recognized by the state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized state of the user 10 are input to a pre-trained neural network, the probability of each of a plurality of predetermined behavior classifications (for example, "laughing," "getting angry," "asking a question," and "getting sad") is obtained, and the behavior classification with the highest probability is recognized as the behavior of the user 10.

[0052] As described above, in this embodiment, the robot 100 identifies the user 10 and acquires the contents of the user's utterance. When acquiring and using the contents of the utterance, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user 10. The behavior control system of the robot 100 in this embodiment takes into consideration the protection of the personal information and privacy of the user 10.

[0053] Next, the process of the behavior decision unit 236 when the robot 100 performs a response process to respond to the behavior of the user 10 will be described.

[0054] The behavior determining unit 236 determines a behavior corresponding to the behavior of the user 10 recognized by the behavior recognizing unit 234 based on the current emotion value of the user 10 determined by the emotion determining unit 232, the history data 222 of past emotion values ​​determined by the emotion determining unit 232 before the current emotion value of the user 10 was determined, and the emotion value of the robot 100. In this embodiment, the behavior determining unit 236 uses one most recent emotion value included in the history data 222 as the past emotion value of the user 10, but the disclosed technology is not limited to this aspect. For example, the behavior determining unit 236 may use a plurality of most recent emotion values ​​as the past emotion value of the user 10, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determining unit 236 may determine a behavior corresponding to the behavior of the user 10 by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values ​​of the robot 100. The behavior determined by the behavior determining unit 236 includes a gesture performed by the robot 100 or the contents of the speech of the robot 100 .

[0055] The behavior decision unit 236 according to this embodiment decides the behavior of the robot 100 as a behavior corresponding to the behavior of the user 10, based on a combination of the past emotion value and the current emotion value of the user 10, the emotion value of the robot 100, the behavior of the user 10, and the behavior decision model 221. For example, when the past emotion value of the user 10 is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides, as a behavior corresponding to the behavior of the user 10, a behavior for changing the emotion value of the user 10 to a positive value.

[0056] The reaction rules as the behavior decision model 221 prescribe the behavior of the robot 100 according to a combination of the past emotional value and the current emotional value of the user 10, the emotional value of the robot 100, and the behavior of the user 10. For example, when the past emotional value of the user 10 is a positive value and the current emotional value is a negative value, and the behavior of the user 10 is sad, a combination of gestures and speech content when asking a question to encourage the user 10 with gestures is prescribe as the behavior of the robot 100.

[0057] For example, in the reaction rule as the behavior determination model 221, the behavior of the robot 100 is determined for all combinations of the patterns of the emotion values ​​of the robot 100 (1296 patterns, which are the fourth power of six values ​​of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); the combination patterns of the past emotion values ​​and the current emotion values ​​of the user 10; and the behavior patterns of the user 10. That is, for each pattern of the emotion values ​​of the robot 100, the behavior of the robot 100 is determined for each of a plurality of combinations of the past emotion values ​​and the current emotion values ​​of the user 10, such as negative values ​​and negative values, negative values ​​and positive values, positive values ​​and negative values, positive values ​​and positive values, negative values ​​and normal values, and normal values ​​and normal values. Note that, when the user 10 makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time", the behavior determination unit 236 may transition to an operation mode in which the behavior of the robot 100 is determined using the history data 222.

[0058] Note that the reaction rules as the behavior determination model 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100, up to one for each of the patterns (1296 patterns) of the emotion value of the robot 100. Alternatively, the reaction rules as the behavior determination model 221 may prescribe at least one of a gesture and a statement as the behavior of the robot 100, for each group of patterns of the emotion value of the robot 100.

[0059] The strength of each gesture included in the behavior of the robot 100 defined in the reaction rules as the behavior determination model 221 is determined in advance. The strength of each utterance content included in the behavior of the robot 100 defined in the reaction rules as the behavior determination model 221 is determined in advance.

[0060] The memory control unit 238 determines whether or not to store data including the behavior of the user 10 in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

[0061] Specifically, when a total intensity value, which is the sum of the sum of the emotion values ​​for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the behavior of the user 10 is to be stored in the history data 222.

[0062] When the memory control unit 238 decides to store data including the behavior of the user 10 in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, smells, etc. of the scene), and the state of the user 10 recognized by the state recognition unit 230 (e.g., the facial expression, emotions, etc. of the user 10).

[0063] The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

[0064] The behavior control unit 250 may recognize a change in the user's 10 emotion in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotion may be recognized based on the voice or facial expression of the user 10. Alternatively, the change in emotion may be recognized based on the detection of an impact by the touch sensor 205 included in the sensor unit 200. When an impact is detected by the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having worsened, or when the reaction of the user 10 is determined to be laughing or happy based on the detection result of the touch sensor 205 included in the sensor unit 200, the emotion of the user 10 may be recognized as having improved. Information indicating the reaction of the user 10 is output to the communication processing unit 280.

[0065] Furthermore, after the behavior control unit 250 executes the behavior determined by the behavior determining unit 236 in the execution mode determined according to the emotion of the robot 100, the emotion determining unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determining unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is not bad. Furthermore, the emotion determining unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determining unit 236 being performed on the user in the execution mode determined by the behavior control unit 250 is bad.

[0066] Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, the behavior control unit 250 controls the control object 252 to make the robot 100 assume a droopy posture.

[0067] The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. In addition, the communication processing unit 280 receives updated reaction rules from the server 300. When the communication processing unit 280 receives the updated reaction rules from the server 300, it updates the reaction rules as the behavioral determination model 221.

[0068] The server 300 communicates between the robots 100, 101, and 102 and the server 300, receives user reaction information transmitted from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

[0069] The related information collecting section 270 collects information related to the favorite information acquired about the user 10 from external data (websites such as news sites and video sites) at a predetermined timing based on the favorite information acquired about the user 10.

[0070] Specifically, the related information collecting unit 270 acquires preference information indicating matters of interest to the user 10 from the contents of speech of the user 10 or a setting operation by the user 10. The related information collecting unit 270 periodically collects news related to the preference information through ChatGPT Plugins (Internet search).<URL: https: / / openai.com / blog / chatgpt-plugins> For example, if the user 10 is a fan of a specific professional baseball team, the related information collection unit 270 collects news related to the game results of the specific professional baseball team from external data using ChatGPT Plugins at a predetermined time every day.

[0071] The emotion determining unit 232 determines the emotion of the robot 100 based on information related to the preference information collected by the related information collecting unit 270.

[0072] Specifically, the emotion determining unit 232 inputs text representing information related to the preference information collected by the related information collecting unit 270 into a pre-trained neural network for determining emotions, obtains emotion values ​​indicating each emotion, and determines the emotion of the robot 100. For example, when the collected news related to the game results of a specific professional baseball team indicates that the specific professional baseball team won, the emotion determining unit 232 determines that the emotion value of "joy" of the robot 100 is large.

[0073] When the emotion value of the robot 100 is equal to or greater than the threshold value, the memory control unit 238 stores information related to the preference information collected by the related information collection unit 270 in the collected data 223.

[0074] Next, the processing of the behavior decision unit 236 when the robot 100 performs autonomous processing to act autonomously will be described.

[0075] The behavior decision unit 236 determines, at a predetermined timing, one of a plurality of types of robot behaviors including no behavior, as the behavior of the robot 100, using at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and the behavior decision model 221. Here, a case where a sentence generation model having a dialogue function is used as the behavior decision model 221 will be described as an example.

[0076] Specifically, the behavior determination unit 236 inputs text representing at least one of the state of the user 10, the emotion of the user 10, the emotion of the robot 100, and the state of the robot 100, and text asking about the robot's behavior, into a sentence generation model, and determines the behavior of the robot 100 based on the output of the sentence generation model.

[0077] For example, the multiple types of robot behaviors include the following (1) to (10). (1) The robot does nothing. (2) Robots dream. (3) The robot talks to the user. (4) The robot creates a picture diary. (5) The robot suggests activities. (6) The robot will suggest people for the user to meet. (7) The robot introduces news that is of interest to the user. (8) The robot will edit photos and videos. (9) The robot learns together with the user. (10) Robots evoke memories.

[0078] The behavior determination unit 236 inputs the state of the user 10 and the state of the robot 100 recognized by the state recognition unit 230, the current emotion value of the user 10 determined by the emotion determination unit 232, text indicating the current emotion value of the robot 100, and text asking about one of a plurality of types of robot behaviors including not taking any action, into the sentence generation model at regular time intervals, and determines the behavior of the robot 100 based on the output of the sentence generation model. Here, when the user 10 is not present around the robot 100, the text input to the sentence generation model does not need to include the state of the user 10 and the current emotion value of the user 10, and may include an indication that the user 10 is not present.

[0079] As an example, in response to the question, "The robot is in a very happy state. The user is in a normal happy state. The user is sleeping. Which of the following (1) to (10) is the best behavior for the robot?", the text "(1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user..." is input to the sentence generation model. Based on the output of the sentence generation model, "It can be said that either (1) doing nothing or (2) the robot dreams is the most appropriate behavior," the behavior of the robot 100 is determined to be "(1) doing nothing" or "(2) the robot dreams."

[0080] As another example, in response to the question "The robot is in a slightly lonely state. The user is not present. The robot's surroundings are dark. Which of the following (1) to (10) is the best behavior for the robot?", the text "(1) The robot does nothing. (2) The robot dreams. (3) The robot talks to the user..." is input to the sentence generation model. Based on the output of the sentence generation model "It can be said that either (2) The robot dreams or (4) The robot creates a picture diary is the most appropriate behavior," the behavior of the robot 100 is determined to be "(2) The robot dreams" or "(4) The robot creates a picture diary."

[0081] When the behavior decision unit 236 decides to create an original event, that is, "(2) The robot dreams," as the robot behavior, it uses the sentence generation model to create an original event that combines multiple event data from the history data 222. At this time, the storage control unit 238 stores the created original event in the history data 222.

[0082] When the behavior decision unit 236 decides that the robot 100 will speak, i.e., "(3) The robot speaks to the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the robot's utterance contents corresponding to the user state and the user's emotion or the robot's emotion. At this time, the behavior control unit 250 outputs a sound representing the determined robot's utterance contents from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined robot's utterance contents in the behavior schedule data 224 without outputting a sound representing the determined robot's utterance contents.

[0083] When the behavior decision unit 236 decides that "(7) The robot introduces news that the user is interested in" as the robot behavior, it uses the sentence generation model to decide the robot's utterance contents corresponding to the information stored in the collected data 223. At this time, the behavior control unit 250 outputs the sound representing the determined robot's utterance contents from the speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined robot's utterance contents in the behavior schedule data 224 without outputting the sound representing the determined robot's utterance contents.

[0084] When the behavior decision unit 236 determines that the robot 100 will create an event image, i.e., "(4) The robot creates a picture diary," as the robot behavior, the behavior decision unit 236 generates an image representing the event data for the event data selected from the history data 222 using an image generation model, generates an explanatory text representing the event data using a text generation model, and outputs a combination of the image representing the event data and the explanatory text representing the event data as an event image. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the event image in the behavior schedule data 224 without outputting the event image.

[0085] When the behavior decision unit 236 determines that the robot behavior is "(8) The robot edits photos and videos," i.e., that the robot edits images, it selects event data from the history data 222 based on the emotion value, and edits and outputs the image data of the selected event data. Note that when the user 10 is not present around the robot 100, the behavior control unit 250 stores the edited image data in the behavior schedule data 224 without outputting the edited image data.

[0086] When the behavior decision unit 236 determines that the robot behavior is "(5) The robot proposes an activity," that is, that the robot proposes an action for the user 10, the behavior decision unit 236 uses a sentence generation model to decide the user's behavior to be proposed based on the event data stored in the history data 222. At this time, the behavior control unit 250 outputs a voice proposing the user's behavior from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores in the behavior schedule data 224 that the user's behavior is proposed without outputting a voice proposing the user's behavior.

[0087] When the behavior decision unit 236 determines that the robot behavior is "(6) The robot proposes a person who the user should meet," that is, a person who should have contact with the user 10, the behavior decision unit 236 uses a sentence generation model based on the event data stored in the history data 222 to determine a person who should have contact with the proposed user. At this time, the behavior control unit 250 outputs a sound representing the proposal of a person who should have contact with the user from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores in the behavior schedule data 224 the proposal of a person who should have contact with the user without outputting a sound representing the proposal of a person who should have contact with the user.

[0088] When the behavior decision unit 236 decides that the robot 100 will make an utterance related to studying, that is, "(9) The robot studies together with the user," as the robot behavior, the behavior decision unit 236 uses the sentence generation model to decide the content of the robot's utterance to encourage studying, give study questions, or give advice on studying, corresponding to the user's state and the user's or the robot's emotion. At this time, the behavior control unit 250 outputs a sound representing the determined content of the robot's utterance from a speaker included in the control target 252. Note that, when the user 10 is not present around the robot 100, the behavior control unit 250 stores the determined content of the robot's utterance in the behavior schedule data 224 without outputting a sound representing the determined content of the robot's utterance.

[0089] When the behavior determining unit 236 determines that the robot behavior is "(10) The robot recalls a memory," that is, that the robot recalls event data, it selects the event data from the history data 222. At this time, the emotion determining unit 232 judges the emotion of the robot 100 based on the selected event data. Furthermore, the behavior determining unit 236 uses a sentence generation model based on the selected event data to create an emotion change event that represents the utterance content and behavior of the robot 100 for changing the user's emotion value. At this time, the memory control unit 238 stores the emotion change event in the behavior schedule data 224.

[0090] For example, the fact that the video the user was watching was about pandas is stored as event data in the history data 222. When the event data is selected, "On the topic of pandas, what are some things you should say to the user the next time you meet them? Name three." is input to the sentence generation model. If the output of the sentence generation model is "(1) Let's go to the zoo, (2) Let's draw a picture of a panda, (3) Let's go buy a stuffed panda," the robot 100 inputs "Which of (1), (2), and (3) would make the user happiest?" to the sentence generation model. If the output of the sentence generation model is "(1) Let's go to the zoo," the robot 100 will say "(1) Let's go to the zoo" the next time it meets the user. This is created as an emotion change event and stored in the action schedule data 224.

[0091] Also, for example, event data with a large emotion value of the robot 100 is selected as an impressive memory of the robot 100. In this way, an emotion change event can be created based on the event data selected as an impressive memory.

[0092] When the behavior decision unit 236 detects an action of the user 10 toward the robot 100 from a state in which the user 10 is not taking any action toward the robot 100 based on the state of the user 10 recognized by the state recognition unit 230, the behavior decision unit 236 reads the data stored in the action schedule data 224 and decides the behavior of the robot 100.

[0093] For example, when the user 10 is not present around the robot 100 and the user 10 is detected, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100. Also, when the user 10 is asleep and the behavior decision unit 236 detects that the user 10 has woken up, the behavior decision unit 236 reads out the data stored in the behavior schedule data 224 and decides the behavior of the robot 100.

[0094] (Empathy with the input text) Next, the process of the specific processing unit 290 that functions as an emotional expression generation control unit when the robot 100 performs specific processing for adding an emotional expression to an input sentence will be described.

[0095] As shown in FIG. 3, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0096] The input unit 292 accepts input of text as information to be conveyed to the user. Specifically, the following input forms are possible. Note that the text is not limited to Japanese (first language) and may be in a foreign language (second language), and may be equipped with a machine translation function for translating from the first language to the second language and from the second language to the first language, as necessary.

[0097] (Input type 1) Sentences generated based on the user's speech input (Input form 2) Text generated by scanning text on paper or other media (Input form 3) Text generated based on key input from an input device (keyboard, touch panel, etc.)

[0098] The processing unit 294 performs a specific process using a sentence generation model.

[0099] In this embodiment, the processing unit 294 of the specific processing unit 290 is intended to, when a sentence to be output (spoken) by the robot 100 is input, empathize with the input sentence (described in detail later) and then output (spoke) the sentence.

[0100] The empathy in the specific processing unit 290 according to this embodiment is characterized in that, in addition to the essential purpose of providing the information to be conveyed to the user through the speech of the robot 100, which is entrusted to the text, it also establishes a position for the robot as a friendly partner between the user and the robot.

[0101] The output unit 296 outputs the processing result (information on the sentence to which the emotional expression has been added) in the processing unit 294. That is, the information on the sentence to which the emotional expression has been added is sent to the control target 252 (see FIG. 2) via the output unit 296 and spoken.

[0102] Specifically, empathy is an inherent function of the robot 100 described above, in which an emotional value representing the intensity of each of the emotions of "joy," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sadness," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral" is determined, and empathetic voice is output from a speaker belonging to the control object 252 (see FIG. 2).

[0103] In particular, in this embodiment, it is assumed that the robot 100 will speak as a conversation partner for a lonely user, and it is designed to empathize with the user by prioritizing shifting the user's emotions to the positive side.

[0104] More specifically, by controlling the pitch, speed, accent, and intonation of speech based on the emotion value, the system can bring a user's negative emotions back to normal (level 0) or shift them to a more positive state.

[0105] Fig. 4 is a functional block diagram in which the processing of specific processing unit 290 shown in Fig. 3 is classified by function. Note that each block in Fig. 4 is classified by function and does not limit the hardware configuration of specific processing unit 290. Also, some or all of the functions may be operated as so-called software (a program that runs on a CPU).

[0106] As shown in FIG. 4, when a sentence to be empathized with is input to the input unit 292 , the sentence is sent to a sentence form analysis unit 300 in the processing unit 294 .

[0107] The sentence morphology analysis unit 300 classifies sentences by phrase and word, and comprehensively analyzes the meanings and connections of each classified phrase and word, as well as the usage patterns of particles and conjunctions used in the phrases, to identify parts where empathy should be achieved (parts of empathy). The sentence morphology analysis unit 300 also quantifies the degree of empathy (pronunciation) at each identified part. Note that quantification is just one example, and empathy may be identified in language, such as "raising the ending," "using a dialect," "feeling spoiled," "leaving a short pause," or "adding a word or phrase not in the sentence," with automatic generation of emotions in a sentence generation model such as ChatGPT in mind.

[0108] The sentence morphology analysis unit 300 is connected to the speech conversion unit 302. The sentence morphology analysis unit 300 sends information about the sentence and the analysis result to the speech conversion unit 302.

[0109] The speech conversion unit 302 retrieves pronunciation information for speaking a sentence from a sentence-pronunciation database 304, and generates a sound source for speaking the sentence (speech sound source).

[0110] The speech conversion unit 302 is connected to the emotional expression adding unit 306. The speech conversion unit 302 sends to the emotional expression adding unit 306 an utterance sound source corresponding to the sentence and the analysis result.

[0111] The emotional expression adding unit 306 adds an emotional expression to each part of the speech sound source that should be emotionally affected, based on the analysis result, from the emotional expression database 308 .

[0112] Fig. 6(A) shows an example in which empathy points are specified for an original sentence to be spoken by the robot 100, and the degree of empathy is set for each item for each empathy point. Note that the specification of the empathy points, items, setting values, etc. in Fig. 6(A) do not limit the configuration for imparting emotional expressions of the present invention.

[0113] As shown in Fig. 6(A), when the original sentence to be spoken by the robot 100 is "I heard about a super easy way to make delicious peanut butter. Are you interested?", empathy points a to j are identified as shown in the top row of Fig. 6(A). The division of the empathy points may be phrase units or word units in Japanese, or syllables (vowel units) in English, or a combination of these.

[0114] The second row from the top of FIG. 6(A) shows the pitch level of the sound, and a frequency value (Hz) is set for each of the empathy points a to j. Although empathy points a to j are set, for example, the pitch between each of the empathy points a to j may be set separately.

[0115] For reference, "pitch" is the height of a particular sound, while "interval" refers to the rise and fall of a sound in the interval between sounds.

[0116] The third row from the top in Figure 6(A) onwards indicates the speed level (tempo), strength level (accent), and intonation level, each of which can be set in 8-bit steps (0 to 255) as shown in Figure 6(B).

[0117] The speech sound source to which the emotional expression has been added is sent via the output unit 296 to the speaker, which is the control target 252 (see FIG. 2), and speech is executed.

[0118] 5 is an operational flowchart of the robot 100 performing a specific process of empathizing with an utterance for a sentence (i.e., sentence emotion generation control). The operational flow shown in FIG. 5 is automatically executed repeatedly, for example, at regular time intervals.

[0119] In step 350, it is determined whether or not a sentence has been input to input unit 292. If the determination in step 350 is negative, this routine ends. If the determination in step 350 is positive, the routine proceeds to step 352.

[0120] In step 352, the form of the input sentence is analyzed. More specifically, the sentence is classified into phrases and words, and the meanings and connections of each classified phrase and word, as well as the forms of use of particles, conjunctions, etc. used in the phrases, are comprehensively analyzed, and the process proceeds to step 354.

[0121] In step 354 , based on the analysis results in step 352 , a portion in which empathy should be sought is identified, and the process proceeds to step 356 .

[0122] In step 356, the sentence is converted into a speech source using the sentence-pronunciation database 304, and then the process proceeds to step 358, where an emotional expression is transferred to each part of the sentence using the emotional expression database 308 based on the analysis result of the sentence. As an example, as shown in Fig. 6(A), when the original sentence to be spoken by the robot 100 is "I heard how to make a super easy and delicious peanut butter. Are you interested?", the emotional transfer parts a to j are specified as shown in the top row of Fig. 6(A), and the high and low (pitch) level of the sound is set as a frequency value, and the speed level (tempo), strength level (accent), and intonation level are set as 8-bit (0 to 255) strength levels shown in Fig. 6(B). In this embodiment, by controlling the pitch, speed, accent, and intonation of the speech, the user's negative emotions can be returned to normal (level 0) or shifted to a more positive state.

[0123] In the next step 360, the output unit 296 outputs the sentence into which the emotional expression has been transferred to the speaker, which is the control target 252 (see FIG. 2), and this routine ends.

[0124] In this embodiment, in order to establish a position as a friendly partner between the user and the robot, the robot is designed to empathize with the user by shifting the user's emotions to the positive side as an emotional expression. The following are examples of empathy patterns. Note that the following examples are merely examples, and other empathy patterns are not intended to be negative.

[0125] (Emotional Empathy Pattern 1) A pattern in which appropriate emotions are automatically assigned to input text and the input text is used. (Emotional Empathy Pattern 2) A pattern that uses emotions assigned by voice emotion recognition from voice input and sentences obtained by voice utterance recognition. (Emotional Empathy Pattern 3) A pattern that uses emotions obtained by facial emotion recognition and sentences obtained by voice utterance recognition from images taken with a camera and voice input. (Emotional Empathy Pattern 4) A pattern where text and emotion or text and character image are manually input (for example, I want to make it speak with a happy voice saying ○○○, I want it to speak with ○○○ like my parents, I want it to speak with ○○○ like my lover, I want it to speak with ○○○ like my teacher)

[0126] (Modification) In this embodiment, the robot 100 establishes itself as a friendly partner with the user by uttering emotionally charged sentences. However, the robot 100 may observe the user's reaction after the utterance and reflect it in the next utterance of the robot 100.

[0127] 7 is a functional block diagram in which the processing of the specific processing unit according to the modified example is classified by function. Note that the same configurations (functions) as those in the functional block diagram of the present embodiment (see FIG. 3) are denoted by the same reference numerals, and the description of the configurations (functions) will be omitted.

[0128] As shown in FIG. 7, the processing unit 294 includes a captured image acquisition unit 310, which acquires image information captured by a camera function (e.g., the 2D camera 203 shown in FIG. 2) mounted on the robot 100, which mainly shows the facial expression of the user immediately after speaking.

[0129] The captured image acquisition unit 310 is connected to the image analysis unit 312 and sends the acquired image information to the image analysis unit 212 .

[0130] The image analysis unit 314 judges the user's facial expression from the image information. The judgment is made by analyzing the facial expression in response to the user's utterance (for example, whether it is positive or not, whether it is smiling or not, etc.) and specifying the degree of interest in the utterance.

[0131] The analysis result (degree of interest) from the image analysis unit is sent to the emotional expression adjustment unit 314. The emotional display adjustment unit 314 determines an adjustment value for the level of each item shown in FIG. 6(A) based on a predetermined interest degree-adjustment value conversion table, and sends the adjustment value to the emotional expression assignment unit 306.

[0132] Emotional expression adding section 306 adjusts the level of each item based on emotional expression database 308 using the adjustment value. Note that if the adjustment value is 0, no particular adjustment is required.

[0133] For example, in FIG. 6(A), if the user who heard the speech responds with "interested," the robot's next speech about how to make the product will be adjusted to speak with an increased intensity level and pitch (sounding a little excited) compared to normal.

[0134] This allows users to enjoy conversation as if they were talking to someone who gets along with them.

[0135] In addition, in this embodiment, the robot 100 has been described as an example of an agent for interacting with a user, but the external appearance of the robot 100 may be that of a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on a stuffed toy such as a bear or dog.

[0136] Furthermore, in this embodiment, the control for empathizing with the input sentence has been described as one of the functions of the robot 100 (so-called AI robot) shown in Figures 1 and 2, but the control for empathizing with the input sentence may be made to function as an independent control form (for example, a smartphone application) without being limited to the robot 100. In other words, a device applied as an agent for interacting with a user may be a smartphone or smart glasses.

[0137] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 shown in Fig. 8 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0138] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0139] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0140] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, JAVA, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0141] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0142] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0143] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0144] 5 system, 10 user, 20 communication network, 100 robot, 200 sensor unit, 210 sensor module unit, 220 storage unit, 228 control unit, 290 specific processing unit, 252 controlled object, 300 sentence morphology analysis unit, 302 speech conversion unit, 304 sentence-pronunciation database, 306 emotional expression assignment unit, 308 emotional expression database, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip

Claims

1. an input unit for accepting input of text; a processing unit that performs a specific process using a sentence generation model on the sentence received by the input unit; an output unit that controls a behavior of the robot so as to output a result of the specific processing; The processing unit includes: a sentence structure analysis unit that analyzes sentence structures including word division and phrase connections of the sentence input from the input unit to identify parts of the sentence that empathize with the user; a speech conversion unit that converts the sentence into speech information for speaking; an emotional expression adding unit that adds an emotional expression including at least one of pitch, speed, intensity, and intonation to each of the empathy points when making an utterance based on the utterance information converted by the utterance conversion unit; A behavior control system having the above configuration.

2. The processing unit, 2. The behavior control system according to claim 1, further comprising a sentence-pronunciation database in which sentences and the pronunciations of the sentences are stored in association with each other in said speech conversion section.

3. The processing unit, 2. The behavior control system according to claim 1, further comprising an emotional expression database in which predetermined patterns of emotional expression are stored as emotional expressions for shifting the emotion of the user, who is the target of the utterance, to a more positive side.

4. The emotional expression database comprises:

4. The behavior control system according to claim 3, further storing at least one of: a first emotional expression pattern which is a pattern in which an appropriate emotion is automatically assigned to an input sentence from a sentence input and the input sentence is used; a second emotional expression pattern which uses an emotion assigned by voice emotion recognition from a voice input and a sentence obtained by voice utterance recognition; a third emotional expression pattern which uses an emotion acquired by facial emotion recognition from an image taken by a camera and a voice input and a sentence acquired by voice utterance recognition; and a fourth emotional expression pattern obtained by manual input of a sentence and an emotion or a sentence and a person image.

5. The sentence input to the input unit is The behavior control system of claim 1, wherein the text generated is any one of text generated based on voice input when the user speaks, text generated by scanning text on a paper medium, etc., and text generated based on key input from an input device.

Images