User support system, user support device, user support program and user support method

Abstract

To provide a user support system capable of providing effective support to a user by using a communication robot that can hug the user.SOLUTION: A user support system (10) supports a user (100) through a robot (12) capable of hugging the user. The robot(12) has contact operation imparting units (28, 42) capable of performing a plurality of types of contact operations on the user at a plurality of contact positions in a state where the robot is hugging the user, as a result of a plurality of actuators controlled through an actuator control circuit (42) of the robot (12). Three types, questioning, listening and responding, are set as a dialogue act between the robot which is a receiver and the user who is a speaker, and type determination means (56, S32) determine the type of a contact operation on the basis of the type of the dialogue act. Further, position determination means (56, S33) determine the position of contact on the basis of the type of the dialogue act.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a user assistance system, a user assistance device, a user assistance program, and a user assistance method, and more particularly to a user assistance system, a user assistance device, a user assistance program, and a user assistance method in which, for example, a huggable communication robot executes a dialogue act while hugging a user. [Background technology]

[0002] The potential for social touch has been actively researched for application in social robots (communication robots) due to its physical and psychological benefits. Social touch from robots, including hugging, has been reported to have positive effects similar to those of human-to-human interactions. In particular, hugging is known to benefit mental health by reducing cortisol levels and releasing oxytocin.

[0003] The present inventors have proposed a huggable communication robot in Patent Document 1 etc. The huggable robot is designed to be able to perform contact actions (gestures) toward the user while hugging, such as lightly tapping or stroking the user.

[0004] Non-Patent Document 1 discloses that such contact motions are presented to the user's back or head at an appropriate speed and pressure to provide, for example, dialogue support to the user. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-134692 [Non-patent literature]

[0006] [Non-Patent Document 1] Y. Onishi, H. Sumioka, and M. Shiomi, “Moffuly-II: A Robot that Hugs and Rubs Heads,” Int. J. of Social Robotics, pp. 1-11, 2023. DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0007] However, when providing assistance as in Non-Patent Document 1, the type and location of such contact movements in the dialogue remain unclear.

[0008] Social touch, if inappropriate in type or location, or even at the wrong time or with the wrong intensity, can be perceived as inappropriate and lead to negative interactions, so this issue needs to be addressed.

[0009] Therefore, a primary object of the present invention is to provide a novel user assistance system, a novel user assistance device, a novel user assistance program, and a novel user assistance method.

[0010] Another object of the present invention is to provide a user assistance system, a user assistance device, a user assistance program, and a user assistance method that can provide effective assistance to a user using a huggable communication robot. [Means for solving the problem]

[0011] In order to solve the above problems, the present invention employs the following configuration: Note that the reference numerals and supplementary explanations in parentheses indicate the correspondence with the embodiments described later to aid in understanding the present invention, and do not limit the present invention in any way.

[0012] The first embodiment is a user assistance system that assists a user with a robot that can hug the user. The robot has a contact action providing unit that can perform multiple types of contact actions on the user while hugging the user, and the user assistance system is equipped with a type determination means that sets multiple types of dialogue acts as dialogue acts between the robot and the user, and further determines the type of contact action based on the type of dialogue act.

[0013] In a first embodiment, a user assistance system (10: a reference number illustrating a corresponding part in the embodiment but not intended to be limiting; the same applies below) includes a robot (12) capable of hugging a user, and a contact action providing unit (28, 36, 42) of the robot can execute multiple types of contact actions toward the user while hugging the user. Multiple types of dialogue acts, such as questioning, listening, and responding, are set as dialogue acts between the robot and the user. A type determining means (36, 76e, S32) determines the type of contact action, such as tapping or stroking, based on the type of dialogue act.

[0014] According to the first embodiment, the type of contact gesture is determined based on the type of dialogue act, so that user assistance can be provided effectively.

[0015] The second embodiment is directed to a user support system subordinate to the first embodiment, in which the contact action applying unit can apply contact actions to multiple contact positions of the hugging user, and the system is equipped with a position determining means for determining the contact positions based on the type of dialogue act determined by the type determining means.

[0016] In the second embodiment, the contact action applying unit of the robot (12) can apply contact actions to multiple contact positions of the hugging user, and the user support system (10) uses the position determining means (36, 76e, S33) to determine, for example, the user's head or back as the contact position based on the type of contact action determined by the type determining means (36, 76e, S32).

[0017] According to the second embodiment, the position of the contact motion is determined based on the type of contact motion, thereby enabling effective user assistance.

[0018] The third embodiment is a user assistance system subordinate to the first embodiment, and further includes an execution decision means for storing a contact action in a dialogue act prior to the current dialogue act as a pre-action, and for deciding whether to execute a contact action by the contact action providing means in the current dialogue act based on the information of the pre-action stored in the pre-action storage means.

[0019] In the third embodiment, the user assistance system (10) includes a pre-action storage means (78e), which stores, for example, a dialogue theme and the contents of pre-actions as history information. Here, the pre-actions include not only contact actions in the immediately preceding dialogue act but also contact actions in previous dialogue acts. The execution decision means (36, 76e, FIG. 9) decides whether to execute a contact action based on the information about the pre-actions.

[0020] According to the third embodiment, whether or not to execute a contact action is determined based on information about a previous action, thereby enabling effective user assistance.

[0021] The fourth embodiment is a user assistance system subordinate to the first embodiment, further comprising a pre-action storage means for storing a contact action during a dialogue act prior to the current dialogue act as a pre-action, and a type determination means for determining the type of the contact action based on the pre-action stored in the pre-action storage means and the type of the current dialogue act.

[0022] In the fourth embodiment, the user assistance system (10) includes a pre-action storage means (78e), which stores, for example, a dialogue theme and the contents of pre-actions as history information. Here, the pre-actions include not only contact actions in the immediately preceding dialogue act but also contact actions in previous dialogue acts. The type determination means (36, 76e, S32) determines the type of contact action based on the information on the pre-actions and the type of the current dialogue act.

[0023] According to the fourth embodiment, the type of contact action is determined based on the information of the previous action and the current dialogue act, so that effective user assistance can be achieved.

[0024] The fifth embodiment is a user assistance system dependent on the second embodiment, further comprising a pre-action storage means for storing a contact action during a dialogue act prior to the current dialogue act as a pre-action, and a type determination means for determining the position of the contact action based on the pre-action stored in the pre-action storage means and the type of the current dialogue act.

[0025] In the fifth embodiment, the user assistance system (10) includes a pre-action storage means (78e) that stores, for example, a dialogue theme and the contents of pre-actions as history information. The position determination means (36, 76e, S33) determines the type of contact action based on the information on the pre-actions and the type of the current dialogue act.

[0026] According to the fifth embodiment, the position of the contact action is determined based on the information of the previous action and the current dialogue act, so that the user can be effectively assisted.

[0027] The sixth embodiment is a user assistance system that assists a user with a robot that can hug the user. The robot has a contact action providing unit that can perform multiple types of contact actions on the user at multiple contact positions while hugging the user, and further has multiple types of dialogue acts set as dialogue acts between the robot and the user. The user assistance system is equipped with a type determination means that determines the type of contact action based on the type of dialogue act, and a position determination means that determines the contact position based on the type of dialogue act determined by the type determination means.

[0028] The seventh embodiment is a user assistance device that assists a user with a robot that can hug the user. The robot has a contact action providing unit that can perform multiple types of contact actions on the user at multiple contact positions while hugging the user, and further has multiple types of dialogue acts set as dialogue acts between the robot and the user. The user assistance device is equipped with a type determination means that determines the type of contact action based on the type of dialogue act, and a position determination means that determines the contact position based on the type of dialogue act determined by the type determination means.

[0029] An eighth embodiment is a user assistance program for a user assistance device that assists a user with a robot that can hug the user. The robot has a contact action providing unit that can perform multiple types of contact actions on the user at multiple contact positions while hugging the user, and further has multiple types of dialogue acts set as dialogue acts between the robot and the user. The user assistance program causes a computer of the user assistance device to function as a type determination means that determines the type of contact action based on the type of dialogue act, and as a position determination means that determines the contact position based on the type of dialogue act determined by the type determination means.

[0030] The ninth embodiment is a user assistance method that assists a user with a robot that can hug the user. The robot has a contact action providing unit that can perform multiple types of contact actions on the user at multiple contact positions while hugging the user, and further, multiple types of dialogue acts are set as dialogue acts between the robot and the user. The type of contact action is determined based on the type of dialogue act, and the contact position is determined based on the determined type of dialogue act. [Effects of the Invention]

[0031] According to the present invention, it is possible to effectively provide, for example, conversation support to a user by making an appropriate contact action toward the user while hugging the user.

[0032] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the preferred embodiments with reference to the drawings. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram showing a user support system according to one embodiment of the present invention. [Figure 2] FIG. 2 is an illustrative view showing an embodiment of a communication robot used in the embodiment of FIG. [Figure 3] FIG. 3 is a diagram showing an example of the structure of the communication robot of the embodiment shown in FIG. [Figure 4] FIG. 4 is a block diagram showing the electrical configuration of the communication robot shown in FIGS. [Figure 5] FIG. 5 is a block diagram showing the electrical configuration of the server in the embodiment shown in FIG. [Figure 6] FIG. 6 is an illustrative view showing a state in which a user is being hugged by a robot. [Figure 7] FIG. 7 is an illustrative view showing an example of a memory map of the memory in the embodiment of FIG. [Figure 8]FIG. 8 is a flowchart showing an example of the overall operation of the server in the user support system shown in FIG. [Figure 9] FIG. 9 is a flowchart showing an example of the operation of the action execution decision in the embodiment of FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of determining the operating parameters in the embodiment of FIG. [Figure 11] FIG. 11 is a diagram showing an example of the distribution that determines the timing of starting an action in the embodiment of FIG. 10, where (A) shows the case where the dialogue act is a question, (B) shows the case where the dialogue act is listening, and (C) shows the case where the dialogue act is a response. [Figure 12] FIG. 12 is a diagram showing an example of a distribution that determines the operation duration in the embodiment of FIG. [Figure 13] FIG. 13 is a diagram showing an example of a distribution for determining the post-action pause time in the embodiment of FIG. 10, where (A) shows a case where the conversation theme is negative, and (B) shows a case where the conversation theme is positive. [Figure 14] FIG. 14 is a flow diagram showing an example of the execution of the contact action in the embodiment of FIG. [Figure 15] FIG. 15 is a flow diagram showing an example of the operation of process A in the embodiment of FIG. [Figure 16] FIG. 16 is a flow diagram showing an example of the operation of process B in the embodiment of FIG. BEST MODE FOR CARRYING OUT THE INVENTION

[0034] 1, a user assistance system 10 (hereinafter simply referred to as the "assistance system") of this embodiment includes a social robot or communication robot (hereinafter simply referred to as the "robot") 12. This robot 12 is a robot that can hug a user or give a social touch (contact action) to the user, and provides speech assistance to the user by the contact action during the hug.

[0035] In this embodiment, the robot 12 is a large, stuffed bear-shaped robot resembling a teddy bear (trade name), as shown in Figure 2. However, it should be noted in advance that any huggable robot having a different appearance and structure can be used as the robot 12 used in the embodiment of Figure 1.

[0036] The robot 12 is connected to a server 16, which serves as a user support device, via a network 14 such as the Internet or a telephone communication line. The server 16 is a general-purpose computer such as a PC, PDA, smartphone, or tablet terminal. The server 16 may be installed near the robot 12.

[0037] 1, the server 16 is further connected to a microphone 18 that collects environmental sounds and the user's voice, and a camera 20 that captures environmental images and images of the user. Voice data collected by the microphone 18 is input to the server 16, and the server 16 can use the voice data to perform voice recognition. Video data from the camera 20 is input to the server 16, and the server 16 can process the video data to, for example, identify the user.

[0038] In this embodiment, the robot 12 is designed to be able to converse with the user. To this end, the server 16 is set up in advance with topics necessary for such a conversation with the user and scenarios corresponding to those topics, as will be described later.

[0039] The robot 12 is a "bear-shaped" stuffed toy robot as shown in Figures 2 and 3, and the bottom 26 of its torso 24 (rump: the rear of the torso in a quadruped (particularly a mammal)) is placed on the floor 22. Arms 28 are provided on both sides of the upper end of the torso 24 so that they can extend forward. The length of each arm 28 is selected so that when the robot 12 hugs a user (person), the arms 28 can reach around the user's back. As an example, the length is approximately 80 cm.

[0040] A head 30 is formed on top of the body 24 via a neck (not shown). The overall length of the robot 12, i.e., the height from the base 26 to the top of the head 30, is, for example, approximately 2 m. This size was determined with the intention that the robot 12 can be used by not only children but also adults, and is a size that gives the user a sense of security as if they were being hugged.

[0041] Although feet 32 ​​are provided on both the left and right sides of the torso bottom 26, these feet 32 ​​are not used for the robot 12 to stand up, but are simply provided in a state where they are thrown forward. Therefore, when the robot 12 hugs the user 60 as shown in Fig. 6 described below, the feet 32 ​​simply function as cushions.

[0042] 3, the robot 12 includes a torso frame 24a for maintaining the torso 24 in an upright position, and a bottom frame 26a and a shoulder frame 26b attached horizontally to the top and bottom ends of the torso frame 24a. Arm frames 28a extend forward from both ends of the shoulder frame 26b. A neck frame 30b is provided above the horizontal center of the shoulder frame 26b, aligned with the torso frame 24a, and a head frame 30a is provided above the neck frame 30b to support the head 30.

[0043] These frames may be made of metal such as aluminum, or resin such as fiber-reinforced plastic or engineering plastic, and are basically designed with a cross-sectional shape and thickness that ensures safe strength. For example, a pipe-shaped frame may be used.

[0044] These frames are wrapped in an elastic material such as polypropylene cotton, and then covered with an outer cover 33. The outer cover 33 is made of fabric, and is made of fur or felt to give it a "bear" appearance.

[0045] However, the part of the head 30 that the robot 12 touches with the user's face when hugging the user is covered with a removable and washable fabric, which helps maintain hygiene.

[0046] Elbow joints 34 are provided on arm frame 28a at positions corresponding to the left and right elbows. That is, a frame corresponding to the forearm of arm 28 and a frame corresponding to the upper arm are connected via elbow joints 34. As an example, elbow joint 34 allows the forearm to bend up to 90° relative to the upper arm. Therefore, by controlling the angle of elbow joint 34, arm 28 can be bent. In other words, arm 28 can tightly hug the user (e.g., hug back). Because arm 28 is long, about 80 cm as described above, when arm 28 is bent, the forearm or at least the portion beyond the wrist (hand) can be placed behind the user's back, depending on the size of the user.

[0047] Furthermore, in this embodiment, an actuator is incorporated in the elbow joint 34 so that the robot 12 can hug the user with its arm 28. However, a relatively weak digital servo (torque: 11 kg / cm) is used as the actuator so that the user can push back relatively easily. This is for safety reasons.

[0048] Although not shown in Figures 2 and 3, a speaker and a microphone are embedded in the face of the head 30 of the robot 12, allowing it to speak to the user and listen to what the user is saying. XIMERA (product name: ATR Speech and Language Research Laboratories) was used as speech synthesis software for speaking.

[0049] 4, the robot 12 includes a CPU 36 that controls the overall operation of the robot 12. The CPU 36 is connected to a communication unit 38 via a bus 37, and therefore the CPU 36 is communicably connected to the network 14, i.e., the server 16, via the communication unit 38, either wired or wirelessly.

[0050] In this embodiment, all utterances and movements of the robot 12 are basically executed in response to commands given from the server 16, so the CPU 36 receives text data relating to the commands and utterances from the server 16 and stores the commands and data in memory 40 via the bus 37. The CPU 36 then gives command values ​​to an actuator control circuit 42 via the bus 37 in accordance with the commands and data stored in memory 40, thereby controlling the operation of each of the actuators A1-An. The actuator control circuit 42 generates pulse power in the number corresponding to the command value given from the CPU 36 and gives it to the corresponding stepping motor, thereby driving each of the actuators A1-An.

[0051] However, in addition to the actuator using such a stepping motor, any actuator such as an actuator using a servo motor or a fluid actuator can be used.

[0052] Here, one of the actuators A1-An is the actuator (digital servo) of the elbow joint 34 shown in Fig. 3, and the others are actuators of other joints described later. By controlling the actuators A1-An in accordance with commands from the server 16, the actuator control circuit 42 can make the robot 12 hug the user and make contact with the user while hugging.

[0053] The sensor I / F (interface) 46 is connected to the CPU 36 via the bus 37 and receives outputs from the tactile sensor 46 and the camera 48 .

[0054] The tactile sensor or skin sensor 46 is, for example, a touch sensor, and constitutes part of the tactile sense of the robot 12. That is, the tactile sensor 46 detects whether a human or other object has touched the robot 12. For example, the tactile sensor 46 is provided at a predetermined location on the front of the body 24 of the robot 12, and can detect whether a user has hugged the robot 12.

[0055] The tactile sensor 46 also functions as a pressure sensor. For example, such a tactile sensor 46 is formed of a polymer piezoelectric sensor sheet, and a plurality of sensor sheets are appropriately distributed on the inner surface of the arm 28. Such a tactile sensor 46 can detect the pressure applied to the arm 28 of the robot when it is hugging the user.

[0056] Furthermore, the pressure data detected by the tactile sensor 46 is output from the CPU 36 and provided to the server 16 via the communication unit 38 and the network 14 .

[0057] The camera 48 is an image sensor and constitutes part of the robot 12's visual system. In other words, the camera 48 is used to detect video or images seen by the robot 12's eyes. In this embodiment, data (image data) corresponding to the video (video or still image) captured by the camera 48 is provided to the CPU 36 via the sensor I / F 44. The CPU 36 not only detects changes in the captured video, but also transmits the image data to the server 16 via the communication unit 38 and the network 14 (FIG. 1). The server 16 then outputs the received image data to the display 66 (FIG. 5). Therefore, the video captured by the camera 48 is displayed on the display 66.

[0058] The speaker 52 and microphone 54 are also connected to the input / output I / F 50. The speaker 52 outputs sound when the robot 12 speaks. When the operator of the server 16 speaks directly, the sound is output via the network 14, the communication unit 38, and the input / output I / F 50.

[0059] The microphone 54 is a sound sensor and constitutes part of the hearing of the robot 12. The microphone 54 has directionality and is mainly used to detect the voice of a user (person) who is interacting (communicating) with the robot 12.

[0060] The memory 40 of the robot 12 is, for example, a RAM or an HDD (hard disk drive), and is pre-programmed with a speech synthesis program. Therefore, the robot 12 can be made to speak by speech synthesis. When an instruction (a speech command) to speak by speech synthesis is input from the server 16, the CPU 36 outputs to the speaker 52 a speech synthesized in accordance with character data relating to the dialogue theme, dialogue act, and speech content provided by the server 16.

[0061] In this embodiment, basically, all utterances and actions of the robot 12 are executed in response to commands given from the server 16. For this purpose, the memory 40 stores the text data relating to the dialogue themes, dialogue acts, and utterance contents received from the server 16, as well as the parameters of the contact actions executed so far.

[0062] 5, the server 16 functioning as a user assistance device includes a CPU 56, to which a memory 58 is connected via a bus 57. The memory 58 includes a RAM, a ROM, a HDD, etc. The CPU 56 controls the robot 12 by executing a computer program described below. The program is stored in advance in the ROM or the HDD, and is loaded into the RAM and executed as needed.

[0063] An input interface 60 is also connected to the bus 57. The microphone 18 shown in Fig. 1 is connected to the input interface 60, and for example, an operator's voice signal from the microphone 18 is input from the input interface 60 to the CPU 56 (memory 58).

[0064] The input interface 60 is further connected to the camera 20, and the video signal from the camera 20 is input from the input interface 60 to the CPU 56 (memory 58).

[0065] A communication unit 62 is further connected to the bus 57, and this communication unit 62 communicates with the robot 12 and the like via the network 14. For example, it is used to receive data transmitted from the robot 12 (video signals from the camera 48, audio signals from the microphone 54, and electrical signal data from the tactile sensor 46) and to transmit various commands to the robot 12.

[0066] An output interface 64 is also connected to the bus 57. The output interface 64 outputs a video signal from, for example, the camera 20 to a display 66 to display an image required by the operator, and also outputs an audio signal to a speaker 68.

[0067] The robot interface 70 transmits commands (instructions including setting data) from the server 16 to the CPU 36 (FIG. 3) of the robot 12, causing the robot 12 to execute contact actions (such as tapping or stroking) during a hug.

[0068] However, in this embodiment, when the robot 12 is equipped with a communication unit 38 as shown in FIG. 4, communication between the robot 12 and the server 16 can also be carried out through the communication unit 62 and this communication unit 38, so commands may be sent directly from the CPU 56 to the robot 12 without going through the robot interface 70.

[0069] In the user assistance system 10 of this embodiment, as shown in FIG. 6, the robot 12 hugs the user 100 and either converses with the user 100 or listens to what the user 100 is saying.

[0070] In the state shown in FIG. 6, the user 100 gets between the arms 28 of the robot 12, and in response to the user 100 hugging the robot 12 with its arms 104, the robot 12 closes its arms 28 and wraps its tips around the torso 102 (back) or head 106 of the user 100 to hug the robot 12.

[0071] While previous research has demonstrated that social touch by robots can be useful as a dialogue receiver, the use of social touch in real dialogue flows has not yet been fully developed.

[0072] Therefore, the inventors have pursued the effective use of social touches such as light patting and rubbing by the robot 12 in dialogue with the speaker (user 100) in the assistance system 10 of the embodiment shown in Figure 1, and have constructed an assistance system that supports users in interacting with huggable robots.

[0073] The inventors created a dialogue scenario in order for the robot 12, which is the receiver, to support the user 100, which is the speaker. This scenario was used for the contents of utterances of participants in data collection in an experiment conducted by the inventors, and for implementing the robot.

[0074] Hugging is done as a form of active listening and to support the speaker. Hugging not only encourages speakers with negative emotions and thoughts and helps them transform their worries into positive ones, but also has a wide range of meanings for support, such as encouraging speakers with positive emotions and thoughts and helping them achieve their goals. Therefore, we designed a dialogue that sorts out negative and positive thoughts and provides comprehensive support for the speaker's emotions and thoughts. In both cases, the speaker is encouraged to look deeply within themselves and organize their thoughts.

[0075] In the dialogue between the user 100 and the robot 12 in this assistance system 10, the robot 12 as the receiver asks a question, listens to the speaker's answer, and then replies. The dialogue progresses by repeating this cycle. In the embodiment, four dialogue acts are defined: questioning, listening, responding, and greeting.

[0076] The questions for the recipient were designed based on the column method, which encourages objective observation by organizing the flow of a person's thoughts and cognition. The dialogue scenario was designed based on the column method and included eight items: "situation," "reason," "emotion," "action," "thought," "alternative thought," "change in thought and action," and "awareness through dialogue." The dialogue flow is outlined below.

[0077] First, the listener asks the speaker to organize their thoughts and then asks them about eight items.

[0078] Ask the following types of situational questions, such as, "Tell me about some of your recent worries and goals." Follow up with questions to explore the specific reasons for your concerns, such as, "Can you tell me why or how you felt?"

[0079] Regarding emotions, behaviors, and thoughts, the following questions are asked. For example, "How did you feel / behave / think in that situation?" and "If you had imagined yourself in that situation, how would you have felt / behaved / thought?" There is another way to think about this question. For example, "What would you tell someone close to you if they had the same worries or goals?" This question asks for changes. For example, "You currently have some behaviors or thoughts. How would you try to change yourself?" The next question is directed at self-awareness in the dialogue. For example, "Tell me how the dialogue has deepened your self-awareness."

[0080] While listening to the speaker's response to each question, the receiver does not interrupt and remains fully present until the speaker has finished answering. After listening to the speaker's response to each question, the receiver responds with acknowledgment, repetition, empathy, praise, and gratitude. Possible responses include: "Oh, I see," "So that's what it was," "That's definitely true," "That's great," "Thanks for letting me know," etc. Finally, the receiver signals the end of the dialogue and thanks the speaker for their contribution.

[0081] Fig. 6 is a diagram showing an example of a memory map of memory 58 of server 16 shown in Fig. 5. As shown in Fig. 6, memory 58 includes RAM as described above, which includes a program storage area 72 and a data storage area 74. A control program for server 16 is stored in program storage area 72. The control program and necessary data may be stored in advance in, for example, a flash memory or a hard disk included in memory 58, and may be read out as needed and loaded into RAM.

[0082] The control programs of the server 16 include a display control program 76a, an operation detection program 76b, a voice recognition program 76c, a dialogue management program 76d, a contact action determination program 76e, and the like.

[0083] The display control program 76a is a program that displays an image on the display 66 (FIG. 5).

[0084] The operation detection program 76b is a program that detects an operation on an input device (not shown) of the server 16.

[0085] The voice recognition program 76c is a program that converts voice data acquired by voice recognition into character data, and refers to a keyword list storage area 78c (described later) in the data storage area 74 to detect keywords.

[0086] However, it is also possible to use commercially available software such as Google (trademark) speech recognition as the speech recognition program 76c.

[0087] The dialogue management program 76d refers to the dialogue scenario storage area 78d (described later) in the data storage area 74, selects an appropriate response in accordance with the keywords detected by the voice recognition program 76c, and sends text data relating to the dialogue theme, dialogue act, and utterance content to the robot 12 via the robot interface 70 (FIG. 5), and also sends the user's speech start command to the contact action determination program 76e.

[0088] The contact action determination program 76e determines a contact action according to a flow chart shown in Fig. 8, which will be described later, and sends a contact action command and a speech start command to the robot 12 via the robot interface 70. This contact action determination program 76e constitutes a user assistance program.

[0089] The contact action determination program 76e includes a contact action execution determination program shown in Fig. 9, a contact action parameter determination program shown in Fig. 10, and a contact action execution program shown in Fig. 14. In detail, the contact action execution program includes Process A and Process B shown in Fig. 15 and Fig. 16.

[0090] Although not shown, the program storage area 72 also stores other programs necessary for controlling the server 16.

[0091] The data storage area 74 includes an image generation data storage area 78a, an operation data storage area 78b, a keyword list storage area 78c, a dialogue scenario storage area 78d, a temporary storage area 78e, and the like.

[0092] Image generation data storage area 78a stores image generation data including polygon data, texture data, and other data that are set in advance to generate data for various screens to be displayed on display 66.

[0093] The operation data storage area 78b stores data input from an input device (not shown) in chronological order.

[0094] The keyword list storage area 78c stores in advance character data indicating a plurality of keywords that the inventors have envisioned based on experiments.

[0095] The dialogue scenario storage area 78d stores scenarios that follow the dialogue flow as described above.

[0096] The data storage area 74 stores other data required for the execution of the control program of the server 16, and also includes flags and other counters (timers) required for the execution of the control program.

[0097] In addition, a temporary storage area 78e may be set in the data storage area 74 as shown in FIG. 7, and character data relating to the dialogue theme, dialogue act, and utterance content transmitted to the robot 12 as described above, as well as parameters of contact actions performed so far, may be temporarily stored.

[0098] Furthermore, the temporary storage area 78e stores, as history information, the dialogue theme and the contents of the preliminary actions shown in Table 1. Note that the preliminary actions include not only the contact actions in the immediately preceding dialogue act, but also the contact actions in the dialogue acts before that.

[0099] [Table 1]

[0100] The operation of the assistance system 10 of the embodiment shown in FIG. 1 will now be described with reference to the flow charts shown in FIGS. 8-10 and 14-16.

[0101] In the first step S1 of Fig. 8, the server 16 first determines whether to execute a contact action according to the subroutine shown in Fig. 9. However, it is assumed that the user 100 has already been hugged by the robot 12 as shown in Fig. 6.

[0102] 9, in a first step S11, the CPU 56 inputs the dialogue acts and robot utterances set in the dialogue scenario storage area 78d in accordance with the dialogue management program 76d. That is, it determines whether a preset dialogue scenario has been read. As explained above, such a dialogue scenario is stored in the dialogue scenario storage area 78d of the memory 58 by the dialogue management program 76d as character data relating to the dialogue theme, dialogue acts, and utterances.

[0103] The four dialogue acts are set as shown in Table 2. The role and content of each dialogue act are as shown in Table 2.

[0104] [Table 2]

[0105] For example, the role of the dialogue act "question" is to ask a question to the user, and the content of the utterance is a question for organizing thoughts. The role of the dialogue act "listen" is to listen to what the user is saying, and therefore there is no content of the utterance.

[0106] When the dialogue act is "response," its role is to respond based on the user's utterance, and the content of the utterance is expected to include acceptance, empathy, support, gratitude, self-disclosure, information transmission, and parroting.

[0107] When the dialogue act is "greeting," the role is to greet the user, and a greeting is set as the utterance content.

[0108] However, since "greeting" as a dialogue act does not have any special meaning, the following explanation will focus on the three dialogue acts excluding "greeting."

[0109] Then, in the next step S12, the CPU 56 refers to information such as the dialogue theme and pre-actions stored in the temporary storage area 78e (Figure 7) of the memory 58 as shown in Table 1, and executes the conditional judgments in the following steps S13, S14, and S15.

[0110] In step S13, the CPU 56 determines whether condition A is satisfied. Condition A is a condition that the number of seconds from the end of the preliminary action to the present is within the maximum value tmax of the post-action stop time. The maximum value tmax is set to be within 81.13 seconds, for example. In other words, in step S13, it is determined that the predetermined time tmax or more has not elapsed since the end of the preliminary action.

[0111] After determining "YES" in step S13, the CPU 56 determines in step S14 whether condition B is met. Condition B is a condition that the prior action was not performed during the same dialogue act as the current one, but was performed in a dialogue that occurred one or more dialogue acts prior to the current one. In other words, in step S14, it is determined whether the prior action was performed during a dialogue act that occurred one or more dialogue acts prior to the current one.

[0112] If the answer in step S14 is "YES," the CPU 56 then determines in step S15 whether condition C is met. Condition C is a condition that a random number between "0" and "1" is generated, and the random number is equal to or less than the value of the frequency probability corresponding to a preset dialogue act or pre-action state. For example, it is equal to or less than the value for the combination of dialogue act, pre-action time sequence, and dialogue theme in Table 3, which shows the probability distribution.

[0113] [Table 3]

[0114] For example, if we consider the dialogue act immediately before the current one, and the dialogue theme is negative, if the dialogue act is a question, and the probability is 0.508 or less, the answer will be "YES" in step S15. If the dialogue act is listening, and the probability is 0.848 or less, the answer will be "YES" in step S15. If the dialogue act is a reaction, and the probability is 0.624 or less, the answer will be "YES" in step S15.

[0115] For example, if we consider the dialogue act immediately before the current one, and the dialogue theme is positive, if the dialogue act is a question, and the probability is 0.422 or less, the answer is "YES" in step S15. If the dialogue act is listening, and the probability is 0.657 or less, the answer is "YES" in step S15. If the dialogue act is a response, and the probability is 0.608 or less, the answer is "YES" in step S15.

[0116] In Table 3, for dialogue acts two or more prior to the current dialogue act, the probability is lower in all cases than in the corresponding dialogue act one prior to the current dialogue act.

[0117] If the determination in step S15 is "YES", the process returns to the main routine (step S3 in FIG. 8) in the same way as if the determination in step S13 and step S14 were "NO".

[0118] However, if the determination in step S15 is "NO," the CPU 56 transmits a robot utterance command to the robot 12 via the communication unit 62 or through the robot interface 70 in step S16. Thereafter, the CPU 56 waits until the next dialogue act is selected in step S17.

[0119] 9 is completed, the process returns to step S3 in FIG. 8, and the contact motion parameter determination subroutine shown in FIG. 10 is executed.

[0120] 10, the first step S31 refers to the dialogue theme, dialogue act, and pre-action information stored in memory 58. Then, steps S32-S37 are executed, with steps S32 and S33 and steps S34-S37 executed in parallel. Although it is not necessary to execute these steps in parallel, it is more convenient to execute steps S32 and S33 first.

[0121] In step S32, first, the type of contact action is determined. In this step S32, an appropriate contact action type is determined from a plurality of candidate contact action types according to the dialogue theme, dialogue act, and state of the pre-action.

[0122] In this embodiment, as shown in FIG. 6, two types of contact motions, a tapping motion and a stroking motion, are set as the motions that the robot 12 will perform when hugging the user 100, and therefore, in this step S32, it is determined whether the tapping motion or the stroking motion should be performed.

[0123] For example, when deciding between two types of actions, hitting and stroking, the decision can be made using the probability distribution shown in Table 4, which is set for each dialogue act and pre-action type. A random number between "0" and "1" is generated, and if the value is equal to or less than the value corresponding to the combination of dialogue act and pre-action type in Table 4, the action type is set to hitting, and if the value is greater than the value shown in Table 4, the action type is set to stroking.

[0124] [Table 4]

[0125] For example, if the type of pre-action is in the initial state, it means that a contact action has not been performed in the past, and if the dialogue act is a question or listening, a tapping action is determined if the value is below "0.625", and if the dialogue act is a response, a tapping action is determined if the value is below "0.364". However, since the probability of a value below "0.625" is greater than the probability of a value below "0.364", if the type of pre-action is in the initial state and the dialogue act is a question or listening, a tapping action is ultimately determined.

[0126] Similarly, regardless of whether the type of pre-action is a tapping action or a stroking action, if the dialogue act is a question or attentive listening, the tapping action is determined to be given priority.

[0127] However, if the preliminary action is a tapping action and the dialogue act is a response, it may be determined as a stroking action.

[0128] In this way, the server 16 determines the type of contact gesture in step S32. In this way, in this embodiment, the type of contact gesture is determined based on the type of dialogue act, so that effective user assistance can be provided.

[0129] Next, in step S33, a position (motion position) for performing a contact motion is determined. In this step S33, one appropriate motion position is determined from a plurality of motion position candidates according to the dialogue theme, dialogue act, and state of the pre-motion.

[0130] For example, when determining the action position from two options, the user's back and head, as in the embodiment, the determination can be made using the probability distribution shown in Table 5, which is set for each dialogue act, pre-action position, and type of contact action to be performed that has already been determined in step S32. A random number between "0" and "1" is generated, and if the value is equal to or less than the value corresponding to the combination of dialogue act, pre-action position, and action type in Table 5, the action position is determined to be the user's back, and if the value is greater than the value shown in Table 5, the action position is determined to be the user's head.

[0131] [Table 5]

[0132] For example, when the location of the preliminary action is the back and the action type determined in step S32 is a hitting action, if the dialogue act is a question or a response, the back is determined as the action location because the value will never be greater than "1".

[0133] For example, if the position of the preliminary action is the head and the action type determined in step S32 is a hitting action, then if the dialogue act is a question, the head is determined as the action position because it will never be below "0." If the dialogue act is a reaction, the back is determined as the action position because it will never be above "1."

[0134] In this way, the server 16 determines the position of the contact action in step S33. In this way, in this embodiment, the contact position is determined based on the type of contact action, so that effective user assistance can be provided.

[0135] On the other hand, in step S34, the server 16 determines the operation start timing.

[0136] In step S34, the time to start the contact action is determined appropriately according to the dialogue act. Based on the dialogue act, the optimal action timing is determined around a trigger such as the time when the robot or user speaks.

[0137] For example, if there are three types of dialogue acts - question, listening, and response - it is possible to determine the timing of starting an action for each of them. The starting point is the time when the robot or user starts speaking, and the start of the speech is set as "0.0" and the end of the speech as "1.0", and the relative timing of starting an action within the speech section is determined.

[0138] As an example, the relative action decision timings are determined according to the distributions in FIGS. 11A-11C.

[0139] When the dialogue act shown in Figure 11(A) is a question The minimum value is "-0.29" and the maximum value is "1.71". The gamma distribution k = 4888.78, Θ = 0.76 × 10 -2 , I=-36.54, a random number is generated and the value is used as the operation start timing.

[0140] When the dialogue act shown in Figure 11(B) is active listening Gamma distribution k = 0.66, Θ = 0.16, I = 0.50, I = 0.76 × 10 with a minimum value of "0.00" and a maximum value of "1.00". -16 A random number is generated according to the above and the value is used as the operation start timing.

[0141] When the dialogue act shown in Figure 11(C) is a response The minimum value is "-0.44" and the maximum value is "1.12". The gamma distribution k=608.44, Θ=0.16×10 -1 , I=-9.22, a random number is generated and the value is used as the operation start timing.

[0142] In the next step S35, the operation duration is determined, and in this step S36, the operation start time is determined based on the value of the relative operation start timing determined in step S34.

[0143] When the dialogue act is a question or a response, the speech section is calculated as the number of seconds from the content of the speech of the robot 12, and the movement start time is determined by multiplying this by the relative movement start timing. The robot 12 starts speaking from the starting point, and starts the contact movement after the movement start time has elapsed after the start of speech.

[0144] However, if the action start time is a negative value, the action starts before the robot 12 speaks, so the absolute value of the action start time is used as the speech start time, and the robot starts speaking after the speech start time has elapsed from the start of the action.

[0145] When the dialogue act is listening, the action start time is the product of the action start timing value and the user's average speaking time of 2.70 seconds. The contact action begins after the action start time has elapsed after the user starts speaking.

[0146] In the next step S36, the server 16 determines the duration of the action, in seconds, within a range from a minimum value tmin greater than 0 (zero) to a maximum value tmax, depending on the dialogue theme, dialogue act, and the state of the pre-action.

[0147] For example, according to the distribution shown in Figure 12 obtained through experiments by the inventors, it is possible to determine the value of the action duration by random number generation according to a gamma distribution k = 0.91, Θ = 3.28, I = 0.90, within the range of a minimum value tmin of 0.90 seconds and a maximum value tmax of 19.30 seconds, regardless of the dialogue theme, dialogue act, or state of the pre-action.

[0148] 10, the server 16 determines the pause time after the contact action by the robot 12. In this step S37, the server 16 determines the pause time after the action in seconds within the range of a minimum value tmin greater than 0 (zero) to a maximum value tmax, depending on the dialogue theme, dialogue act, and state of the previous action.

[0149] For example, as shown in Figure 13(A) or Figure 13(B), depending on the dialogue theme, the value of the operation duration can be determined by random number generation according to a gamma distribution k = 0.52, Θ = 19.61, I = 0.60 if the dialogue theme is negative, or according to a gamma distribution k = 0.69, Θ = 16.05, I = 0.60 if the dialogue theme is positive, within a range of a minimum value tmin of 0.60 seconds and a maximum value tmax of 81.13 seconds.

[0150] Thereafter, in step S38, the CPU 56 stores the contact operation parameters determined in steps S32-S36 as described above in the temporary storage area 78e of the data storage area 74 of the memory 58, and transmits them to the robot 12 via the communication unit 62 or through the robot interface 70.

[0151] According to the parameters of the contact action determined in this way, the robot 12 is made to execute the contact action during hugging in step S5 of Fig. 8. Specifically, the robot 12 operates according to the flow diagram shown in Fig. 14.

[0152] In the first step S51, the CPU 56 refers to the information on the dialogue acts and contact gestures stored in the temporary storage area 78e of the data storage area 74 (FIG. 7).

[0153] Then, in step S52, the CPU 56 determines whether or not the user 100 (FIG. 6) has spoken, that is, whether or not the user has spoken first. This can be determined by the presence or absence of a voice signal from the microphone 54 of the robot 12.

[0154] When the determination in step S52 is "YES," the dialogue act is attentive listening, and the CPU 56 outputs a contact action command in response to the user's utterance. As an example, in process A shown in Fig. 15, the CPU 56 outputs a contact action command in accordance with the action start time in response to the user's utterance.

[0155] When it is detected in step S61 that a trigger has been input, the process waits until the previously determined operation start time has elapsed in step S62, and after that time has elapsed, the process outputs a contact operation command in step S63. That is, the contact operation command is output in step S53 (FIG. 14).

[0156] If the determination in step S52 is "NO," the dialogue act is a question or a response, and the CPU 56 determines in step S54 whether the contact action corresponds to the first contact action in the current dialogue act section. In other words, it determines whether the contact action is the first contact action since the robot 12 started speaking. Here, it can be determined whether a contact action command has already been output.

[0157] If the determination in step S54 is "YES," the CPU 56 outputs a contact movement command in response to the user's utterance in step S55. As an example, in process B shown in Fig. 16, the CPU 56 outputs a contact movement command in response to the utterance of the robot 12 and in accordance with the movement start time.

[0158] In step S71, it is determined whether the previously determined operation start time is a value equal to or greater than 0 (zero). In this step S71, it is determined whether the operation start time has arrived. In other words, it is determined whether the operation start time is positive or negative.

[0159] If "YES" is determined in step S71, the action start time is positive, which means that the action start time has not yet arrived. In this case, the robot 12 will first speak and then execute a contact action. Therefore, in step S72, a robot speech command is output. Then, in step S73, the robot waits until the action start time has elapsed, and then in step S74, a contact action command is output.

[0160] If the determination in step S71 is "NO," the action start time is negative, so the robot 12 must perform a contact action before speaking. First, the robot outputs a contact action command in step S75, and then waits until the speech start time has elapsed in step S76, and then outputs a robot speech command in step S77.

[0161] The determination of "NO" in step S54 means that this is the second or more contact action during the dialogue act section in the robot 12's utterance. In this case, since the robot 12 has already started speaking, a contact action command is immediately output in step S56.

[0162] In this way, after the contact action command is output in step S53, S55 or S56, the process waits until the action duration time has elapsed in step S57. In other words, the contact action of the robot 12 is permitted for the duration time.

[0163] After the contact operation duration has elapsed, an operation stop command is output in step S58, and then in step S79, the process waits until the post-operation stop time has elapsed, and returns to step S7 (FIG. 8) of the main routine.

[0164] In step S7, it is determined whether or not to end the dialogue, and if "YES" the dialogue is ended, otherwise the dialogue returns to the first step S1.

[0165] The processing of each step in the flow charts shown in FIGS. 9, 10, and 13 to 16 is merely an example, and the processing order of each step may be changed as long as the same results are obtained.

[0166] Furthermore, the specific values ​​of the angles, durations, etc. given above are merely examples and can be changed as needed. [Explanation of symbols]

[0167] 10...User support system 12...Communication robot 16...server 28...Arms 36, 56...CPU 40, 58...Memory 100...users

Claims

1. A user assistance system that assists a user with a robot that can hug the user, the robot has a contact action providing unit that can perform a plurality of types of contact actions on the user while hugging the user, A plurality of types of dialogue acts are set as dialogue acts between the robot and the user, and A user assistance system comprising a type determination means for determining a type of the contact action based on the type of the dialogue act.

2. the contact action providing unit is capable of providing contact actions to a plurality of contact positions of the hugging user, 2. The user assistance system according to claim 1, further comprising: a position determining means for determining the contact position based on the type of dialogue act determined by said type determining means.

3. a prior action storage means for storing a contact action in a dialogue act prior to the current dialogue act as a prior action; 2. The user assistance system according to claim 1, further comprising an execution decision means for deciding whether to execute a contact action by said contact action applying means in a current dialogue act based on information of a pre-action stored in said pre-action storage means.

4. a prior action storage means for storing a contact action in a dialogue act prior to the current dialogue act as a prior action; 2. The user assistance system according to claim 1, wherein said type determining means determines the type of said contact action based on a previous action stored in said previous action storage means and the type of the current dialogue act.

5. a prior action storage means for storing a contact action in a dialogue act prior to the current dialogue act as a prior action; 3. The user assistance system according to claim 2, wherein said type determining means determines the position of said contact action based on the previous action stored in said previous action storage means and the type of the current dialogue act.

6. A user assistance system that assists a user with a robot that can hug the user, the robot has a contact action providing unit that can perform a plurality of types of contact actions on the user at a plurality of contact positions while hugging the user, and further, a plurality of types of dialogue acts are set as dialogue acts between the robot and the user; type determination means for determining the type of the contact gesture based on the type of the dialogue act; and A user assistance system comprising: a position determining means for determining the contact position based on the type of dialogue act determined by the type determining means.

7. A user assistance device that assists a user by a robot that can hug the user, wherein the robot has a contact action providing unit that can perform a plurality of types of contact actions on the user at a plurality of contact positions while hugging the user, and further, a plurality of types of dialogue acts are set as dialogue acts between the robot and the user; type determination means for determining the type of the contact gesture based on the type of the dialogue act; and A user assistance device comprising: a position determining means for determining the contact position based on the type of dialogue act determined by the type determining means.

8. A user assistance program for a user assistance device that assists a user with a robot that can hug the user, wherein the robot has a contact action providing unit that can perform a plurality of types of contact actions on the user at a plurality of contact positions while hugging the user, and further, a plurality of types of dialogue acts are set as dialogue acts between the robot and the user, and the user assistance program causes a computer of the user assistance device to: type determination means for determining the type of the contact gesture based on the type of the dialogue act; and a user assistance program that causes the program to function as position determining means for determining the contact position based on the type of dialogue act determined by the type determining means;

9. A user assistance method for assisting a user by a robot capable of hugging the user, wherein the robot has a contact action providing unit that can perform a plurality of types of contact actions on the user at a plurality of contact positions while hugging the user, and further, a plurality of types of dialogue acts are set as dialogue acts between the robot and the user; determining a type of the contact gesture based on the type of the dialogue act; determining the contact position based on the determined type of dialogue act.

Images