Robot system, computer, and robot control method

Abstract

To notify a user of a state of an apparatus while reducing discomfort of the user.SOLUTION: Provided is a robot system that includes: a control part for controlling an operation of a robot; a sensor part for observing a situation around the robot; and a notification part for executing information notice from the robot to a user, wherein the control part determines a content of the information notice to the user on the basis of a situation detected by an installed apparatus, the notification part notifies the user of the first information notice according to the status of the facility equipment, and the control unit determines the mode of the second information notice to be notified to the user subsequent to the first information notice depending on whether the user's response to the first information notice is a predetermined response.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a robot system, and more particularly to a technology for notifying a user of the status of facility equipment. [Background technology]

[0002] A technology has been developed that notifies the user if food is left unremoved from a cooking appliance after cooking has finished.

[0003] The following prior art exists as background art in this technical field: Patent Document 1 (JP 2019-86197 A) describes a cooking device that has, as heating modes, a residual heat mode in which residual heat is generated inside the cooking chamber after cooking is completed, and a non-residual heat mode in which residual heat is not generated inside the cooking chamber after cooking is completed, and that issues a warning that the food has been forgotten to be removed if a predetermined time has passed after cooking is completed without the food being removed, in which the warning that the food has been forgotten to be removed after cooking in the residual heat mode is issued by a first method, and the warning that the food has been forgotten to be removed after cooking in the non-residual heat mode is issued by a second method, and the warning methods used for the first and second methods are different from each other. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-86197 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 reliably notifies the user of the status of the cooking appliance by changing the notification method, such as by using a first method to notify the user of forgotten removal of food after cooking in residual heat mode and a second method to notify the user of forgotten removal of food after cooking in non-residual heat mode. However, the user may find the notification unpleasant. Furthermore, even if the user acknowledges the notification, the content of the notification often requires the user to interrupt their current work, which is not a desired task and can be annoying.

[0006] The present invention aims to notify a user of the status of a device while reducing the user's discomfort. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a robot system includes a control unit that controls the operation of a robot, a sensor unit that observes a situation around the robot, and a notification unit that executes information notification from the robot to a user, wherein the control unit determines the content of the information notification to the user based on a situation detected by an equipment, the notification unit notifies the user of a first information notification according to the situation of the equipment, and the control unit determines the mode of a second information notification to be notified to the user following the first information notification depending on whether the user's reaction to the first information notification is a predetermined reaction. The notification unit performs the second information notification in accordance with a timing at which a change in at least one of the user's movement and facial expression is detected. It is characterized by: [Effects of the Invention]

[0008] According to one aspect of the present invention, it is possible to reliably notify a user, and by providing a notification similar to two-way communication, it is possible to notify the user of the status of a device while reducing the user's discomfort. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of a robot according to a first embodiment. [Figure 2] 1 is a flowchart of a process executed by the facility device of the first embodiment. [Figure 3] 1 is a flowchart of a process executed by a robot according to a first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of an information notification mode. [Figure 5] FIG. 10 is a diagram illustrating an example of a first information notification in the first embodiment. [Figure 6]FIG. 10 is a diagram showing an example of a joyful motion in the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a joyful motion in the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a second information notification in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a third information notification in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a disappointed motion in the first embodiment. [Figure 11] FIG. 10 is a block diagram showing the configuration of a robot according to a second embodiment. [Figure 12] 10 is a flowchart of a process executed by a robot according to a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a first information notification in the second embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a joyful motion in the second embodiment. [Figure 15] FIG. 10 is a diagram showing an example of a joyful motion in the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a second information notification in the second embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of a third information notification in the second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of a disappointed motion in the second embodiment. [Figure 19] FIG. 10 is a block diagram showing the configuration of a robot according to a third embodiment. [Figure 20] FIG. 11 is a block diagram showing the configuration of a server according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Example 1 FIG. 1 is a block diagram showing the configuration of a robot 1 according to a first embodiment.

[0011] The robot 1 of the first embodiment is an electrically powered device that runs on a storage battery, recognizes the external environment (for example, determines the state of a person), and operates autonomously. The robot 1 is preferably a friendly, lovable robot that is roughly human-like (for example, child-like) or pet-like, so that people who receive notifications from the robot 1 can respond comfortably.

[0012] The robot 1 includes a control unit 10, a sensor unit 50, a communication unit 60, a drive unit 70, and an expression unit 90.

[0013] The control unit 10 has an information notification determination unit 20, an action decision unit 30, and an action execution unit 40. The control unit 10 is implemented in a microcomputer including a central processing unit (CPU) and a memory. The CPU is a processing unit that executes programs stored in a storage device. The memory is a storage device that has a non-volatile storage area that retains stored data even when the power is turned off, and a volatile storage area that temporarily stores data and programs used by the CPU when executing a program. The various functions of the control unit 10 are implemented by the CPU executing the programs stored in the memory.

[0014] The information notification determination unit 20 is a control unit that determines surrounding people and objects and their states from information external to the robot 1 detected by the sensor unit 50, and includes a person determination unit 21, an object determination unit 22, a human state determination unit 23, a voice determination unit 24, and an expected action recording unit 25. The person determination unit 21 analyzes images captured by the camera 51 to determine people in the images. The person determination unit 21 also calculates the distance to the person using the measurement results of the distance sensor 53. This distance is used to approach the person or make physical contact (shaking, hitting) with them. The object determination unit 22 analyzes images captured by the camera 51 to determine objects (e.g., desks and chairs) in the images. The object determination unit 22 also calculates the distance to the object using the measurement results of the distance sensor 53. This distance is used to approach the object or make physical contact (shaking, hitting) with them. The human state determination unit 23 analyzes the image captured by the camera 51, the detection result of the touch sensor 52, and the analysis result by the voice determination unit 24, and determines the state of the person determined from the image (for example, standing, sitting, operating a smartphone, noticing the information notification, being grateful, being annoyed, trying to chase away the robot 1, ignoring it, not hearing it, stroking it, etc.). The human state determination unit 23 also determines the behavior of the notification target, compares the determined behavior of the target (human state determination result) with the expected behavior recorded in the expected behavior recording unit 25, and determines whether the information was conveyed to the person. The voice determination unit 24 analyzes the sound collected by the microphone 54 and determines the words uttered by the person. The expected behavior recording unit 25 pre-records actions that are expected to be performed by a person after noticing the notification content (for example, standing up, operating a smartphone, moving, changing the direction of the body, uttering such as "Oh, that's right").

[0015] The behavior decision unit 30 is a control unit that determines the behavior of the robot 1, and includes a behavior selection instruction unit 31, an urgency determination unit 32, a timing determination unit 33, and a timer 34. The behavior selection instruction unit 31 selects an behavior of the robot 1 based on the determination results by the information notification determination unit 20 of the state of the facility equipment and the state of the person, etc., and instructs the behavior execution unit 40 to perform the behavior. The urgency determination unit 32 determines the urgency of the state of the facility equipment. The timing determination unit 33 determines the timing of information notification using the time measurement result of the timer 34 or the result of the person state determination unit 23.

[0016] The behavior execution unit 40 is a control unit that generates control signals for the robot 1 to operate, and includes a movement control unit 41, a main body control unit 42, a voice control unit 43, and a facial expression control unit 44. The movement control unit 41 controls the movement of the robot 1 by rotating its wheels. For example, as described below with reference to Figures 9 and 10, the robot 1 has a front guide wheel and a rear running wheel. It moves forward and backward by rotating the running wheels driven by a right wheel motor 71 and a left wheel motor 72, and changes direction by independently controlling the rotation speed of the running wheels. The main body control unit 42 controls the movement of the main body of the robot 1. For example, as shown in Figure 7, the robot 1 has a base to which wheels are attached, a main body located on the base, and a head attached to the main body. The main body control unit 42 controls body swaying movements by tilting the head, rotating or tilting the main body on the base, etc. The voice control unit 43 controls the sound emitted by the speaker 91 of the robot 1. The voice control unit 43 may control the generation of sounds such as (infant) words similar to language, moans, animal or animal-like cries, warning sounds, etc. The facial expression control unit 44 controls the facial expression of the robot 1 to be displayed on the face display 92.

[0017] The sensor unit 50 is an observation device that detects information outside the robot 1, and includes a camera 51, a touch sensor 52, a distance sensor 53, and a microphone 54. The camera 51 captures images of the surroundings of the robot 1. The touch sensor 52 is provided on the surface of the robot 1 (for example, on the head) and detects contact with the robot 1 by a person. The distance sensor 53 measures the distance to objects around the robot 1. The microphone 54 collects sounds around the robot 1.

[0018] The communication unit 60 is a control unit that controls communication between the robot 1 and other devices, and includes a WiFi communication unit 61 and a Bluetooth communication unit 62. The WiFi communication unit 61 connects to a wireless LAN network using the WiFi protocol. The Bluetooth communication unit 62 communicates with other devices in close proximity using the Bluetooth protocol.

[0019] The drive unit 70 is a power source for operating each part of the robot 1 and includes a right wheel motor 71, a left wheel motor 72, a torso drive motor 73, a head drive motor 74, a right arm drive motor 75, a left arm drive motor 76, a right finger drive motor 77, a left finger drive motor 78, a blower fan 79, and an LCD drive motor 80. The right wheel motor 71 and the left wheel motor 72 drive the right and left wheels, respectively, provided on the bottom of the robot 1. The torso drive motor 73 is a motor for swinging the torso of the robot 1. The head drive motor 74 is a motor for swinging the head of the robot 1. The right arm drive motor 75 and the left arm drive motor 76 are motors for moving the right arm and left arm of the robot 1, respectively. The right finger drive motor 77 and the left finger drive motor 78 are motors for moving the fingers of the robot 1. The blower fan 79 is a blower for blowing air from the robot 1. The blower fan 79 blows air to the person to be notified, allowing the person to feel a tactile sensation without coming into contact with the person to be notified. The liquid crystal drive motor 80 is a motor that raises and lowers the lift-up display 93.

[0020] The expression unit 90 is an interface that outputs notifications from the robot 1 to people, and includes a speaker 91, a face display 92, and a lift-up display 93. The speaker 91 outputs sounds from the robot 1. The face display 92 is a display device that is provided on the front of the head of the robot 1 and displays the facial expressions of the robot 1. The lift-up display 93 is a display device that rises, for example, from the back side of the robot 1 to a height that exceeds the head (for example, the height of the face of a standing person), and displays a predetermined message to be conveyed to people.

[0021] FIG. 2 is a flowchart of the process executed by the facility device of the first embodiment, and explains the washing completion notification process executed by the washing machine, which is an example of the facility device.

[0022] First, the facility device notifies the user that the operation has finished (S500), for example, by sounding an end sound when the washing is finished.

[0023] Next, the facility device determines whether the next process that the user should perform is being performed (S501). For example, a washing machine determines whether the door of the washing tub has been opened and laundry has been removed.

[0024] If the user performs the predetermined process, the facility device ends the process. On the other hand, if a predetermined time (e.g., 10 minutes) has passed without the user performing the predetermined process (Y in S502), the facility device transmits the identification information and status of the facility device to the robot 1 (S503).

[0025] In Figure 2, a washing machine is used as an example of equipment, but other household electrical appliances such as washer-dryers, microwave ovens, televisions, and self-propelled vacuum cleaners, as well as home equipment such as bathtubs and toilets, and office equipment such as copiers, may also be used, and they may be installed indoors or outdoors.The status of equipment includes conditions that require human operation, instructions, or signals in the next stage, such as leaving the equipment unattended, completing processing, forgetting to turn off the power, using the equipment for longer than usual, an abnormality, a malfunction, a prompt to start processing, deterioration of parts, replenishment of consumables, and the presence of an object that interferes with the operation of the equipment.

[0026] FIG. 3 is a flowchart of the process executed by the robot 1 of the first embodiment.

[0027] First, when the robot 1 receives a situation from an equipment (S510), the person determination unit 21 determines whether a person is nearby (for example, within a visible range) from the image captured by the camera 51 (S511). If no person is nearby, the movement control unit 41 controls the rotation of the right wheel motor 71 and the left wheel motor 72 to start a person search movement (S512). This person search movement may involve random movement within the robot 1's travelable area, or may involve movement to a place where a person is likely to be (such as the living room) according to a predetermined pattern.

[0028] If it is determined that a person is nearby, the movement control unit 41 controls the rotation of the right wheel motor 71 and the left wheel motor 72 to move closer to the person to be notified (S513). After moving close to the person to be notified, the voice control unit 43 controls the speaker 91 to output the first information notification (S514). The first information notification is a sound notification as shown in FIG. 5, and a soft sound such as an animal cry or an inarticulate voice like that of a child is desirable to make the robot feel familiar and affectionate. The notification sound should preferably be different from the sound made during normal operation. The content of the notification from the robot 1 is that the current task needs to be interrupted, and is not a desired task, so a soft sound is desirable.

[0029] The human state determination unit 23 determines the reaction of the notification target person from the video captured by the camera 51 (S515). If the notification target person does not react as expected to the first information notification (N in S516), the process proceeds to step S520 for the second information notification. On the other hand, if the notification target person reacts as expected to the first information notification (Y in S516), the facial expression control unit 44 controls the facial display 92 to display a smile that expresses happiness or relief (S517). For example, as shown in FIG. 6, if the notification target person realizes that they forgot to take out their clothes in response to the first information notification and stands up, the robot 1 will have a relieved and happy expression. If the robot 1 is satisfied with the notification target person's behavior, it conveys positive feelings to the notification target person, allowing the notification target person to act in a good mood.

[0030] After that, as shown in Figure 7, when the person receiving the notification thanks or strokes Robot 1, Robot 1 feels relieved, happy, and starts dancing. Seeing the behavior of Robot 1, the person receiving the notification feels positive emotions and is able to act in a good mood.

[0031] Thereafter, the movement control unit 41 controls the rotation of the right wheel motor 71 and the left wheel motor 72 so that the robot 1 moves away from the person to be notified (S518).

[0032] On the other hand, if the person to be notified does not respond as expected (N in S516), the urgency determination unit 32 determines the urgency of the notification in order to send a second notification (S520). The urgency of the notification is predetermined according to the facility device that sent the notification and the type of notification, and is recorded in a predetermined table. For example, it is advisable to determine the urgency of a notification as high if a person has collapsed, a pot has boiled over on a gas stove, or other events that could pose a risk to life or property if no immediate action is taken.

[0033] If the urgency of the notification is not high, the human state determination unit 23 determines whether the state of the person to be notified has changed before a predetermined time (e.g., 30 seconds) has elapsed (S522, S523). Interruption of the current work is unpleasant for the person to be notified. For this reason, if a notification is sent when there is a change in the movement of the person to be notified, such as a change in the direction of the face, crossing or uncrossing the legs, stretching, or a change in facial expression, the notification can be sent without interrupting the person to be notified's concentration on the work.

[0034] On the other hand, if the condition of the notification target has changed (Y in S522), if a predetermined time (30 seconds) has elapsed since the notification of a low level of urgency (Y in S523), or if a predetermined time (e.g., 5 seconds) has elapsed since the notification of a high level of urgency (S521), the speech control unit 43 controls the speaker 91 to output a second information notification (S524). The second information notification may be made closer and at a louder volume than the first information notification, as shown in FIG. 8 . The facial expression control unit 44 may also control the display of an angry expression on the face display 92. In this way, when the level of urgency is high, the time until the second notification is shortened, thereby increasing the frequency of notifications, thereby enabling an early response to the emergency. The second information notification may be made in a different manner from the first information notification (e.g., at a louder volume than the first information notification, or by a different sound or method (e.g., visual) than the first information notification). The notification method for the second information notification may also be changed depending on the level of urgency.

[0035] The human state determination unit 23 determines the reaction of the notification target person from the video captured by the camera 51 (S525). If the notification target person reacts as expected (Y in S525), the process proceeds to step S517, where the facial expression control unit 44 controls the face display 92 to display a smile that expresses happiness or relief. On the other hand, if the notification target person does not react as expected (N in S525), the process proceeds to step S526 to send a third notification.

[0036] On the other hand, if the notification target does not respond as expected to the second information notification (N in S525), the urgency determination unit 32 determines the urgency of the notification for the third information notification (S526). For example, as shown in Fig. 8, if the notification target does not understand the intention of the second information notification, becomes annoyed, and ignores it, processing for the third information notification is started.

[0037] If the urgency of the notification is not high, the person state determining unit 23 determines whether the state of the person to be notified has changed before a predetermined time (for example, 20 seconds) has elapsed (S528, S529).

[0038] On the other hand, if the condition of the notification target changes (Y in S528), if a predetermined time (20 seconds) has elapsed since the notification of a low level of urgency (Y in S529), or if a predetermined time (e.g., 3 seconds) has elapsed since the notification of a high level of urgency (S527), the voice control unit 43 controls the speaker 91 to output a third information notification (S530). The third information notification may be closer than the second information notification and at a louder volume. The main body control unit 42 may also drive at least one of the torso drive motor 73, the right arm drive motor 75, and the left arm drive motor 76 to shake the desk on which the notification target is working. Furthermore, the facial expression control unit 44 may control the face display 92 to display an angry expression. As the number of notifications increases (e.g., the first, second, and third notifications), the time until notification is shortened to encourage early awareness. The time until notification may be the same regardless of the number of notifications. In this way, when the level of urgency is high, the time until the third notification is shortened, enabling an early response to the emergency. The third information notification may be given in a different manner than the second information notification (for example, at a louder volume than the second information notification, or by a different sound or method (for example, visual) than the second information notification). The notification method for the third information notification may also be changed depending on the level of urgency. The upper limit of the number of information notifications may also be changed depending on the level of urgency, and notifications may be given four, five times, and so on until the person to be notified notices it.

[0039] The human state determination unit 23 determines the reaction of the notification target person from the video captured by the camera 51 (S531). If the notification target person reacts as expected (Y in S531), the process proceeds to step S517, where the facial expression control unit 44 controls the facial display 92 to display a smile that expresses happiness or relief. On the other hand, if the notification target person does not react as expected (N in S531), the facial expression control unit 44 controls the facial display 92 to display a face that expresses dissatisfaction or disappointment (S532). By displaying a dissatisfied or disappointed expression, if the notification target person intentionally ignores the information notification from the robot 1, the notification target person understands that the robot 1 understands the notification target person's intention, and the notification target person feels attached to the robot 1 because the robot 1 followed their intention.

[0040] For example, as shown in Figure 9, Robot 1 tries to get the target person's attention by shaking the desk as a third notification without using sound. The target person notices the third notification, but does not understand the intention, finds it annoying, and ignores it. Then, as shown in Figure 10, the target person does not react as expected, and Robot 1 leaves with a dissatisfied look on its face, as the target person does not understand the intention of the notification.

[0041] FIG. 4 is a diagram showing an example of an information notification mode.

[0042] The manner of information notification is determined depending on the number of times, and the behavior executing unit 40 holds definitions of the manner of information notification, for example in table format, and refers to this information to determine the manner of information notification for each time.

[0043] The sound of the information notification may be different depending on the number of times the information notification is made, such as a soft sound like an animal cry or a child's voice for the first and second times, and a hard sound like an animal cry or a child's voice that strongly calls out to the notification recipient for the third time. The volume of the information notification may be normal for the first time and loud for the second and third times, so that the volume varies depending on the number of times the information notification is made. The facial expression of the robot 1 (display content on the facial display 92) when notifying the information may be normal for the first time and angry for the second and third times, so that the facial expression may vary depending on the number of times the information notification is made. The distance between the robot 1 and the notification recipient when notifying the information may be medium for the first time and close for the second and third times, so that the distance to the notification recipient may vary depending on the number of times the information notification is made. The robot 1's behavior when notifying the information may be normal for the first time, raising its arms for the second time, and shaking the desk or chair used by the notification recipient for the third time, so that the posture may vary depending on the number of times the information notification is made.

[0044] More specifically, like the "Call 1" pattern, which uses different sounds depending on the number of times the information is notified, the first time it could be an animal cry or a baby's cry, the second time it could be a sound with the audible frequency increased and an easy-to-hear frequency band emphasized, the third time it could be a scream or cry to emphasize the urgency, the fourth time it could be an unpleasant sound such as a cracking sound, a rustling sound or a siren, and the fifth time it could be a crying sound like a child throwing a tantrum.

[0045] Also, like the "Call 2" pattern, which uses different sounds depending on the number of times the information is notified, the first time could be an animal cry or a child's voice, the second time a predetermined keyword, the third time a request "Do this...", the fourth time a questioning sound "Do you not need to do this?", and the fifth time a commanding sound "Do this...".

[0046] Also, like the "Call 3" pattern, which uses different sounds depending on the number of times the information is notified, the first time it could be an animal cry or a baby's voice, the second time it could be the sound of a predetermined keyword, the third time it could be a shriek, and the fourth time it could be an angry or hysterical voice.

[0047] Also, like the "Expression 1" pattern, which displays information differently depending on the number of times it is notified, it is a good idea to use a normal expression the first time, an angry expression using the eyes, mouth, and eyebrows the second time, a furious expression with a change in facial color the third time, and a crying expression the fourth time.

[0048] Also, like the pattern of "Facial Expression 2," which notifies information in different ways depending on the number of times the information is notified, the first time it is displayed with a normal facial expression, the second time it displays predetermined characters, the third time it displays the predetermined characters in large size, the fourth time it flashes the characters or background, and the fifth time it increases the light intensity of the face display 92 or speeds up the flashing speed.

[0049] Also, like the pattern of "Movement 1," which performs different actions depending on the number of times information is notified, the first time it is in a normal posture, the second time it swings its arms up, the third time it swings its arms up widely and / or quickly, the fourth time it shakes its body, and the fifth time it moves around or shakes its body vigorously. The fifth time it may perform an action that it does not normally perform, such as throwing a tantrum, kicking its legs, or turning its head.

[0050] Also, as in the pattern of "Action 2," which uses different actions depending on the number of times the information is notified, the first time is in a normal posture, the second is an action of raising the arms, the third is an action of shaking the desk or chair, the fourth is an action of touching the body of the person to be notified (for example, the legs), and the fifth is an action of touching the body of the person to be notified in a different manner (multiple times, quickly, at different temperatures, etc.).

[0051] In this way, with each notification, the content of the notification, the content of the sound, the volume, the sound quality, the display, the action, etc. change, making the action bigger, faster, exaggerated, or by moving other parts, making it easier for the person being notified to notice and strengthening the demand for action.

[0052] According to the first embodiment of the present invention, by changing the method of the subsequent second information notification depending on the user's reaction to the first information notification, the user can be notified reliably, and by not only notifying but also notifying the user of the device status while reducing the user's discomfort through a notification similar to two-way communication. Note that instead of using a positive expression (S517) in the case of an expected reaction (Y in S516), the second notification may be implemented with a different reaction only in the case of an unexpected reaction (N in S516), or conversely, the second notification may be implemented in the case of an expected reaction (Y in S516).

[0053] <Example 2> Next, a second embodiment of the present invention will be described. While the robot 1 of the first embodiment described above is self-propelled, the robot 1 of the second embodiment is portable and does not move autonomously (for example, installed on a desk). In the second embodiment, differences from the first embodiment will be mainly described, and the same configurations and processes as those of the first embodiment will be assigned the same reference numerals, and their description will be omitted.

[0054] FIG. 11 is a block diagram showing the configuration of the robot 1 according to the second embodiment.

[0055] The robot 1 of Example 2 includes a control unit 10, a sensor unit 50, a communication unit 60, a drive unit 70, and an expression unit 90. The control unit 10 includes an information notification determination unit 20, a behavior decision unit 30, and a behavior execution unit 40. The information notification determination unit 20 is a control unit that determines surrounding people and objects and their states from information external to the robot 1 detected by the sensor unit 50, and includes a person determination unit 21, an object determination unit 22, a human state determination unit 23, a voice determination unit 24, and an expected action recording unit 25. The behavior decision unit 30 is a control unit that determines the behavior of the robot 1, and includes a behavior selection instruction unit 31, an urgency determination unit 32, a timing determination unit 33, and a timer 34. The behavior execution unit 40 is a control unit that generates control signals for the robot 1 to operate, and includes a main body control unit 42, a voice control unit 43, and a facial expression control unit 44. The sensor unit 50 is an observation device that detects information external to the robot 1, and includes a camera 51, a touch sensor 52, a distance sensor 53, and a microphone 54. The communication unit 60 is a control unit that controls communication between the robot 1 and other devices, and includes a WiFi communication unit 61 and a Bluetooth communication unit 62. The drive unit 70 is a power source for operating each part of the robot 1, and includes a torso drive motor 73, a head drive motor 74, a right arm drive motor 75, a left arm drive motor 76, a right finger drive motor 77, a left finger drive motor 78, a blower fan 79, and an LCD drive motor 80. The expression unit 90 is an interface that outputs notifications from the robot 1 to a person, and includes a speaker 91, a face display 92, and a lift-up display 93. The configurations and functions of these parts are the same as those of the parts with the same reference numerals in the first embodiment.

[0056] 12 is a flowchart of the process executed by the robot 1 of Example 2. Since the robot 1 of Example 2 does not have a movement function, it does not execute the human search process of the robot 1 of Example 1 and does not approach the person to be notified.

[0057] First, when the robot 1 receives a status from the facility equipment (S510), the person determination unit 21 determines whether a person is nearby (for example, within a visible range) from the video captured by the camera 51 for a predetermined time (for example, 30 seconds) (S511, S542). There is a possibility that the person has simply left their seat, so the robot 1 waits for the predetermined time for the person to return.

[0058] If it is determined that a person is nearby, the voice control unit 43 controls the speaker 91 to output the first information notification to the determined person as the notification target (S514). The first information notification is a notification by sound, as shown in FIG. 13, and a soft sound such as an animal cry or an unclear voice like that of a child is desirable to make the robot feel familiar and affectionate. The notification sound should be different from the sound made during normal operation. The content of the notification from the robot 1 is to request the robot to interrupt its current work, and since it is a work that the robot does not actively want to do, a soft sound is desirable.

[0059] The human state determination unit 23 determines the reaction of the notification target from the video captured by the camera 51 (S515). If the notification target does not react as expected to the first information notification (N in S516), the process proceeds to step S520 for the second information notification. On the other hand, if the notification target reacts as expected to the first information notification (Y in S516), the facial expression control unit 44 controls the facial display 92 to display a smile expressing happiness or relief (S517). Furthermore, the main body control unit 42 may drive the right arm drive motor 75 and the left arm drive motor 76 to perform a cheer gesture. For example, as shown in FIG. 14, if the notification target responds to the first information notification by removing the headphones and realizing that they forgot to take out their clothes and standing up, the robot 1 will show a relieved and happy expression and perform a cheer gesture. Also, as shown in FIG. 15, if the notification target responds to the first information notification by checking the status of the washing machine on their smartphone, the robot 1 will show a relieved and happy expression and perform a cheer gesture. If Robot 1 is satisfied with the behavior of the person to be notified, it conveys positive feelings to the person to be notified, which causes the person to behave in a good mood.

[0060] Thereafter, the main body control unit 42 controls the robot 1 so that it returns to the operation before the information notification (S548).

[0061] On the other hand, if the person to be notified does not respond as expected (N in S516), the urgency determination unit 32 determines the urgency of the notification in order to send a second notification (S520). The urgency of the notification is predetermined according to the facility device that sent the notification and the type of notification, and is recorded in a predetermined table. For example, it is advisable to determine the urgency of a notification as high if a person has collapsed, a pot has boiled over on a gas stove, or other events that could pose a risk to life or property if no immediate action is taken.

[0062] If the urgency of the notification is not high, the human state determination unit 23 determines whether the state of the person to be notified has changed before a predetermined time (e.g., 30 seconds) has elapsed (S522, S523). Interruption of the current work is unpleasant for the person to be notified. For this reason, if a notification is sent when there is a change in the movement of the person to be notified, such as a change in the direction of the face, crossing or uncrossing the legs, stretching, or a change in facial expression, the notification can be sent without interrupting the person to be notified's concentration on the work.

[0063] On the other hand, if the condition of the notification target has changed (Y in S522), if a predetermined time (30 seconds) has elapsed since the notification of a low level of urgency (Y in S523), or if a predetermined time (e.g., 5 seconds) has elapsed since the notification of a high level of urgency (S521), the speech control unit 43 controls the speaker 91 to output a second information notification (S524). The second information notification may be made closer and at a louder volume than the first information notification, as shown in FIG. 16 . The facial expression control unit 44 may also control the display of an angry expression on the face display 92. In this way, when the level of urgency is high, the time until the second notification is made is shortened, enabling an early response to the emergency. The second information notification may be made in a manner different from the first information notification (e.g., at a louder volume than the first information notification, or by a different sound or method (e.g., visual) than the first information notification). The notification method for the second information notification may also be changed depending on the level of urgency.

[0064] The human state determination unit 23 determines the reaction of the notification target person from the video captured by the camera 51 (S525). If the notification target person reacts as expected (Y in S525), the process proceeds to step S517, where the facial expression control unit 44 controls the face display 92 to display a smile that expresses happiness or relief. On the other hand, if the notification target person does not react as expected (N in S525), the process proceeds to step S526 to send a third notification.

[0065] On the other hand, if the notification target does not respond as expected to the second information notification (N in S525), the urgency determination unit 32 determines the urgency of the notification for the third information notification (S526). For example, as shown in Fig. 16, if the notification target ignores the second information notification, processing for the third information notification is started.

[0066] If the urgency of the notification is not high, the person state determining unit 23 determines whether the state of the person to be notified has changed before a predetermined time (for example, 20 seconds) has elapsed (S528, S529).

[0067] On the other hand, if the condition of the notification target changes (Y in S528), if a predetermined time (20 seconds) has elapsed since the notification of a low level of urgency (Y in S529), or if a predetermined time (e.g., 3 seconds) has elapsed since the notification of a high level of urgency (S527), the voice control unit 43 controls the speaker 91 to output a third information notification (S530). The third information notification may be closer than the second information notification and at a louder volume. The main body control unit 42 may also drive at least one of the torso drive motor 73, the right arm drive motor 75, and the left arm drive motor 76 to shake the desk on which the notification target is working. Furthermore, the facial expression control unit 44 may control the face display 92 to display an angry expression. As the number of notifications increases (e.g., the first, second, and third notifications), the time until notification is shortened to encourage early awareness. The time until notification may be the same regardless of the number of notifications. In this way, when the level of urgency is high, the time until the third notification is shortened, enabling an early response to the emergency. The third information notification may be given in a different manner than the second information notification (for example, at a louder volume than the second information notification, or by a different sound or method (for example, visual) than the second information notification). The notification method for the third information notification may also be changed depending on the level of urgency. The upper limit of the number of information notifications may also be changed depending on the level of urgency, and notifications may be given four, five times, and so on until the person to be notified notices it.

[0068] The human state determination unit 23 determines the reaction of the notification target person from the video captured by the camera 51 (S531). If the notification target person reacts as expected (Y in S531), the process proceeds to step S517, where the facial expression control unit 44 controls the facial display 92 to display a smile that expresses happiness or relief. On the other hand, if the notification target person does not react as expected (N in S531), the facial expression control unit 44 controls the facial display 92 to display a face that expresses dissatisfaction or disappointment (S532). By displaying a dissatisfied or disappointed expression, if the notification target person intentionally ignores the information notification from the robot 1, the notification target person understands that the robot 1 understands the notification target person's intention, and the notification target person feels attached to the robot 1 because the robot 1 followed their intention.

[0069] For example, as shown in Fig. 17, robot 1 tries to get the notification recipient's attention by using other methods (for example, by tapping on the desk) in addition to sound as the third information notification. The notification recipient notices the third information notification, but does not understand the intention, finds it annoying, and ignores it. Then, as shown in Fig. 18, if the notification recipient does not notice the third information notification and ignores it, the notification recipient does not react as expected, and robot 1 becomes dissatisfied because the intention of the information notification is not understood.

[0070] Thereafter, the main body control unit 42 controls the robot 1 so that it returns to the operation before the information notification (S548).

[0071] As described above, according to the second embodiment of the present invention, even in the case of a robot 1 that cannot travel autonomously, the method of the subsequent second information notification can be changed depending on the user's reaction to the first information notification, thereby ensuring that the user is notified of the status of the device while reducing the user's discomfort, by providing a notification that is similar to two-way communication rather than just a notification.

[0072] Example 3 Next, a third embodiment of the present invention will be described. While the robots 1 in the first and second embodiments described above autonomously determine their movements, the robot 1 in the third embodiment moves according to instructions from the server 2. In the third embodiment, differences from the first and second embodiments will be mainly described, and the same configurations and processes as those in the first and second embodiments will be assigned the same reference numerals and descriptions thereof will be omitted.

[0073] FIG. 19 is a block diagram showing the configuration of the robot 1 according to the third embodiment.

[0074] The robot 1 of Example 3 includes a control unit 10, a sensor unit 50, a communication unit 60, a drive unit 70, and an expression unit 90. The control unit 10 includes an information notification determination unit 20, a behavior decision unit 30, and a behavior execution unit 40. The information notification determination unit 20 is a control unit that determines surrounding people and objects based on information external to the robot 1 detected by the sensor unit 50, and includes a person determination unit 21 and an object determination unit 22. The behavior decision unit 30 is a control unit that determines the behavior of the robot 1, and includes a behavior instruction unit 35 and a timer 34. The behavior execution unit 40 is a control unit that generates control signals for the robot 1 to operate, and includes a movement control unit 41, a body control unit 42, a voice control unit 43, and a facial expression control unit 44. The sensor unit 50 is an observation device that detects information external to the robot 1, and includes a camera 51, a touch sensor 52, a distance sensor 53, and a microphone 54. The communication unit 60 is a control unit that controls communication between the robot 1 and other devices, and includes a Wi-Fi communication unit 61 and a Bluetooth communication unit 62. The drive unit 70 is a power source for operating each part of the robot 1, and includes a right wheel motor 71, a left wheel motor 72, a torso drive motor 73, a head drive motor 74, a right arm drive motor 75, a left arm drive motor 76, a right finger drive motor 77, a left finger drive motor 78, a blower fan 79, and a liquid crystal drive motor 80. The expression unit 90 is an interface that outputs notifications from the robot 1 to a person, and includes a speaker 91, a face display 92, and a lift-up display 93. The configurations and functions of these parts are the same as those of the parts with the same reference numerals in the first embodiment.

[0075] The action instruction unit 35 instructs the action execution unit 40 to perform an action based on the action selected by the action selection unit 231 of the server 2.

[0076] The robot 1 communicates with the server 2 via the WiFi communication unit 61 and receives operation instructions from the server.

[0077] FIG. 20 is a block diagram showing the configuration of the server 2 according to the third embodiment.

[0078] The server 2 includes an information notification determination unit 220 , a behavior determination unit 230 , and a communication unit 260 .

[0079] The information notification determination unit 220 is a control unit that determines the state of surrounding people and objects from external information detected by the robot 1, and includes a human state determination unit 223, a voice determination unit 224, and an expected action recording unit 225. The human state determination unit 223 analyzes the image captured by the camera 51 of the robot 1, the detection result of the touch sensor 52 of the robot 1, and the determination result by the voice determination unit 24 of the robot 1, and determines the state of the person determined from the image (e.g., standing, sitting, operating a smartphone, noticing the information notification, being grateful, being annoyed, trying to chase away the robot 1, ignoring, not hearing, stroking, etc.). In addition, the human state determination unit 223 compares the determined behavior of the subject (human state determination unit result) with the expected action recorded in the expected action recording unit 25, and determines whether the information was conveyed to the person. The voice determination unit 224 analyzes the sound collected by the microphone 54 and determines the words uttered by the person. The expected action recording unit 225 pre-records actions that are expected to be performed by a person after noticing the content of the notification (for example, standing up, operating a smartphone, moving, turning one's body, uttering words such as "Oh, that's right").

[0080] The behavior decision unit 230 is a control unit that determines the behavior of the robot 1, and includes a behavior selection unit 231, an urgency determination unit 232, and a timing determination unit 233. The behavior selection unit 231 selects the behavior of the robot 1 based on the determination results by the information notification determination unit 20 of the state of the facility equipment and the state of the person, etc. The urgency determination unit 232 determines the urgency of the state of the facility equipment. The timing determination unit 233 uses a timer 34 to determine the timing of information notification.

[0081] The communication unit 60 is a control unit that controls communication between the server 2 and other devices, and connects to a wireless LAN network using the Wi-Fi protocol.

[0082] The server 2 is configured by a computer having a processing unit (CPU) that executes a program, a storage device accessible by the processing unit, a non-volatile auxiliary storage device, and a communication interface.

[0083] The programs executed by the processor are provided to the server 2 via removable media (CD-ROM, flash memory, etc.) or a network, and are stored in a non-volatile auxiliary storage device, which is a non-transitory storage medium. For this reason, the server 2 should have an interface for reading data from removable media.

[0084] The server 2 may be a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. The server 2 may also be a hardware-independent device, or may be implemented as software within a device for other control purposes.

[0085] Although the third embodiment has been described above based on the configuration of the first embodiment, the control by the server of the third embodiment can also be applied to the robot 1 of the second embodiment.

[0086] As explained above, by having the server 2 with high computing power decide the behavior of the robot 1, rather than the robot 1 with poor computing processing power, complex control can be performed at high speed.

[0087] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0088] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0089] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0090] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0091] 1. Robot 2 Server 10 Control Unit 20, 220 Information notification judgment section 21 Person Judgment Department 22 Object determination section 23, 223 Person status determination unit 24, 224 Audio detection section 25, 225 Expected operation recording section 30, 230 Action Decision Section 31, 231 Action selection instruction section 32, 232 Urgency Judgment Department 33, 233 Timing judgment unit 34 Timer 35 Behavior Direction Department 40 Action Execution Department 41 Movement control unit 42 Main unit control section 43 Voice control unit 44 Facial Expression Control Unit 50 Sensor unit 51 Camera 52 Touch Sensor 53 Distance Sensor 54 Mike 60, 260 Communications Department 61 WiFi communication unit 62 Communications Department 70 Drive unit 71 Right wheel motor 72 Left wheel motor 73 Torso drive motor 74 Head drive motor 75 Right arm drive motor 76 Left arm drive motor 77 Right finger drive motor 78 Left finger drive motor 79 Blower fan 80 LCD drive motor 90 Expression part 91 Speaker 92 Face Display 93 Height-adjustable display

Claims

1. 1. A robotic system comprising: a control unit that controls the operation of the robot; a sensor unit that observes the surroundings of the robot; a notification unit that executes information notification from the robot to a user, The control unit determines content of information notification to the user based on a situation detected by the facility device, the notification unit notifies the user of a first information notification according to a status of the facility device; the control unit determines a mode of a second information notification to be notified to the user subsequent to the first information notification depending on whether a reaction of the user to the first information notification is a predetermined reaction; The robot system is characterized in that the notification unit issues the second information notification in accordance with the timing at which a change in at least one of the user's movement and facial expression is detected.

2. The robot system according to claim 1, The robot system is characterized in that, if the user's reaction to the first information notification is a previously expected reaction, the control unit determines the manner of the second information notification to be one that expresses positive emotions, including happiness or gratitude.

3. The robot system according to claim 1, The control unit is characterized in that, if the user's reaction to the first information notification is not a previously expected reaction, the control unit determines the manner of the second information notification to be different from that of the first information notification.

4. The robot system according to claim 1, The control unit changes the timing of the second information notification depending on the urgency of the situation detected by the facility device.

5. The robot system according to claim 1, The robot system is characterized in that the notification unit notifies the user of the second information in a manner that is more noticeable to the user than the first information notification.

6. The robot system according to claim 1, The robot system is characterized in that the notification unit notifies the user of the second information in a manner that gives the user a sensation different from that of the first information notification.

7. The robot system according to claim 1, The control unit determines whether the user's reaction is a predetermined reaction based on the surrounding conditions observed by the sensor unit.

8. A robot system according to claim 1, The robot is configured by implementing the control unit, the sensor unit, and the notification unit, A robot system characterized in that the robot operates in accordance with the behavior determined by the control unit.

9. A robot system according to claim 1, The robot is configured to output information notifications to the user, and a server determines the behavior of the robot, At least a part of the control unit is implemented in the server, Other parts of the control unit, the sensor unit, and the notification unit are implemented in the robot, the server determines an action of the robot and instructs the robot to take the determined action; A robot system characterized in that the robot operates according to instructions regarding actions determined by the server.

10. A robot control method executed by a robot system, comprising: The robot system includes a control unit that controls the operation of the robot, a sensor unit that observes the situation around the robot, and a notification unit that notifies the user of information from the robot, The robot control method includes: a step in which the control unit determines content of information notification to the user based on a situation detected by the facility device; a step in which the control unit instructs the notification unit to notify the user of a first information notification according to a status of the facility device; a step in which the control unit determines a mode of a second information notification to be notified to the user subsequent to the first information notification, depending on whether a reaction of the user to the first information notification is a predetermined reaction; and a step of instructing the notification unit to provide the second information notification in accordance with the timing at which the control unit detects a change in at least one of the user's movement and facial expression.

11. A computer that instructs a robot to perform an action, A computing device that executes a program and a storage device that stores the program, The robot has a control unit that controls the robot's operation, a sensor unit that observes the surrounding situation, and a notification unit that notifies the user of information, The computer determining the content of the information notification to the user based on the situation detected by the facility device; instructing the robot to issue a first information notification according to the status of the facility device; a computer that determines the manner of a second information notification to be notified to the user following the first information notification depending on whether the user's reaction to the first information notification is a predetermined reaction, and instructs the robot to notify the second information notification in accordance with the timing at which a change in at least one of the user's movement and facial expression is detected.

12. A computer-implemented robot control method, comprising: the computer includes an arithmetic unit that executes a program and a storage device that stores the program; The robot has a control unit that controls the robot's operation, a sensor unit that observes the surrounding situation, and a notification unit that notifies the user of information, The robot control method includes: the computing device determines the content of the information notification to the user based on the situation detected by the facility device; the computing device instructs the robot to issue a first information notification according to a status of the facility device; a computing device that determines the manner of a second information notification to be notified to the user following the first information notification based on whether the user's reaction to the first information notification is a predetermined reaction, and instructs the robot to notify the second information notification in accordance with the timing at which a change in at least one of the user's movement and facial expression is detected.

Images