Communication system, control method, and control program

Abstract

To provide a communication system capable of reducing the annoyance felt by a communication partner who communicates with an operator via a robot even if a communication failure occurs between the robot and an operation terminal.SOLUTION: A communication system (1) according to one aspect of the present invention is a communication system in which an operator remotely controlling a robot (10) using an operation terminal (20) communicates with a communication partner (C) existing around the robot (10) via the robot (10). The communication system includes a control unit (142c) that causes the robot (10) to perform a preset connecting operation when a communication failure occurs between the operation terminal (20) and the robot (10).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a communication system, a control method, and a control program. [Background technology]

[0002] 2. Description of the Related Art With recent advances in communication technology, it has become possible for people who are far away to communicate with each other in real time, and as a result, various communication systems have been developed.

[0003] For example, Patent Document 1 discloses a communication system that enables an operator remotely controlling a robot using an operation terminal to communicate with communication partners present around the robot via the robot. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-132284 Summary of the Invention [Problem to be solved by the invention]

[0005] The present applicant has found the following problem: In a general communication system, if a communication failure occurs between the robot and the operation terminal while an operator and a communication partner are communicating with each other via a robot, the communication partner who is communicating with the operator via the robot may feel annoyed.

[0006] The present disclosure has been made in consideration of such problems, and realizes a communication system, control method, and control program that can reduce the annoyance felt by the communication partner who communicates with the operator via a robot, even if a communication failure occurs between the robot and the operation terminal. [Means for solving the problem]

[0007] A communication system according to one aspect of the present disclosure is a communication system in which an operator remotely controlling a robot using an operation terminal communicates with a communication partner present around the robot via the robot, The control unit causes the robot to execute a preset connecting operation when a communication failure occurs between the operation terminal and the robot.

[0008] The above-mentioned communication system includes a first determination unit that determines a communication state between the operation terminal and the robot, When the communication state is a preset first poor communication state, it is preferable that the control unit switches from a remote operation mode in which the robot is remotely operated via the operation terminal to an autonomous control mode in which the robot performs the connecting operation.

[0009] In the above-mentioned communication system, the first determination unit determines a communication level based on a communication speed between the robot and the operation terminal, It is preferable that the control unit selects a transition action corresponding to the communication level from among a plurality of transition actions having different required times that are preset, and causes the robot to execute the selected transition action.

[0010] In the above-mentioned communication system, it is preferable that the control unit selects a transition action that corresponds to the situation in which the robot is actually being used from a plurality of transition actions that are preset in accordance with the situation in which the robot is being used, and causes the robot to execute the selected transition action.

[0011] In the above-mentioned communication system, the connecting action is preferably an action that imitates an action that causes a person to stop talking while he or she is talking.

[0012] The above mentioned communication system is a second determination unit that determines whether a person who spoke immediately before the communication state became a second communication poor state set in advance was the operator or the communication partner; a storage unit that stores, in a case where the communication partner is a person who spoke immediately before the communication state becomes the second communication poor state, a voice uttered by the communication partner toward the operator via the robot during the period from when the communication state becomes the second communication poor state until the second communication poor state is resolved; It is preferable to have

[0013] A control method according to one aspect of the present disclosure is a control method for a communication system in which an operator remotely controlling a robot using an operation terminal communicates with a communication partner present around the robot via the robot, the control method comprising: When a communication failure occurs between the robot and the operation terminal, the robot is caused to execute a preset connecting operation.

[0014] A control program according to one aspect of the present disclosure is a control program for a communication system in which an operator remotely controlling a robot using an operation terminal communicates with a communication partner present around the robot via the robot, the control program comprising: When a communication failure occurs between the robot and the operation terminal, the computer executes a process for causing the robot to execute a preset connecting operation. Effect of the Invention

[0015] According to the present disclosure, a communication system, control method, and control program are realized that can reduce the annoyance felt by a communication partner who communicates with an operator via a robot even if a communication failure occurs between the robot and an operation terminal. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining a communication system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an external configuration of the robot according to the first embodiment. [Diagram 3] FIG. 1 is a block diagram for explaining a system configuration of a robot according to a first embodiment. [Figure 4] 1 is a block diagram for explaining a system configuration of an operation terminal according to a first embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating a system configuration of a management server according to the first embodiment. [Figure 6] FIG. 4 is a flowchart for explaining the operation of the robot in the autonomous control mode in the communication system of the embodiment 1. [Figure 7] FIG. 4 is a flowchart for explaining the operation of the robot in the autonomous control mode in the communication system of the embodiment 1. [Figure 8] FIG. 11 is a block diagram for explaining a system configuration of a robot according to a second embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a hardware configuration included in the communication system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, specific embodiments to which the present disclosure is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0018] First embodiment <Communication System> 1 is a diagram for explaining a communication system according to the present embodiment. As shown in FIG. 1, the communication system 1 according to the present embodiment includes a robot 10, an operation terminal 20, and a management server 30.

[0019] The communication system 1 of this embodiment is a communication system in which a robot 10 remotely controlled by an operation terminal 20 communicates, such as by conversation, with a communication partner C (see FIG. 2) existing around the robot 10.

[0020] The robot 10 can communicate with a communication partner C, for example, as a home robot or a guide robot. The robot 10 is equipped with a display panel, a microphone, and a speaker for communication. The detailed configuration of the robot 10 will be described later.

[0021] The robot 10 and the operation terminal 20 are disposed at positions remote from each other, and an operator can remotely control the robot 10 using the operation terminal 20. The robot 10 and the operation terminal 20 are configured to be able to communicate with each other regarding image data, audio data, and the like, as will be described later.

[0022] The robot 10 and the management server 30, and the operation terminal 20 and the management server 30 are also configured to be able to communicate with each other. The management server 30 may be located away from the robot 10 and the operation terminal 20, may be located near the operation terminal 20, or may be located near the robot 10. The configuration and operation of the communication system 1 of this embodiment will be described in detail below.

[0023] <External configuration of the robot> First, the external configuration of the robot 10 will be described. Fig. 2 is a diagram showing an example of the external configuration of the robot of this embodiment. As shown in Fig. 2, the robot 10 includes a moving unit 101 and a main body 102. The moving unit 101 is a unit for moving the robot 10.

[0024] For example, wheels (not shown) are provided on the underside of the moving unit 101, and the wheels can be driven via a motor (not shown) based on an operation signal received from the operation terminal 20, thereby allowing the robot 10 to move or rotate freely on a flat surface.

[0025] The main body 102 is mounted on the upper part of the moving unit 101, and includes a trunk 111, a connection part 112, and a head 113. The trunk 111 is mounted on the upper part of the moving unit 101, and the connection part 112 connects the trunk 111 and the head 113.

[0026] The torso 111 is equipped with an arm 121 supported on the front side of the torso 111 and a hand 122 provided at the tip of the arm 121. The arm 121 and the hand 122 are driven via a motor (not shown) based on an operation signal received from the operation terminal 20, and can grasp various objects in controlled postures and perform gestures to express emotions, etc.

[0027] The head 113 is provided with a stereo camera 131, a microphone 132, a speaker 133, and a display panel 134, and is configured to communicate with the communication partner C.

[0028] The stereo camera 131 has a configuration in which two camera units 131A and 131B having the same angle of view are arranged apart from each other, and generates images captured by each of the camera units 131A and 131B as imaging data. The robot 10 transmits the imaging data to the operation terminal 20. Here, the imaging data may be a still image or a video.

[0029] The microphone 132 acquires the voice of the communication partner C and converts it into voice data. The robot 10 transmits the voice data to the operation terminal 20. The speaker 133 may select voice data pre-stored in the robot 10 and output the voice of the selected voice data, or may output a voice generated in response to an instruction received from the operation terminal 20. It can also output the voice of the voice data received from the operation terminal 20.

[0030] Display panel 134 is, for example, a liquid crystal panel, and displays a face image (a captured real image or a pre-generated processed image) to communication partner C. By displaying a real image or a processed image of a face on display panel 134, it is possible to give communication partner C the impression that display panel 134 is a pseudo face.

[0031] Furthermore, the display panel 134 can display information such as characters and pictures (e.g., icons) to the communication partner C. The above-mentioned face images, characters, pictures, and other information displayed on the display panel 134 may be stored or generated by the robot 10, or may be received from the operation terminal 20 as display data.

[0032] Here, the mobile unit 101 can change the direction of the robot 10 so that the direction that the head 113 faces can be changed. This allows the stereo camera 131 to capture an image of an object in any direction, and the microphone 132 to obtain sound from any direction. Also, the speaker 133 and the display panel 134 can each emit sound in any direction and present display content in any direction.

[0033] <Robot system configuration> Next, the system configuration of the robot 10 will be described. Fig. 3 is a block diagram for explaining the system configuration of the robot of this embodiment. As shown in Fig. 3, the robot 10 includes, as the system configuration, a stereo camera 131, a microphone 132, a speaker 133, a display panel 134, an arm driving section 140, a moving unit driving section 141, a processing section 142, a communication section 143, and a storage section 144.

[0034] The arm driving unit 140 is a means for driving the arm 121 and the hand 122, and can be configured using, for example, a motor. The arm driving unit 140 drives the arm 121 and the hand 122 based on a control signal input from the processing unit 142.

[0035] The moving unit driving section 141 is a means for driving the moving unit 101, and can be configured using, for example, a motor. The moving unit driving section 141 drives the moving unit 101 based on a control signal input from the processing section 142.

[0036] The processing unit 142 controls the processing of each part of the robot 10 based on a remote operation mode based on remote operation of the operation terminal 20, or an autonomous control mode that performs a bridging operation until a communication failure between the robot 10 and the operation terminal 20 is resolved, and is connected to each of the other components via a bus.

[0037] The processing unit 142 includes a first determination unit 142a, a second determination unit 142b, and a control unit 142c. The first determination unit 142a determines the communication state between the robot 10 and the operation terminal 20. The first determination unit 142a determines the communication level as the communication state according to, for example, the communication speed between the robot 10 and the operation terminal 20.

[0038] Here, for example, by measuring the ping value (that is, the response speed) when a ping is sent from the robot 10 to the operation terminal 20, the communication speed between the robot 10 and the operation terminal 20 can be measured.

[0039] For example, if the response speed is 0 to 40 ms, the first judgment unit 142a judges the communication condition to be at communication level 5, which is the best communication condition, if the response speed is 41 to 60 ms, the communication condition to be at communication level 4, which is good, and if the response speed is 61 to 100 ms, the communication condition to be at communication level 3, which is slightly poor.

[0040] Furthermore, the first determination unit 142a, for example, determines that the communication state is a poor communication level 2 when the response speed is 101 ms or faster, and determines that the communication state is the worst communication level 1 when there is no response. At this time, for example, when the communication level is "3" or lower, it corresponds to the first poor communication state, and when the communication level is "2" or lower, it corresponds to the second poor communication state. However, a general method can be used as a method for determining the communication state between the robot 10 and the operation terminal 20.

[0041] The second determination unit 142b determines whether the person who spoke immediately before the communication state between the robot 10 and the operation terminal 20 became the second poor communication state was the operator or the communication partner C.

[0042] For example, when the storage unit 144 stores a conversation immediately before the communication state between the robot 10 and the operation terminal 20 becomes the second poor communication state or the voice of the operator operating the operation terminal 20, the second determination unit 142b determines whether the person who spoke immediately before the communication state becomes the second poor communication state is the operator by using a voice authentication technology or the like. Then, when the person who spoke immediately before the communication state becomes the second poor communication state is not the operator, the second determination unit 142b determines that the person who spoke immediately before the communication state becomes the second poor communication state is the communication partner C.

[0043] The control unit 142c controls the stereo camera 131, the microphone 132, the speaker 133, the display panel 134, the arm driving unit 140, the mobile unit driving unit 141, the communication unit 143 and the memory unit 144 based on the judgment results of the first judgment unit 142a and the second judgment unit 142b.

[0044] For example, when the first determination unit 142a determines that the communication level is 5 or 4, the control unit 142c controls the robot 10 in a remote operation mode based on the remote operation of the operation terminal 20.

[0045] That is, in the remote operation mode, the control unit 142c transmits image data of the communication partner C acquired from the stereo camera 131 and audio data of the communication partner C acquired from the microphone 132 to the operation terminal 20 via the communication unit 143.

[0046] The control unit 142c outputs the sound of the sound data received from the operation terminal 20 via the communication unit 143 from the speaker 133. In addition, the control unit 142c causes the display panel 134 to display an image of the imaging data received from the operation terminal 20 via the communication unit 143.

[0047] The control unit 142c generates a control signal based on an operation signal received from the operation terminal 20 via the communication unit 143, and controls the arm driving unit 140 based on the generated control signal to drive the arm 121 and the hand 122. In this way, the arm 121 and the hand 122 of the robot 10 can be moved by remote control.

[0048] The control unit 142c generates a control signal based on an operation signal received from the operation terminal 20 via the communication unit 143, and controls the mobile unit driving unit 141 based on the generated control signal to drive the mobile unit 101. This allows the robot 10 to be moved to an arbitrary location based on remote control of the operation terminal 20.

[0049] For example, when the first determination unit 142a determines that the communication level is 3, the control unit 142c switches the control of the robot 10, except for the transmission and reception of voice data, to the autonomous control mode and controls the robot 10, while maintaining the transmission and reception of voice data between the operator and the communication partner C. In other words, at communication level 3, although the communication state is not good, it is at a level where voice dialogue can be continued, so that all the control of the robot 10 is not switched to the autonomous control mode.

[0050] For example, when the first determination unit 142a determines that the communication level is 1 or 2, the control unit 142c switches all control of the robot 10 to the autonomous control mode and controls the robot 10 in the autonomous control mode. In other words, at the communication level 1 or 2, the communication state is not good and it is at a level where voice dialogue is not possible, so the control unit 142c switches all control of the robot 10 to the autonomous control mode.

[0051] In such an autonomous control mode, the control unit 142c selects a transition action according to the communication level from among a plurality of transition actions having different required times that are preset, and causes the robot 10 to execute the selected transition action.

[0052] The transition action may be an action that imitates an action that causes a person to stop talking during conversation. For example, the memory unit 144 stores action data indicating the transition actions, which are an action of coughing for one second and waiting for two seconds (however, any number of seconds may be used), an action of yawning for three seconds, an action of holding one's ears and saying "ringing in the ears" for three seconds, an action of scratching one's eyes with one's hand for five seconds and saying "dirt in my eye" for five seconds, and an action of sneezing for five seconds.

[0053] At this time, the action data includes display data in which the character's face moves to imitate the above-mentioned action, sound data emitted when the character moves to imitate the action, and gesture data for the arm 121 and the like to be moved in conjunction with the character moving to imitate the action.

[0054] In the case of communication level 3, the control unit 142c acquires (selects) motion data indicating a coughing motion from the storage unit 144. In the case of communication level 2, the control unit 142c acquires motion data indicating a yawning motion or a motion of saying "my ears are ringing" while holding one's ears from the storage unit 144. In the case of communication level 1, the control unit 142c acquires motion data indicating a motion of saying "I have dirt in my eyes" while scratching one's eyes with one's hand or a sneezing motion from the storage unit 144.

[0055] Then, the control unit 142c causes the speaker 133 to output the sound of the sound data included in the acquired motion data, and causes the display panel 134 to display an image of the display data included in the motion data. The control unit 142c also causes the arm 121 and the hand 122 to perform the gesture of the gesture data included in the selected motion data.

[0056] As a result, in the case of communication level 3, the sound of the coughing voice data is output from speaker 133, and an image of the display data of the character's facial expression when coughing is displayed on display panel 134. In the case of communication level 2, the sound of the yawning voice data is output from speaker 133, and an image of the display data of the character's facial expression when yawning is displayed on display panel 134.

[0057] Alternatively, the voice of the voice data "ringing in the ears" is output from the speaker 133, and an image of the display data of the facial expression of the character when suffering from tinnitus is displayed on the display panel 134. At the same time, the arm 121 and the hand 122 execute the gesture of the gesture data as if the character is holding the ears on the face of the character displayed on the display panel 134.

[0058] In the case of communication level 1, the voice of the voice data "dust in my eye" is output from speaker 133, and an image of the display data of the facial expression of the character when dust gets in his eye is displayed on display panel 134. At the same time, arm 121 and hand 122 execute the gesture of the gesture data to cover the eye of the character's face displayed on display panel 134.

[0059] Alternatively, the sound of the sneezing sound data is output from the speaker 133, and an image of the display data of the character's facial expression when sneezing is displayed on the display panel 134. In this way, the robot 10 can be made to behave as if the character displayed on the display panel 134 is moving in imitation of the above-mentioned movements.

[0060] The "connecting action" is a phenomenon including not only a gesture as described above but also a sound accompanying the gesture. However, the connecting action is not limited to the above-mentioned action as long as it can connect the time during which the communication failure between the robot 10 and the operation terminal 20 continues.

[0061] When the second determination unit 142b determines that the speaker immediately before the communication state becomes the second poor communication state is communication partner C, the control unit 142c stores the voice data acquired from the microphone 132 in the memory unit 144 from when the communication state becomes the second poor communication state until the second poor communication state is resolved (for example, from when the communication level becomes "2" or lower to when the communication level becomes "3" or higher).

[0062] The communication unit 143 is configured to be capable of wired or wireless communication with each of the operation terminal 20 and the management server 30. The storage unit 144 stores operation data indicating a connecting operation to be executed by the robot 10, and voice data acquired from the microphone 132. At this time, the storage unit 144 may store data received from the operation terminal 20 (e.g., voice data and image data) and display data of information such as characters and pictures required for communication.

[0063] <Operation terminal configuration> Next, the operation terminal 20 will be described. As shown in Fig. 1, the operation terminal 20 is a terminal for an operator to remotely operate the robot 10, and is a computer that can be configured using, for example, a personal computer or a tablet terminal. The operation terminal 20 includes a camera 201 and a display unit 202 on the outside of the operation terminal 20.

[0064] 4 is a block diagram for explaining the system configuration of the operation terminal of this embodiment. As shown in FIG. 4, the operation terminal 20 includes a camera 201, a display unit 202, a speaker 203, a microphone 204, an input unit 205, a communication unit 206, a storage unit 207, and a control unit 208.

[0065] The control unit 208 has a function of controlling the operation terminal 20 , and the camera 201 , the display unit 202 , the speaker 203 , the microphone 204 , the input unit 205 , and the communication unit 206 are connected to the control unit 208 .

[0066] The camera 201 captures an image of the operator of the operation terminal 20 in front of the display unit 202 of the operation terminal 20 and generates the image data as imaging data. The display unit 202 displays an image of the imaging data transmitted from the robot 10. The speaker 203 outputs the sound of the sound data transmitted from the robot 10.

[0067] This allows the operator of the operation terminal 20 to check the face and voice of the communication partner C. The microphone 204 acquires the operator's voice and converts it into voice data.

[0068] The input unit 205 is a means for the operator to input information to the operation terminal 20. The operator operates the input unit 205 to input various information for remotely operating the robot 10. The control unit 208 transmits the input information input from the input unit 205 to the robot 10 from the communication unit 206 as an operation signal.

[0069] For example, the input unit 205 can be configured using a touch panel or a keyboard. For example, when the input unit 205 is configured using a touch panel, the operator can remotely control the robot 10 by pressing an icon or the like displayed on the display unit 202. Also, when the input unit 205 is configured using a keyboard, the operator can remotely control the robot 10 by inputting predetermined information via the keyboard.

[0070] Remote control performed by input from such an input unit 205 includes, for example, movement of the entire robot 10, driving of the arm 121 and the hand 122, output of sound from the speaker 133, and determination and change of display content on the display panel 134, but is not limited to these.

[0071] The communication unit 206 is configured to be capable of wired or wireless communication with each of the robot 10 and the management server 30. The storage unit 207 can store, under the control of the control unit 208, image data acquired from the camera 201, audio data acquired from the microphone 204, input information acquired from the input unit 205, and the like. The storage unit 207 may also store image data, audio data, and the like acquired from the robot 10.

[0072] The control unit 208 controls the processing of each unit of the operation terminal 20, and is connected to each of the other components via a bus. The control unit 208 can transmit imaging data acquired from the camera 201 and audio data acquired from the microphone 204 to the robot 10 via the communication unit 206. The control unit 208 can also transmit input information related to remote operation of the robot 10, which is input from the input unit 205, to the robot 10 as an operation signal via the communication unit 206.

[0073] The control unit 208 causes the speaker 203 to output the sound of the voice data received from the robot 10 via the communication unit 206, and causes the display unit 202 to display the image of the imaging data received from the robot 10 via the communication unit 206. Furthermore, the control unit 208 can store various acquired data in the memory unit 207, and can also read out or delete any stored data.

[0074] <Management Server Configuration> Next, a description will be given of the management server 30. As shown in Fig. 1, the management server 30 is a server for managing the robot 10. The management server 30 is configured to be able to communicate with the robot 10 and the operation terminal 20, respectively.

[0075] Fig. 5 is a block diagram for explaining the system configuration of the management server of this embodiment. As shown in Fig. 5, the management server 30 includes a communication unit 301 and a storage unit 302. The communication unit 301 is configured to be able to communicate with the communication unit 143 of the robot 10 and the communication unit 206 of the operation terminal 20 via wired or wireless communication. The storage unit 302 stores various information necessary for controlling the communication unit 301.

[0076] The operation terminal 20 and the management server 30 may each be configured to communicate with a plurality of robots 10 and manage the plurality of robots 10. The management server 30 may also constitute the same computer system as the operation terminal 20.

[0077] <Explanation of basic operations of the communication system in remote operation mode> Next, a basic operation of the communication system 1 of this embodiment in the remote operation mode will be described. The operation of the communication system 1 of this embodiment in the remote operation mode will be briefly described since it is substantially the same as the operation of a general communication system 1 in the remote operation mode. Note that the following description will be given on the premise that a person is already present in front of the robot 10 and that the person has been identified as a communication partner C who will communicate with the operator.

[0078] In this case, a method for identifying the communication partner C can be performed, for example, by the operator selecting the communication partner C from multiple facial portions of the image displayed on the display unit 202 of the operation terminal 20 via the input unit 205.

[0079] When the remote operation mode of the communication system 1 is started, on the operation terminal 20 side, the camera 201 always captures an image of the operator and generates the captured image data, and the microphone 204 always captures the operator's voice and converts it into voice data. Then, the control unit 208 of the operation terminal 20 transmits the captured image data and the voice data to the robot 10 via the communication unit 206.

[0080] At this time, when the operator is inputting information for remotely operating the robot 10 via the input unit 205 of the operation terminal 20, the control unit 208 transmits the input information input from the input unit 205 to the robot 10 as an operation signal via the communication unit 206.

[0081] The control unit 142c of the robot 10 causes the display panel 134 to display the image of the imaging data received via the communication unit 143, and causes the speaker 133 to output the sound of the sound data. At this time, when the control unit 142c receives an operation signal via the communication unit 143, it generates a control signal based on the operation signal, and controls the arm driving unit 140 based on the generated control signal to drive the arm 121 and the hand 122, and also controls the moving unit driving unit 141 to drive the moving unit 101.

[0082] On the robot 10 side, the stereo camera 131 always captures the communication partner C and generates the captured image data, and the microphone 132 always captures the voice of the communication partner C and converts it into voice data. Then, the control unit 142c of the robot 10 transmits the captured image data and the voice data to the operation terminal 20 via the communication unit 143.

[0083] The control unit 208 of the operation terminal 20 causes the display unit 202 to display the image of the imaging data received via the communication unit 206, and also causes the speaker 203 to output the sound of the sound data. As a result, in the communication system 1, the operator can have a conversation with the communication partner C via the display unit 202 while remotely operating the robot 10, and the communication partner can have a conversation with the operator via the display panel 134 of the robot 10.

[0084] At this time, it is preferable that the storage unit 144 of the robot 10 stores the voice data acquired from the microphone 132 and the voice data received from the operation terminal 20.

[0085] <Explanation of the robot's operation in autonomous control mode> Next, an operation of the robot 10 in the communication system 1 of this embodiment in the autonomous control mode will be described. Figures 6 and 7 are flow charts for explaining the operation of the robot in the communication system of this embodiment in the autonomous control mode.

[0086] First, the first determination unit 142a of the robot 10 measures the response speed between the robot 10 and the operation terminal 20 in the remote operation mode described above, and determines the communication level based on the measured response speed (S1). Then, the control unit 142c determines whether the communication level is "3" or less (S2). If the communication level is greater than "3" (NO in S2), the control unit 142c continues the remote operation mode.

[0087] On the other hand, when the communication level is "3" or less, the control unit 142c judges whether the communication level is "3" or not (S3). When the communication level is "3" (YES in S3), the control unit 142c switches the control of the robot 10, except for the transmission and reception of the voice data, to an autonomous control mode in which the robot 10 executes a connecting operation when the communication level is 3, while maintaining the transmission and reception of the voice data between the operator and the communication partner C, and controls the robot 10 (S4).

[0088] In detail, the control unit 142c causes the speaker 133 to output the voice of the voice data received from the operation terminal 20, and transmits the voice data of the communication partner C acquired by the microphone 132 to the operation terminal 20. The control unit 208 of the operation terminal 20 causes the speaker 203 to output the voice of the voice data received from the robot 10.

[0089] At the same time, the control unit 142c acquires motion data indicating a coughing motion from the storage unit 144, outputs the sound of the coughing sound data to the speaker 133, and displays an image of the display data of the character's facial expression when coughing on the display panel 134. When the robot 10 has thus completed the execution of a series of connecting motions, the process returns to step S1.

[0090] On the other hand, if the communication level is not "3" (NO in S3), the second determination unit 142b determines whether the person who spoke immediately before the communication level became "2" or lower is the operator (S5). If the speaker is the operator (YES in S5), the control unit 142c determines whether the communication level is "2" (S6).

[0091] Next, if the communication level is "2" (YES in S6), the control unit 142c switches the robot 10 to an autonomous control mode in which the robot 10 executes a connecting operation for the communication level 2, and controls the robot 10 (S7).

[0092] In detail, the control unit 142c acquires motion data indicating the motion of yawning from the memory unit 144, outputs the sound of the yawning sound data to the speaker 133, and displays an image of the display data of the facial expression on the character's face when yawning on the display panel 134.

[0093] Alternatively, the control unit 142c obtains from the memory unit 144 action data indicative of the action of saying "my ears are ringing" while holding the ears, outputs the voice data of "my ears are ringing" to the speaker 133, and displays on the display panel 134 an image of the display data of the facial expression on the character's face when the character is experiencing tinnitus.

[0094] Furthermore, the control unit 142c causes the arm 121 and the hand 122 to execute the gesture of the gesture data, such as pressing the ears on the face of the character displayed on the display panel 134. Here, the control unit 142c can arbitrarily select a motion of yawning or a motion of uttering "My ears are ringing" while pressing the ears. When the robot 10 has thus completed the execution of a series of connecting motions, the process returns to step S1.

[0095] On the other hand, if the communication level is not "2" (i.e., the communication failure level is "1") (NO in S6), the control unit 142c switches the robot 10 to an autonomous control mode in which the robot 10 executes a connecting operation for the communication level 1, and controls the robot 10 (S8).

[0096] In detail, the control unit 142c acquires from the storage unit 144 motion data indicative of a motion of uttering "There's something in my eye" while wiping an eye with a hand, outputs the voice data of "There's something in my eye" to the speaker 133, and displays an image of the display data of the character's facial expression when something gets in the eye on the display panel 134. At the same time, the control unit 142c causes the arm 121 and the hand 122 to execute the gesture of the gesture data to wiping the eye on the character's face displayed on the display panel 134.

[0097] Alternatively, the control unit 142c acquires action data indicating a sneezing action from the memory unit 144, outputs the sound of the sneezing sound data to the speaker 133, and displays an image of the display data of the facial expression on the character's face when sneezing on the display panel 134.

[0098] Here, the control unit 142c can arbitrarily select the action of saying "I have some dirt in my eye" while wiping the eye with the hand or the action of sneezing. When the robot 10 finishes executing the series of connecting actions in this way, the process returns to step S1.

[0099] If the speaker is not the operator, that is, the communication partner C (NO in S5), the control unit 142c judges whether the communication level is "2" or not (S9). If the communication level is "2" (YES in S9), the control unit 142c causes the robot 10 to execute the transition operation for the communication level 2 as described above, and stores the voice data acquired from the microphone 132 in the storage unit 144 after the communication level becomes "2" or lower (S10). When the robot 10 has thus completed the execution of a series of transition operations, the process returns to step S1.

[0100] On the other hand, when the communication level is "1" (NO in S9), the control unit 142c causes the robot 10 to execute the transition operation for the communication level 1 as described above, and also causes the storage unit 144 to store the voice data acquired from the microphone 132 after the communication level becomes "2" or lower (S11). When the robot 10 has thus completed the execution of a series of transition operations, the process returns to step S1.

[0101] Here, when the communication level recovers to "3" or higher, the control unit 142c may transmit the voice data stored in the storage unit 144 to the operation terminal 20. Then, the operation terminal 20 may convert the voice data while the communication level was "2" or lower into text using, for example, a transcript function, and display the text on the display unit 202.

[0102] In this way, the communication system 1 and the control method of the present embodiment cause the robot 10 to execute a connecting operation when a communication failure occurs between the robot 10 and the operation terminal 20. Therefore, even if a communication failure occurs between the robot 10 and the operation terminal 20, it is possible to prevent the robot 10 from freezing, and to reduce the annoyance felt by the communication partner C who is communicating via the robot 10.

[0103] Moreover, the communication system 1 and the control method of the present embodiment store the voice data acquired from the microphone 132 of the robot 10 in the storage unit 144 after a communication failure occurs between the robot 10 and the operation terminal 20. Therefore, after the communication failure is resolved, the voice data transmitted to the operation terminal 20 by the control unit 142c of the robot 10 can be converted to text using, for example, a transcript function, and displayed on the display unit 202 of the operation terminal 20.

[0104] This allows the operator to recognize the speech of the communication partner C while a communication failure occurs between the robot 10 and the operation terminal 20. At this time, if the stereo camera 131 of the robot 10 captures an image of the communication partner C while the communication failure occurs, the captured image data may be stored in the storage unit 144, and after the communication failure is resolved, the captured image data may be transmitted to the operation terminal 20 for display.

[0105] Although the communication system 1 of the present embodiment includes the second determination unit 142b, it may be omitted.

[0106] In addition, in this embodiment, after a communication failure occurs between the robot 10 and the operation terminal 20, the voice data stored in the memory unit 144 is converted to text and displayed on the display unit 202 of the operation terminal 20, but the voice with an adjusted playback speed (e.g., sped up) may be output from the speaker 203.

[0107] Embodiment 2 8 is a block diagram for explaining the system configuration of the robot of this embodiment. Since the robot 40 of this embodiment has substantially the same configuration as the robot 10 of embodiment 1, duplicated explanations will be omitted and the same elements will be described using the same reference numerals.

[0108] In the robot 40 of this embodiment, when the control unit 41a of the processing unit 41 knows that communication between the robot 40 and the operation terminal 20 will be interrupted for a certain period of time at a preset time for security reasons or the like, it switches from remote operation mode to autonomous control mode and causes the robot 40 to perform a connecting operation while communication is interrupted.

[0109] Therefore, the robot 40 of this embodiment can omit the first determination unit 142a and the second determination unit 142b compared to the robot 10 of the first embodiment, as shown in Fig. 8. Here, the connecting operation can be appropriately selected depending on the period during which the communication between the robot 40 and the operation terminal 20 is interrupted, etc.

[0110] Such a communication system and control method also causes the robot 40 to execute a connecting operation when a communication failure occurs between the robot 40 and the operation terminal 20. Therefore, even if a communication failure occurs between the robot 40 and the operation terminal 20, the robot 40 can be prevented from freezing, and the annoyance felt by the communication partner C who is communicating via the robot 40 can be reduced.

[0111] Other embodiments In the above-mentioned first and second embodiments, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The present disclosure can also be realized by making a CPU (Central Processing Unit) execute a computer program to process each component.

[0112] For example, the communication system according to the above-described embodiment may have the following hardware configuration: Fig. 9 is a diagram showing an example of the hardware configuration included in the communication system.

[0113] 9 includes an interface 51, a processor 52, and a memory 53. The communication systems described in the first and second embodiments are realized by the processor 52 reading and executing a program stored in the memory 53. In other words, this program is a program for causing the processor 52 to function as a communication system.

[0114] Here, a program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, a transitory computer-readable medium or a communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0115] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure.

[0116] For example, the transitional actions in the above-mentioned embodiments are merely examples, and a transitional action according to the situation and communication level in which the robot is actually used may be selected from a plurality of transitional actions preset for each communication level according to the situation such as the season in which the robot is used and the installation location, and the robot may be made to execute the selected transitional action. However, when it is known that communication between the robot 40 and the operation terminal 20 will be interrupted for a certain period of time at a preset time as in the second embodiment, a transitional action may be selected according to the situation in which the robot is used.

[0117] For example, in the above embodiment, the communication level is evaluated in five stages, but the number of evaluation levels can be changed as appropriate. Also, the correspondence between the poor communication state and the communication level can be changed as appropriate.

[0118] For example, the robot in the above embodiment is merely an example, and may be a humanoid robot, a telepresence robot, etc. In short, it is sufficient that the robot is configured to be able to execute a connecting operation when a communication failure occurs between the robot and the operation terminal. [Explanation of symbols]

[0119] 1. Communication System 10. Robot 101 Mobile Unit 102 Main body 111 Torso 112 Connection 113 Head 121 Arm 122 hands 131 Stereo camera, 131A, 131B camera unit 132 Mike 133 Speaker 134 Display Panel 140 Arm drive unit 141 Mobile unit drive section 142 processing unit, 142a first determination unit, 142b second determination unit, 142c control unit 143 Communications Department 144 Storage section 20 Operation terminal 201 Camera 202 Display section 203 Speaker 204 Mike 205 Input section 206 Communications Department 207 Memory section 208 Control Unit 30 Management Server 301 Communications Department 302 Storage section 40 Robot 41 Processing section, 41a Control section 51 Interface 52 processors 53 Memory C. Communication Partner

Claims

1. A communication system in which an operator remotely controls a robot using an operation terminal and communicates with a communication partner present around the robot via the robot, comprising: a determination unit that determines a communication state between the operation terminal and the robot; a control unit that switches, when the communication state is a predetermined poor communication state, from a remote operation mode in which the robot is remotely operated via the operation terminal to an autonomous control mode in which the robot executes a predetermined connecting operation, and causes the robot to execute the connecting operation; Equipped with The determination unit determines a communication level based on a communication speed between the robot and the operation terminal, The control unit selects a transition action corresponding to the communication level from among a plurality of transition actions having different preset required times, and causes the robot to execute the selected transition action.

2. The communication system according to claim 1 , wherein the connecting action is an action that imitates an action that causes a conversation to be interrupted while a person is talking.

3. A communication system in which an operator remotely controls a robot using an operation terminal and communicates with a communication partner present around the robot via the robot, comprising: a first determination unit that determines a communication state between the operation terminal and the robot; a control unit that switches, when the communication state is a preset first communication failure state, from a remote operation mode in which the robot is remotely operated via the operation terminal to an autonomous control mode in which the robot executes a preset connecting operation, and causes the robot to execute the connecting operation; a second determination unit that determines whether a person who spoke immediately before the communication state became a second communication poor state set in advance was the operator or the communication partner; a storage unit configured to store, in a case where the communication partner is a person who spoke immediately before the communication state becomes the second communication poor state, a voice uttered by the communication partner toward the operator via the robot during a period from when the communication state becomes the second communication poor state until the second communication poor state is resolved; A communication system comprising:

4. A control method for a communication system in which an operator remotely controlling a robot using an operation terminal communicates with a communication partner present around the robot via the robot, comprising the steps of: determining a communication state between the operation terminal and the robot; When the communication state is a predetermined poor communication state, switching from a remote operation mode in which the robot is remotely operated via the operation terminal to an autonomous control mode in which the robot executes a predetermined connecting operation; determining a communication level based on a communication speed between the robot and the operation terminal; selecting a transition action corresponding to the communication level from among a plurality of transition actions having different required times that are preset, and having the robot execute the selected transition action; A control method comprising:

5. A control program for a communication system in which an operator remotely controlling a robot using an operation terminal communicates with a communication partner present around the robot via the robot, the control program comprising: A process of determining a communication state between the operation terminal and the robot; When the communication state is a preset first communication failure state, a process of switching from a remote operation mode in which the robot is remotely operated via the operation terminal to an autonomous control mode in which the robot executes a preset transition operation, and causing the robot to execute the transition operation; A process of determining whether a person who spoke immediately before the communication state became a second communication failure state set in advance was the operator or the communication partner; a process of storing a voice uttered by the communication partner to the operator via the robot during a period from when the communication state becomes the second poor communication state until the second poor communication state is resolved, when the communication partner is a person who spoke immediately before the communication state becomes the second poor communication state; A control program that causes a computer to execute the above.

Images