Robot, communication system, communication method, and computer-readable medium

Abstract

The present invention provides a robot, a communication system, a communication method, and a computer-readable medium. A robot, a communication system, a communication method, and a program are provided that can effectively indicate a communication partner as a communication partner. In one embodiment, the robot comprises: a determination unit that determines a communication partner from surrounding people; an acquisition unit that acquires personal information associated with the communication partner determined by the determination unit; and a display processing unit that displays the personal information on the display unit by performing a set of protection processing to protect the personal information.

Description

Technical Field

[0001] The present invention relates to a robot, a communication system, a communication method and a program. Background Art

[0002] In recent years, technology for communication between humans and robots has advanced. For example, International Publication No. 2013 / 114493 describes a technology in which a robot equipped with a communication system illuminates a specific person, enabling the person to recognize and communicate with the robot. Summary of the Invention

[0003] The above technology may not accurately illuminate a specific person. For example, if a large number of people are present around the robot, the illuminated light may be blocked by others, or the person may be distracted and unaware of the light. In such cases, the robot cannot effectively identify itself as a communication partner.

[0004] The present invention is to solve such a problem and to provide a robot, a communication system, a communication method, and a program that can effectively indicate to a communication partner that the person is a communication partner.

[0005] A robot according to one exemplary embodiment of the present invention includes: a determination unit for determining a communication partner from among surrounding individuals; an acquisition unit for acquiring personal information associated with the communication partner determined by the determination unit; and a display processing unit for displaying the personal information on a display unit by performing a set of protection processes for protecting the personal information. This allows the robot to visually identify information related to the communication partner, effectively indicating to the communication partner that the communication partner is a communication partner. Furthermore, since it is difficult for third parties to identify individuals based on the robot's displayed data, the communication partner can communicate with the robot with confidence.

[0006] Furthermore, in the robot, the personal information may include a facial image of the communication partner, the protection processing may include processing the facial image to create a processed image, and the display processing unit may display the processed image on the display unit. This further reduces the possibility of a third party identifying the individual.

[0007] Furthermore, in the robot, the personal information may include a facial image of the communication partner, and the display processing unit may delete the facial image data from the robot as a protection process after displaying the facial image on the display unit. This allows the robot to provide the communication partner with a sense of security that their personal information is appropriately managed, allowing the communication partner to communicate with the robot with peace of mind.

[0008] Furthermore, in the robot, the acquisition unit may include an imaging unit that captures a facial image. This allows the robot to effectively indicate to the communication partner that the communication partner is a communication partner using the facial image alone.

[0009] In the robot, the identification unit may further identify location information of the communication partner, and the display processing unit may display the location information together with the personal information on the display unit. This allows the robot to more effectively indicate to the communication partner that the communication partner is a communication partner.

[0010] In the robot, the identification unit may identify the communication partner based on the distance between the robot and a person existing nearby. This allows the robot to accurately determine the communication partner who is attempting to communicate with the robot.

[0011] A robot according to another exemplary embodiment of the present invention includes: a camera unit for capturing images of people in its surroundings; a transmitter unit for transmitting image data captured by the camera unit to a remote device; a receiver unit for receiving, from the remote device, identification information identifying a communication partner among the people in its surroundings as a communication partner; and a display processor unit for displaying, on the display unit, personal information associated with the communication partner identified by the identification information. This allows the robot to visually recognize information related to the communication partner, effectively indicating to the communication partner that the person is a communication partner.

[0012] The robot may further include an extraction unit that extracts at least a portion of a person as a person region from the captured image data, and a transmission unit that transmits information indicating the person region along with the captured image data to the remote device. This allows the operator to easily select a communication partner from among the people displayed on the display of the remote device using the information indicating the person region.

[0013] Furthermore, the robot may further include an extraction unit that extracts at least a portion of a person who is a candidate for a communication partner from the camera data as a person region based on the distance between the person and the robot. The transmission unit transmits information representing the person region along with the camera data to a remote device. Thus, the operator can easily select a communication partner from among the people displayed on the display of the remote device using the information representing the person region. Furthermore, the robot preselects a person who is believed to be attempting to communicate with the robot, taking into account the distance from the person. Therefore, the operator can more accurately select a person attempting to communicate as a communication partner.

[0014] A communication system according to one exemplary embodiment of the present invention includes: a robot; a remote device for remotely operating the robot; and a camera device for capturing images of people around the robot and acquiring captured image data. The robot includes: a receiving unit for receiving, from the remote device, identification information identifying a communication partner as a communication partner among the surrounding people; an acquiring unit for acquiring personal information associated with the communication partner identified by the receiving unit; and a display processing unit for displaying the personal information via a display unit of the robot. The remote device includes: a display unit for displaying the transmitted image data to an operator; and a specific information transmitting unit for transmitting specific information related to the communication partner specified by the operator to the robot. This allows the robot to visually recognize information related to the communication partner, effectively indicating to the communication partner that they are the communication partner.

[0015] In the communication system, a camera or robot may extract at least a portion of a person as a person region from the camera data, transmit information representing the person region along with the camera data to a remote device, and display the camera data and the data representing the person region to an operator. Alternatively, the remote device may extract at least a portion of a person as a person region from the camera data, and display the camera data and the information representing the person region to an operator. Thus, the operator can easily select a communication partner from among the people displayed on the display of the remote device using the information representing the person region.

[0016] In addition, in the above-mentioned communication system, the camera device or robot can also extract at least a portion of the person who will become the candidate for the communication partner from the camera data as a person region based on the distance between the person and the robot, and send information representing the person region together with the camera data to the remote device, and the display unit of the remote device displays the camera data and the information representing the person region to the operator. Alternatively, the remote device can extract at least a portion of the person who will become the candidate for the communication partner from the camera data as a person region based on the distance between the person and the robot, and the display unit of the remote device displays the camera data and the information representing the person region to the operator. As a result, the operator can easily select the communication partner from the people displayed on the display unit of the remote device through the information representing the person region. In addition, the operator can select the person who is trying to communicate as the communication partner with higher accuracy.

[0017] In a communication method according to one exemplary embodiment of the present invention, a robot performs: an identification step of identifying a communication partner from surrounding individuals; an acquisition step of acquiring personal information associated with the identified communication partner; and a display processing step of displaying the personal information on a display unit by performing a set of protection processes for protecting the personal information. This allows the robot to visually identify information related to the communication partner, effectively indicating to the communication partner that the communication partner is a communication partner. Furthermore, since it is difficult for a third party to identify an individual based on the robot's displayed data, the communication partner can communicate with the robot with confidence.

[0018] In a communication method according to another exemplary embodiment of the present invention, a robot performs: a photographing step of photographing people in its surroundings; a transmitting step of transmitting the captured image data to a remote device; a receiving step of receiving, from the remote device, identification information identifying a communication partner from among the surrounding people; and a display processing step of displaying personal information associated with the identified communication partner on a display unit. This allows the robot to visually recognize information related to the communication partner, effectively indicating to the communication partner that the person is a communication partner.

[0019] A program according to one exemplary embodiment of the present invention causes a robot to execute: an identification step for identifying a communication partner from surrounding individuals; an acquisition step for acquiring personal information associated with the identified communication partner; and a display processing step for displaying the personal information on a display unit by performing a set of protection processes for protecting the personal information. This allows the robot to visually identify information related to the communication partner, effectively indicating to the communication partner that the communication partner is a communication partner. Furthermore, since it is difficult for third parties to identify individuals based on the robot's displayed data, the communication partner can communicate with the robot with confidence.

[0020] A program according to another exemplary embodiment of the present invention causes a robot to execute: a photographing step of photographing people in the surrounding area; a transmitting step of transmitting the captured image data to a remote device; a receiving step of receiving, from the remote device, identification information identifying a communication partner as a communication partner from the surrounding people; and a display processing step of displaying personal information associated with the identified communication partner on a display unit. This allows the robot to visually recognize information related to the communication partner, effectively indicating to the communication partner that they are a communication partner.

[0021] According to the present invention, it is possible to provide a robot, a communication system, a communication method, and a program that can effectively indicate to a communication partner that the person is a communication partner.

[0022] The above and other objects, features and advantages of the present disclosure will become more fully understood from the detailed description given below and the accompanying drawings which are given by way of illustration only and thus should not be considered as limiting the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram showing an example of a communication system according to the first embodiment.

[0024] Figure 2 This is a diagram showing an example of the external structure of the robot according to the first embodiment.

[0025] Figure 3 This is a block diagram showing an example of the robot according to the first embodiment.

[0026] Figure 4 This is a block diagram showing an example of the operation terminal according to the first embodiment.

[0027] Figure 5 This is a block diagram showing an example of the management server according to the first embodiment.

[0028] Figure 6A This is a sequence diagram showing an operation example of the communication system according to the first embodiment.

[0029] Figure 6B This is a sequence diagram showing an operation example of the communication system according to the first embodiment.

[0030] Figure 7 This is a block diagram showing an example of a robot according to the second embodiment.

[0031] Figure 8 This is a flowchart showing an example of the operation of the robot according to the second embodiment.

[0032] Figure 9 This is a block diagram showing a hardware configuration example of an information processing device according to each embodiment. DETAILED DESCRIPTION

[0033] Implementation Method 1

[0034] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0035] First, use Figures 1 to 5 , explaining the structure and operation of the communication system of this embodiment. Figure 6A 、 6B , describing the overall operation of the communication system of this embodiment.

[0036] <Communication System>

[0037] Figure 1: is a diagram for explaining a communication system according to an embodiment. Figure 1 As shown, the communication system 1 of this embodiment includes a robot 10, an operation terminal 20 (remote device), and a management server 30. The communication system 1 of this embodiment is a communication system that connects the robot 10 remotely operated by the operation terminal 20 and the target person C (refer to FIG. Figure 2 ) is a communication system for conducting conversations and other communications.

[0038] The robot 10 is installed at a venue such as a live event, and can provide guidance and promotion to the venue for the target person C. The robot 10 is equipped with a display panel, a microphone, and a speaker for communication. The detailed structure of the robot 10 will be described later.

[0039] The robot 10 and the operation terminal 20 are located at positions spaced apart from each other, and the operator can remotely operate the robot 10 using the operation terminal 20. As described later, the robot 10 and the operation terminal 20 are configured to communicate with each other regarding the robot 10's movement instructions, video data, audio data, and the like. Furthermore, the robot 10 and the management server 30, as well as the operation terminal 20 and the management server 30, are also configured to communicate with each other. Furthermore, the management server 30 can be located at a position spaced apart from the robot 10 and the operation terminal 20, near the operation terminal 20, or even near the robot 10. The structure and operation of the communication system 1 according to this embodiment will be described in detail below.

[0040] <External structure of the robot>

[0041] First, the external structure of the robot 10 is described. Figure 2 As shown, the robot 10 includes a moving member 101 and a main body 102 .

[0042] The moving unit 101 is used to move the robot 10. For example, wheels (not shown) are provided on the bottom surface of the moving unit 101. By driving these wheels, the robot 10 can be freely moved or rotated on a flat surface. The moving unit 101 is controlled by control signals from the operation terminal 20, such as the direction, speed, and stopping of the movement or rotation. By rotating the robot 10, the direction of the head 113 can be changed, that is, the person with whom the robot 10 is communicating can be changed.

[0043] The main body 102 is mounted on the upper portion of the moving member 101 and includes a trunk portion 111, a connecting portion 112, and a head portion 113. The trunk portion 111 is mounted on the upper portion of the moving member 101, and the connecting portion 112 connects the trunk portion 111 and the head portion 113.

[0044] The torso 111 includes an arm 121 supported on the front and a hand 122 at its tip. Control signals from the operation terminal 20 drive the arm 121 and hand 122 via a motor (not shown), allowing the robot to grasp various objects or perform gestures expressing emotions in controlled positions. For example, the arm 121 and hand 122 can be operated in conjunction with each other, allowing the hand 122 to grasp an item such as a manual or snack near the robot 10 and bring it to a person communicating with the robot 10, handing it to the person.

[0045] The head 113 includes a stereo camera 131 (acquisition unit, imaging unit, or imaging device), a microphone 132, a speaker 133, and a display panel 134, and has a structure for communicating with a communication partner C (hereinafter referred to as the communication partner C). The detailed structure of the head 113 is described below.

[0046] The stereo camera 131 has a structure with two camera components 131A and 131B spaced apart from each other and having the same field of view. The images captured by each camera component are generated as image data. This image data can be either still images or dynamic images. In this way, the stereo camera 131 captures people around the robot 10. Furthermore, the stereo camera 131 can not only detect the presence of people (candidates for communication) around the robot 10, but also detect the distance from the robot 10 to the people. The robot 10 transmits the captured image data and distance information to the operation terminal 20.

[0047] The microphone 132 converts sounds generated around the robot 10 or voices of people into audio data. The robot 10 transmits the audio data to the operation terminal 20 .

[0048] The speaker 133 may select voice data stored in the robot 10 and output the selected voice according to an instruction from the operation terminal 20, or may output a voice generated according to an instruction from the operation terminal 20. In addition, the voice data received from the operation terminal 20 may be converted into actual voice and output.

[0049] The display panel 134 is, for example, a liquid crystal panel, and displays a facial image (either an actual facial image captured by camera or a processed image generated from the captured facial image) of the subject C. By displaying the actual facial image or the processed facial image on the display panel 134, the subject C can be given the impression that the display panel 134 is a simulated face. Furthermore, the display panel 134 can also display textual information, pictorial information (e.g., icons), and the like to inform the subject C. The facial image, textual information, and pictorial information displayed on the display panel 134 can be stored or generated by the robot 10 or received as display data from the operation terminal 20.

[0050] The display panel 134 may also be configured by providing a touch panel on a display surface to receive instruction input from the subject C. The robot 10 may perform an operation corresponding to the received instruction input or may transmit the instruction input to the operation terminal 20 .

[0051] Furthermore, by changing the orientation of the robot 10 through the movable member 101, the direction facing the head 113 can be changed. Thus, the stereo camera 131 can capture images of objects in any direction, and the microphone 132 can acquire audio data from any direction. Furthermore, the speaker 133 and display panel 134 can each emit audio or display content in any direction.

[0052] <Robot system structure>

[0053] Next, the system configuration of the robot 10 will be described. Figure 3 This is a block diagram used to illustrate the system structure of the robot. Figure 3 As shown, in the robot 10, as a system structure, in addition to the stereo camera 131, the microphone 132, the speaker 133, and the display panel 134, it also has a control unit 140 (display processing unit and extraction unit), an arm driving unit 141, a moving part driving unit 142, a communication unit 143 (sending unit and receiving unit) and a storage unit 144.

[0054] The control unit 140 controls processing of each part of the robot 10 and is connected to other components via a bus.

[0055] The control unit 140 transmits the image data acquired from the stereo camera 131 and the audio data (e.g., image data and audio data of a person) acquired from the microphone 132 to the operation terminal 20 via the communication unit 143. Furthermore, as necessary, the control unit 140 may store the image data and audio data acquired from the stereo camera 131 and the microphone 132 in the storage unit 144. The control unit 140 can read and use or delete any data stored in the storage unit 144.

[0056] For example, the control unit 140 can display the image data captured by the stereo camera 131 on the display panel 134. The control unit 140 can also extract a portion of the facial image of the selected subject C from the image data and display this facial image on the display panel 134. Furthermore, the control unit 140 can perform processing on the facial image of the subject C as a protection process to create a processed image and display this facial image on the display panel 134. The processed image can be, for example, an image obtained by filtering the facial image of the subject C, or it can be a portrait or avatar based on the facial image of the subject C. The control unit 140 can generate this processed image using a known processing method.

[0057] Furthermore, the control unit 140 may extract at least a portion of a person reflected in the image data as a person region based on the image data obtained from the stereo camera 131. To detect the presence of a person, the control unit 140 may use known techniques such as face detection (detection of the entire face or facial components), face recognition, person detection, and tracking of people in moving images.

[0058] The control unit 140 can also calculate the distance between the person and the robot 10 (hereinafter referred to as the distance to the person) using a known measurement technique. Furthermore, the control unit 140 can extract the voice data of the person reflected in the camera data from the voice data obtained from the microphone 132 and calculate the distance to the person using the voice data.

[0059] The control unit 140 may also transmit information indicating a person's area and the calculated distance to the person, along with the video data obtained from the stereo camera 131, to the operation terminal 20 via the communication unit 143. For example, if the control unit 140 detects a person's face in the video data, it may append a symbol, such as a rectangle, surrounding the face as information indicating a person's area to the video data. Alternatively, the control unit 140 may append the detected distance to a particular person to the video data, displaying it near the person's face. The control unit 140 transmits the video data after these additional processing steps to the operation terminal 20. Furthermore, the transmitted video data may also include information indicating the presence of multiple people and information related to the distance to the people.

[0060] The control unit 140 also outputs the audio data received from the operation terminal 20 via the communication unit 143 as audio from the speaker 133. Furthermore, the control unit 140 displays the display data received from the operation terminal 20 via the communication unit 143 on the display panel 134. Furthermore, the control unit 140 may select audio data and display data stored in the storage unit 144 based on an instruction from the operation terminal 20 and output them from the speaker 133 and display panel 134, respectively. Furthermore, the control unit 140 may store the audio data and display data received from the operation terminal 20 in the storage unit 144 as needed.

[0061] Furthermore, the control unit 140 outputs a control signal to the arm driving unit 141 to drive the arm 121 and hand 122. This allows the arm of the robot 10 to be moved in any desired manner. For example, the control unit 140 can extend the arm 121 forward toward the subject C, who is being photographed by the stereo camera 131, in response to an operation signal transmitted from the operation terminal 20 (or the management server 30), thereby enabling a handshake.

[0062] Furthermore, the control unit 140 drives the movable unit 101 by outputting a control signal to the movable unit driving unit 142. This allows the robot 10 to move in any desired manner. For example, the control unit 140 can generate a control signal for controlling the movable unit 101 based on an operation signal transmitted from the operation terminal 20 (or the management server 30). Furthermore, the control unit 140 can use information obtained from the stereo camera 131 and the microphone 132 to detect obstacles around the robot 10 and generate a control signal using information related to the detected obstacles. This allows the movable unit 101 to be controlled so that the robot 10 moves while avoiding obstacles. The control unit 140 transmits the current operational status of the robot 10 related to such an operation (status related to the ongoing operation, the completed operation, and the unexecuted operation) to the operation terminal 20 via the communication unit 143.

[0063] The arm driving unit 141 is a unit for driving the arm 121 and can be configured using, for example, a motor, etc. The arm driving unit 141 drives the arm 121 and the hand 122 based on a control signal supplied from the control unit 140 .

[0064] The moving member driving unit 142 is a unit for driving the moving member 101 and can be configured using, for example, a motor, etc. The moving member driving unit 142 drives the moving member 101 based on a control signal supplied from the control unit 140 .

[0065] The communication unit 143 is configured to communicate with the operation terminal 20 and the management server 30 via wired or wireless communication. Information such as operation signals, video data, and audio data can be sent and received via the communication unit 143. For example, the communication unit 143 can send and receive video data and audio data with the operation terminal 20 via WebRTC (Web Real-Time Communication) using real-time communication via a web browser or mobile application. Furthermore, the communication unit 143 can receive operation signals from the operation terminal 20 or transmit the current operation status to the operation terminal 20 via a predetermined communication protocol (e.g., Web Socket).

[0066] The storage unit 144 stores the video data and audio data obtained from the stereo camera 131 or the microphone 132, and the data received from the operation terminal 20 (e.g., audio data, display data). In addition, the storage unit 144 may store display data such as text and pictures required for communication. The display data may be, for example, an overview of on-site activities carried out at the venue where the robot 10 is installed, or a map used for navigation. As an example of a map used for navigation, a schematic diagram of the venue (particularly a schematic diagram showing the environment around the robot 10) and a map showing access to nearby transportation facilities from the venue can be cited. The control unit 140 can read out these stored data based on the operation signal from the operation terminal 20 and display them on the display panel 134.

[0067] The above-described configuration of the robot 10 is merely an example, and in the communication system 1 of the present embodiment, the configuration of the robot 10 is not limited to the above-described configuration.

[0068] <Structure of the Operation Terminal>

[0069] Next, the operation terminal 20 will be described. Figure 1 As shown, the operation terminal 20 is a terminal for the operator to remotely operate the robot 10, and can be a computer such as a personal computer or a tablet terminal. Figure 1 As shown, the operation terminal 20 is provided with a camera 201 and a display unit 202 on its exterior.

[0070] Figure 4 This is a block diagram for explaining the system structure of the operation terminal. Figure 4As shown, the operation terminal 20 includes a camera 201, a display unit 202, a speaker 203, a microphone 204, an input unit 205, a communication unit 206 (a confirmation information transmission unit), a storage unit 207, and a control unit 208. The control unit 208 has a function of controlling the operation terminal 20, and the camera 201, display unit 202, speaker 203, microphone 204, input unit 205, and communication unit 206 are connected to the control unit 208.

[0071] The camera 201 captures an image of the operator of the operation terminal 20, who is positioned in front of the display unit 202 of the operation terminal 20. The display unit 202 displays the image data transmitted from the robot 10 (including, for example, an image of a person, information indicating the area surrounding the person, and information indicating the calculated distance to the person), as well as the current operational status of the robot 10. Furthermore, the speaker 203 converts audio data (e.g., a person's voice) received from the operation terminal 20 via the communication unit 143 into audio and outputs the audio. This allows the operator of the operation terminal 20 to recognize the person's face, voice, and other information.

[0072] The microphone 204 converts sounds generated around the operation terminal 20 or human voices (eg, voices of an operator operating the operation terminal 20 ) into audio data. This audio data may be transmitted to the communication unit 143 of the robot 10 via the communication unit 206 .

[0073] The input unit 205 is a unit for the operator to input information to the operation terminal 20. By inputting information to the input unit 205, various information for remotely operating the robot 10 is input. The control unit 208 transmits the input information from the communication unit 206 to the robot 10 as an operation signal.

[0074] For example, the input unit 205 can be configured using a touch panel or a keyboard. For example, if the input unit 205 is configured using a touch panel, the operator can remotely control the robot 10 by pressing an icon displayed on the touch panel. Alternatively, if the input unit 205 is configured using a keyboard, the operator can remotely control the robot 10 by, for example, inputting predetermined information from the keyboard.

[0075] Furthermore, when the image of a person and information indicating the person's area are displayed as captured image data on the display unit 202, the operator can select the person as the communication partner C by specifying the display area of ​​the person on the display unit 202 using the input unit 205. The operator can also specify by pressing the touch panel or inputting using the keyboard.

[0076] Remote operations performed through inputs from the input unit 205 relate to, for example, movement of the entire robot 10 , motions of the arm 121 and hand 122 , sound output from the speaker 133 , and determination and change of display content on the display panel 134 , but are not limited thereto.

[0077] The communication unit 206 is configured to communicate with the robot 10 and the management server 30 via wired or wireless communication. For example, in addition to transmitting information output from the control unit 208 to the robot 10, the communication unit 206 also outputs information transmitted from the robot 10 to the control unit 208, thereby performing remote control of the robot 10. The information transmitted and received by the communication unit 206 is as described above. For example, the communication unit 206 can transmit and receive video and audio data with the communication unit 143 via WebRTC, and can also transmit operation signals to the communication unit 143 via WebSocket, or receive the current operation status from the communication unit 143.

[0078] The storage unit 207 can store information acquired from the camera 201, the speaker 203, the microphone 204, and the input unit 205 under the control of the control unit 208. Furthermore, the storage unit 207 can also store information such as image data and audio data acquired from the robot 10 via the communication unit 206 under the control of the control unit 208.

[0079] The control unit 208 controls processing of each unit of the operation terminal 20 and is connected to other components via a bus.

[0080] The control unit 208 can transmit image data acquired from the camera 201 and audio data (e.g., image data and audio data of the operator) acquired from the microphone 204 to the robot 10 via the communication unit 206. Furthermore, the control unit 208 can transmit input information related to remote operation of the robot 10 inputted from the input unit 205 as an operation signal to the robot 10 via the communication unit 206. Furthermore, the control unit 208 can also transmit facial images, text information, or graphic information inputted or specified from the input unit 205 as display data to the robot 10 via the communication unit 206. As described above, the robot 10 can output this display data and audio data from the speaker 133 or display panel 134 as needed.

[0081] The control unit 208 also outputs the audio data received from the robot 10 via the communication unit 206 as audio from the speaker 203. Furthermore, the control unit 208 displays the image data received from the robot 10 via the communication unit 206 on the display unit 202. Furthermore, the control unit 140 displays the current operating status of the robot 10 received from the robot 10 on the display unit 202.

[0082] Furthermore, the control unit 208 can store various acquired information in the storage unit 207 , and can also read or delete any stored information.

[0083] The above-described configuration of the operation terminal 20 is an example, and in the communication system 1 of the present embodiment, the configuration of the operation terminal 20 is not limited to the above-described configuration.

[0084] <Structure of Management Server>

[0085] Next, the management server 30 will be described. Figure 1 As shown, the management server 30 is a server for managing the robot 10. The management server 30 is configured to be able to communicate with each of the robot 10 and the operation terminal 20.

[0086] Figure 5 This is a block diagram for explaining the system structure of the management server. Figure 5 As shown, the management server 30 includes a communication unit 301 , a communication control unit 302 , and a storage unit 303 .

[0087] The communication unit 301 is configured to communicate with the communication unit 143 of the robot 10 and the communication unit 206 of the operation terminal 20 via wired or wireless communication. The communication control unit 302 communicates with the robot 10 and the operation terminal 20 via the communication unit 301, thereby assisting in the communication of image data and audio data between the robot 10 and the operation terminal 20.

[0088] For example, consider a scenario where communication unit 143 and communication unit 206 transmit and receive video and audio data via WebRTC. In this case, communication control unit 302 controls the WebRTC client of communication unit 143 and the application display of communication unit 206, respectively, enabling mutual communication. For example, communication control unit 302 may activate the application of communication unit 206 when robot 10 and operation terminal 20 begin communication. Furthermore, communication unit 143 may transmit information about the current status of the application display to communication unit 301. In this case, management server 30 functions as a web server.

[0089] The storage unit 303 stores various information necessary for the control of the communication control unit 302. The communication control unit 302 reads the information from the storage unit 303 when necessary and performs control.

[0090] Furthermore, the operation terminal 20 and the management server 30 may each be configured to communicate with and manage a plurality of robots 10. Furthermore, the management server 30 may be configured as the same computer system as the operation terminal 20.

[0091] <Description of Communication System Operation>

[0092] Next, use Figure 6A 、 6B The sequence diagram shown explains the operation of the communication system according to this embodiment.

[0093] First, the operator at the operation terminal 20 operates the robot 10 by moving or rotating it in a desired direction by inputting a remote operation to the input unit 205 (step S11). The control unit 208 generates an operation signal related to the remote operation and transmits it to the robot 10 via the communication unit 206 (step S12).

[0094] The communication unit 143 of the robot 10 receives the operation signal. Based on the operation signal, the control unit 140 outputs a control signal to the moving member driving unit 142 to move or rotate the robot 10. The moving member driving unit 142 drives accordingly, and the robot 10 operates according to the operator's instructions (step S13).

[0095] When the robot 10 detects a person in the image data captured by the stereo camera 131 (step S14), it calculates the distance to each person for all detected people. This is described in detail above. The robot 10 then transmits information indicating all detected human regions and the distances to each person, along with the image data acquired from the stereo camera 131, to the operation terminal 20 via the communication unit 143 (step S15). Furthermore, the image data captured by the stereo camera 131 is temporarily stored in the storage unit 144.

[0096] The communication unit 206 of the operation terminal 20 receives the transmitted image data, and the control unit 208 displays the image data on the display unit 202 (step S16). The operator specifies one of the one or more displayed persons using the input unit 205, thereby determining the target person C to be the communication target from among the persons surrounding the robot 10 (step S17).

[0097] The communication unit 206 of the operation terminal 20 transmits identification information for identifying the selected subject C (step S18 ). The communication unit 143 of the robot 10 receives the identification information.

[0098] Based on the received identification information, the control unit 140 performs a process to establish communication with the subject C. However, if the robot 10 merely speaks to the subject C, there is a possibility that the subject C will not notice it. Therefore, the robot 10 indicates to the subject C that it is a communication partner as follows.

[0099] The control unit 140 functions as a determination unit that determines the subject C based on the determination information received in step S18, extracts the facial image of the subject C (i.e., obtains personal information associated with the subject C), and displays the facial image on the display panel 134 (step S19). The facial image may be an actual still image or a dynamic image extracted from dynamic image data captured by the stereo camera 131, obtained by tracking the facial region of the subject C. Furthermore, the control unit 140 may generate a processed image by processing the actual image rather than the actual image of the subject C, and display the processed image on the display panel 134. The details of the processed image are described above. Furthermore, the image displayed on the display panel 134 is temporarily stored in the storage unit 144.

[0100] The control unit 140 may also, in conjunction with the processing in step S19, cause the robot 10 to further move a predetermined distance in the direction of the subject C. This predetermined distance is shorter than the distance between the robot 10 and the subject C. This movement may be automatically performed by the control unit 140 or remotely performed by an operator via the operation terminal 20. Furthermore, the control unit 140 may further illuminate the display panel 134 in order to draw the subject C's attention in step S19.

[0101] The control unit 140 starts communication (step S20 ) when detecting that the subject person C has noticed the robot 10 . Here, the control unit 140 detects that the subject person C has noticed the robot 10 when, for example, it is determined that any of the following states is true.

[0102] (a) The control unit 140 uses face recognition, face detection, and other technologies to detect that the subject C is looking at the display panel 134 of the robot 10 based on the image data of the stereo camera 131.

[0103] (b) After the display panel 134 starts displaying the face image of the subject C, the control unit 140 detects that the distance or speed of the subject C approaching the robot 10 is greater than a predetermined threshold.

[0104] (c) When the display panel 134 is a touch panel, the control unit 140 detects that the subject C has touched the display panel 134.

[0105] Furthermore, the predetermined threshold value (b) is stored in the storage unit 144 and is used by the control unit 140 for determination.

[0106] Furthermore, when communication begins, control unit 140 outputs the audio data stored in storage unit 144 from speaker 133 as audio. For example, speaker 133 may output a voice message such as "What do you want?" Alternatively, control unit 140 may display display data stored in storage unit 144 on display panel 134, instead of or in addition to the audio output from speaker 133. For example, the display data may be a main screen for subject C to obtain information.

[0107] When subject C speaks, the sound is transmitted to the operation terminal 20 and heard by the operator. Based on the sound, the operator, through remote control via the input unit 205, instructs the robot 10 to perform actions for communicating with subject C. Examples of these actions include emitting a sound from the speaker 133, changing the screen on the display panel 134, or moving the arm 121 or hand 122. Alternatively, the operator can speak to the speaker 203, which will then be output from the robot 10's speaker 133. In this case, subject C and the operator communicate via the communication system 1.

[0108] Alternatively, the operator may operate the arm 121 and hand 122 of the robot 10 through remote control to hand over an object near the robot 10 to the subject C.

[0109] After acquiring the communication, the subject C ends the communication (step S21). Here, as the end of the communication, for example, the following arbitrary state is assumed.

[0110] (d) The control unit 140 detects that the distance or speed of the subject C from the robot 10 is greater than a predetermined threshold.

[0111] (e) When the subject C does not speak to the robot 10 for a period of time exceeding a predetermined time

[0112] (f) When the display panel 134 is a touch panel, the subject C does not touch the display panel 134 for a period of time exceeding a predetermined time.

[0113] When the operator of the operation terminal 20 determines that the communication between the robot 10 and the subject C has ended, the operator remotely controls the robot 10 via the input unit 205 to execute a process to end the communication (step S22). The communication unit 206 of the operation terminal 20 transmits an instruction to end the process to the robot 10 (step S23).

[0114] The communication unit 143 of the robot 10 receives the instruction to terminate the process. The control unit 140 executes the termination process (step S24), stopping the sound output from the speaker 133 and returning the screen displayed on the display panel 134 to the initial screen (the screen displayed before step S19). At this stage, the actual image or processed image of the communication partner C is not displayed on the display panel 134.

[0115] Furthermore, the control unit 140 may perform personal information protection processing in step S24 by deleting the actual image or processed image of the subject C stored in the storage unit 144 for use in step S19. Furthermore, the control unit 140 may refer to the image data temporarily stored in the storage unit 144 and obtained by previously capturing the subject C, and further delete the image data from the storage unit 144.

[0116] In addition, regarding the above Figure 6A 、 6B The following changes can also be made to the processing.

[0117] In the above steps S11 to S13, the robot 10 performs its own movement or rotation through remote control from the operation terminal 20. However, the control unit 140 can also automatically perform the movement or rotation of the robot 10 even without external instructions. For example, the control unit 140 can also control the moving part driving unit 142 once every several to ten seconds to move the robot 10 a predetermined distance or rotate a predetermined angle. The control unit 140 can perform this action in the processing before step S19. In addition, in the camera data of the stereo camera 131, if a person is within a predetermined threshold from the robot 10, or if there is a person whose distance or speed from a predetermined past time to the current approach to the robot 10 is greater than a predetermined threshold, the control unit 140 can also stop the action and make the robot 10 still.

[0118] In step S14 described above, when the robot 10 detects the subject person C in the image data captured by the stereo camera 131, the robot 10 calculates the distance to the subject person C. However, the control unit 208 on the operation terminal 20 side may calculate the distance between the robot 10 and the subject person C using the image data acquired from the stereo camera 131. Furthermore, the control unit 208 may extract the voice data of the subject person C from the voice data acquired from the microphone 132 and calculate the distance to the subject person using this voice data.

[0119] In step S17, information indicating candidates for subject C is displayed on the display unit 202 of the operation terminal 20. The operator selects one of the candidates using the input unit 205 to determine subject C. However, if multiple persons are present in the captured image data, the operation terminal 20 may select one or more specific persons as candidates for subject C and display them on the display unit 202. For example, the control unit 208 may select at least one of the multiple persons present in the captured image data that is closest or within a predetermined threshold from the robot 10 as a candidate for subject C. Alternatively, if the captured image data is a moving image, the control unit 208 may select as a candidate for subject C a person whose distance or speed from a predetermined past time (e.g., several to ten seconds ago) to the current time of approaching the robot 10 is greater than a predetermined threshold. This is because, in this case, it is assumed that the person approaching the robot 10 is interested in the robot 10 and is approaching it.

[0120] The selected person is highlighted on the display unit 202. For example, the control unit 208 can surround the entire selected person or a portion thereof (e.g., the face) with a thick frame on the display unit 202, or can cause the entire selected person or a portion thereof (e.g., the face) to illuminate or flicker.

[0121] The operator can determine the person selected by the control unit 208 as the subject person C by operating the input unit 205. Furthermore, if the control unit 208 has not determined the person selected as the subject person C, the operator can also input this information by operating the input unit 205. Upon receiving this input, the control unit 208 stops highlighting the display on the display unit 202 and, similarly to step S17, displays all persons in the image data captured by the stereo camera 131. The operator then operates the input unit 205 to select one person from among these persons as the subject person C. The predetermined threshold used for selection can be stored in the storage unit 207 or set by the operator.

[0122] Furthermore, the control unit 140 of the robot 10 may determine predetermined persons who are candidates for the subject C, based on the distance from the robot, with respect to the persons present in the camera data. For example, the control unit 140 may determine a person who satisfies at least one of the following criteria: the closest person, a person within a predetermined threshold from the robot 10, or a person whose distance or speed from a predetermined past time to the current approach to the robot 10 is greater than a predetermined threshold. For persons who meet the determination criteria, the control unit 140 may append information emphasizing the presence of the person to the camera data and transmit it to the operation terminal 20. Alternatively, the control unit 140 may generate information indicating the area of ​​the person and the distance to the person, for only those persons who meet the determination criteria, and transmit it to the operation terminal 20 along with the camera data.

[0123] In step S19, the control unit 140 may cause the display panel 134 to display the position information of the subject C along with the actual image or the processed image. The position information of the subject C can be obtained from the image data captured by the stereo camera 131. The control unit 140 displays the position information of the subject C using information such as an image or text indicating the positional relationship between the robot 10 and the subject C.

[0124] In step S19, the facial image (actual image or processed image) may be generated by the control unit 208 of the operation terminal 20 instead of the control unit 140 and transmitted to the robot 10. The communication unit 143 functions as an acquisition unit for acquiring the facial image. The control unit 140 causes the display panel 134 to display the facial image.

[0125] Alternatively, rather than performing steps S22 and S23, the control unit 140 of the robot 10 may automatically determine the states (d) to (f) and determine that communication has ended. The predetermined time values ​​for (e) and (f) are stored in the storage unit 144 and used by the control unit 140 for determination. If communication has been determined to be ended, the control unit 140 executes step S24.

[0126] Furthermore, in step S24, the control unit 140 may output a notification to the operation terminal 20 indicating that the information on the subject person C has been deleted. When the operation terminal 20 temporarily stores the actual image or processed image of the subject person C, or the image data obtained by photographing the subject person C, in the storage unit 207, the control unit 208 may receive the notification and delete at least one of the actual image or processed image of the subject person C, or the image data obtained by photographing the subject person C, from the storage unit 207.

[0127] When a processed image of the subject C is generated in step S19, the processed image may not necessarily be deleted from the storage unit 144 in step S24. This is because personal information protection is considered to have been performed in the process of generating the processed image.

[0128] Furthermore, the robot 10 may include another type of camera as an imaging unit, rather than a stereo camera. Furthermore, instead of an imaging unit provided on the robot 10, an imaging device may be provided as a separate device from the robot 10. This imaging device captures images of people around the robot 10 and transmits the captured image data to the operation terminal 20. Furthermore, the imaging device may extract at least a portion of a person as a person region from the captured image data and transmit information indicating the person region along with the captured image data to the operation terminal 20. Furthermore, the imaging device may extract at least a portion of a person who is a candidate for the subject C from the captured image data, based on the distance between the person and the robot 10, and transmit information indicating the person region along with the captured image data to the operation terminal 20. However, this extraction process may also be performed on the robot 10 side, which receives the captured image data from the imaging device. Details of this process are described above. The imaging device or robot 10 transmits the extracted captured image data to the operation terminal 20, which then performs the aforementioned process using the transmitted captured image data.

[0129] <Explanation of the effect>

[0130] Next, the effects of the communication system according to this embodiment will be described.

[0131] In the first embodiment, the robot 10 transmits image data of people in the surrounding area captured by the stereo camera 131 to the operation terminal 20. In response to an operator's operation, the operation terminal 20 transmits identification information identifying the target person C from among the surrounding people. The control unit 140 displays an actual image or a processed image of the personal information associated with the target person C identified by the identification information on the display panel 134. This allows the target person C to visually recognize information related to the target person C, effectively indicating that the target person C is a communication partner. Even if the robot 10 is installed in a location such as a live event where a large number of people are present and whose personal information has not been provided in advance, it can still communicate with the person by capturing their facial image.

[0132] Furthermore, the control unit 140 can extract at least a portion of a person as a person region from the captured image data, and the communication unit 143 can transmit information indicating the person region along with the captured image data to the operation terminal 20. Thus, the operator can easily select the target person C from among the people displayed on the display unit 202 using the information indicating the person region.

[0133] Furthermore, the control unit 140 can extract at least a portion of a person who is a candidate for the target person C from the camera data as a person region based on the distance between the surrounding people and the robot 10. The communication unit 143 can then transmit information indicating this person region to the operation terminal 20 along with the camera data. This process also facilitates the operator's selection of the target person C. Furthermore, the control unit 140 preselects a person who is believed to be attempting to communicate with the robot 10, taking into account the distance from the robot 10. Consequently, the operator can more accurately select a person attempting to communicate as the target person C.

[0134] In step S19, the control unit 140 may generate a processed image using the actual image and display the processed image on the display panel 134. Furthermore, in step S24, the control unit 140 may delete the actual image or processed image of the subject C stored in the storage unit 144 for use in step S19. In this way, the control unit 140 can display the facial image on the display panel 134 while fully performing protection processing to protect the facial image as personal information. This makes it difficult for a third party to identify the individual based on the displayed data of the robot 10, allowing the subject C to communicate with the robot 10 with peace of mind.

[0135] Furthermore, by generating a processed image and displaying it on display panel 134 by control unit 140, the possibility of a third party identifying an individual can be further reduced compared to a case where robot 10 displays the actual image as is on display panel 134. Furthermore, by deleting images related to subject C from storage unit 144 by control unit 140, subject C can be assured that their personal information is being appropriately managed, allowing subject C to communicate with robot 10 with confidence.

[0136] Furthermore, the control unit 140 can more effectively indicate to the subject person C that he is a communication partner by causing the display panel 134 to display the position information of the subject person C.

[0137] The operation terminal 20 in the communication system 1 displays the transmitted image data on the display unit 202, and the communication unit 206 transmits specific information on the subject C designated by the operator to the robot 10. In this way, the operator of the operation terminal 20 can determine the subject C of the robot 10.

[0138] Alternatively, the robot 10 or the imaging device in the communication system 1 may extract a human region from the captured image data, and the display unit 202 of the operation terminal 20 may display the captured image data and information indicating the human region to the operator. Alternatively, the operation terminal 20 may extract a human region from the captured image data, and the display unit 202 may display the captured image data and information indicating the human region to the operator. Thus, the operator can easily select the target person C from the people displayed on the display unit 202 using the information indicating the human region.

[0139] Alternatively, the robot 10 or the imaging device may extract a human region related to the subject person C from the captured image data, and the display unit 202 of the operation terminal 20 may display the captured image data and information indicating the human region to the operator. Alternatively, the operation terminal 20 may extract a human region related to the subject person C from the captured image data, and the display unit 202 may display the captured image data and information indicating the human region to the operator. This process also facilitates the operator's selection of the subject person C. Furthermore, the operator can more accurately select the person with whom the operator is attempting to communicate as the subject person C.

[0140] Implementation Method 2

[0141] While the communication system 1 including the robot 10 and the operation terminal 20 has been described in the first embodiment, the robot 10 alone can also communicate with the subject C. This will be described in the second embodiment. Hereinafter, configurations and processing details that differ from those in the first embodiment will be described, and descriptions of configurations and processing details common to the first embodiment will be omitted as appropriate.

[0142] The external structure of the robot 10 in the second embodiment is as follows: Figure 2 In addition, Figure 7 Show its internal components. Figure 7 As shown, the robot 10 does not include the communication unit 143 for transmitting and receiving information with the outside. The control unit 140 automatically controls each part of the robot 10 to execute the operation of the robot 10.

[0143] For example, the control unit 140 detects a person or determines a target person C based on the image data and distance information acquired by the stereo camera 131 and the audio data acquired by the microphone 132. This image data, audio data, etc. are appropriately stored in the storage unit 144. Furthermore, the control unit 140 reads movement and action logic from the storage unit 144 and controls the arm driver 141 and the moving member driver 142, respectively, thereby controlling the movement and action of the robot 10. For example, if a person is detected nearby, the control unit 140 may control the moving member 101 to move in a manner that avoids the person or to orient the head 113 toward the person. Furthermore, the control unit 140 may control the arm 121 and hand 122 to move in conjunction with the person's movements, thereby controlling the robot to hand an item to the person or to communicate, such as by shaking hands. For example, if the display panel 134 is a touch panel, the control unit 140 may perform such actions upon receiving an instruction input from the target person C via the touch panel.

[0144] Furthermore, the control unit 140 may output voice related to communication corresponding to the situation from the speaker 133 by reading voice data from the storage unit 144. In addition, the control unit 140 may generate the facial image, text information, or picture information displayed on the display panel 134 by itself or read it from the storage unit 144.

[0145] Based on the information indicating the human area and the calculated distance to the human, the control unit 140 determines whether there is a person C with whom the robot 10 is communicating from one or more detected people. For example, the control unit 140 may select the closest person among the multiple people present in the camera data, or a person within a predetermined threshold from the robot 10, as the person C. Alternatively, if the camera data is a moving image, the control unit 208 may select a person whose distance or speed from a predetermined past time (e.g., a few to ten seconds ago) to the current time of approaching the robot 10 is greater than a predetermined threshold as a candidate for the person C. This is because, in this case, it is assumed that the person approaching the robot 10 is interested in the robot 10 and is approaching it. If the above conditions are not met, the control unit 140 determines that there is no person C with which the robot 10 is communicating.

[0146] The above-described configuration of the robot 10 is merely an example, and in the communication system 1 of the present embodiment, the configuration of the robot 10 is not limited to the above-described configuration.

[0147] <Description of Robot's Actions>

[0148] Next, use Figure 8 The flowchart shown in FIG. 1 illustrates the operation of the robot according to this embodiment.

[0149] First, the control unit 140 of the robot 10 reads the movement logic from the storage unit 144 to make the robot 10 move autonomously (step S31 ).

[0150] When the control unit 140 detects a person in the image data captured by the stereo camera 131, it determines whether the person is the communication target person C (step S32). The control unit 140 thus determines whether the communication target person C exists. As described above, the robot 10 can determine the communication target person C based on the information indicating the human area and the calculated distance to the person.

[0151] If the subject C does not exist ("No" in step S32), the control unit 140 returns to step S31 and executes the process. On the other hand, if the subject C exists ("Yes" in step S32), the control unit 140 uses the image data obtained by photographing the subject C to extract the facial image of the subject C (i.e., obtains personal information associated with the subject C). The control unit 140 then displays the facial image on the display panel 134 (step S33). As described above, the displayed facial image can be either an actual image or a processed image.

[0152] When the control unit 140 detects that the subject C has noticed the robot 10, it starts communication (step S34). The details of this detection are described in Embodiment 1. After the communication starts, the control unit 140 performs an action or outputs a sound through the microphone 132 based on the voice of the subject C obtained from the speaker 133 or the movements of the subject C captured by the stereo camera 131.

[0153] After a predetermined time has passed since the start of communication, the control unit 140 determines whether the communication with the subject person C has ended (step S35). The details of this detection are also described in the first embodiment.

[0154] If the communication has not yet concluded ("No" in step S35), the control unit 140 continues the communication and, after a predetermined time has elapsed, performs the determination process of step S35 again. On the other hand, if the communication has concluded ("Yes" in step S35), the control unit 140 performs termination processing (step S36) by stopping the output of sound from the speaker 133 and returning the screen displayed on the display panel 134 to the initial screen. Furthermore, the control unit 140 may also perform personal information protection processing such as deleting the actual image, processed image, and captured image data of the subject C stored in the storage unit 144.

[0155] Furthermore, the robot 10 may include another type of camera as an imaging unit, rather than a stereo camera. Furthermore, instead of an imaging unit provided on the robot 10, an imaging device may be provided as a separate device from the robot 10. This imaging device captures images of people around the robot 10 and transmits the captured image data to the robot 10. Furthermore, the process of extracting at least a portion of a person as a human region from the captured image data, or the process of extracting at least a portion of a person who is a candidate for the subject C as a human region, may be performed by either the robot 10 or the imaging device.

[0156] <Explanation of the effect>

[0157] Next, the effects of the robot 10 according to this embodiment will be described.

[0158] In the second embodiment, as in the first embodiment, the control unit 140 of the robot 10 can display the facial image on the display panel 134 in a manner that performs a set of protection processing for protecting the facial image as personal information. This allows the robot 10 to effectively indicate to the target person C that they are a communication partner.

[0159] Furthermore, since the robot 10 includes the stereo camera 131 that captures facial images, the robot 10 can effectively indicate to the target person C that he or she is a communication partner using facial images alone.

[0160] Furthermore, the control unit 140 can identify the communication partner based on the distance between the person surrounding the robot 10 and the robot 10. Thus, the robot 10 can accurately determine the communication partner C attempting to communicate with the robot 10. Other effects are similar to those described in the first embodiment.

[0161] In addition, the present invention is not limited to the above-mentioned embodiments and can be appropriately changed without departing from the gist. For example, in embodiments 1 and 2, the robot 10 or the operation terminal 20 can also preferentially select a person who meets a predetermined condition from among the people in the detected camera data as a candidate for the subject C. The predetermined condition can also be, for example, a child. The robot 10 or the operation terminal 2 can determine whether a certain person is a child based on at least any one of the image of the person, the image of the person's face, the person's clothing, the person's height, etc. Alternatively, a person who is believed to need some kind of help, such as a person in a wheelchair, can also be determined in the same way. The robot 10 or the operation terminal 20 can also be controlled in a manner such that the display is emphasized on the display unit 202 as described in embodiment 1, so that the determined person is preferentially displayed on the display unit 202 as a candidate for the subject C.

[0162] In the first embodiment, the management server 30 does not necessarily need to be provided in the communication system 1. Alternatively, other terminals may be provided in the communication system 1. Examples of other terminals include tablet terminals used by workers to check the status of the robot 10 near the robot 10.

[0163] As the image of the subject C displayed on the display panel 134, an image other than the front view of the subject C (e.g., the back view) may be used. For example, when the subject C is identified, the control unit 140 refers to the past image data (e.g., moving image data) stored in the storage unit 144 to identify and extract an image other than the front view of the subject C from the past image data. The control unit 140 can thereby display an image other than the front view of the subject C. This image may also be processed as described in the embodiments.

[0164] The personal information of the subject C displayed on the display panel 134 may be modified beyond the content shown in the embodiment. For example, a computer capable of acquiring personal information of a person may be provided in the communication system 1, and the robot 10 may acquire the personal information of the person by performing processing linked to the computer.

[0165] For example, a camera for capturing facial images of live event attendees may be installed at the entrance of the live event venue where the robot 10 is installed, and a reception computer connected to the camera may be provided to associate the captured facial images of live event attendees with the attendee's reception number and present them to the live event attendees. The reception computer may then store the associated facial images of live event attendees and their reception numbers in its own storage unit.

[0166] When the operation terminal 20 has determined the subject C in step S17, the operation terminal 20 sends an inquiry for the reception number including information on the facial image of the subject C to the reception computer. In response to the inquiry, the reception computer sends the reception number corresponding to the received facial image as personal information to the operation terminal 20. The operation terminal 20 sends the reception number to the robot 10 in step S18. The robot 10 displays the reception number in step S19. It is difficult for a third party to identify the subject C through the information on the reception number, so displaying the reception number corresponds to the "protection process for protecting personal information." In this way, in addition to being able to effectively indicate that the subject C is a communication partner, the robot can also protect personal information. However, the robot 10 may also display the actual facial image or processed image of the subject C on the display panel 134 in addition to the reception number.

[0167] In the above example, the operation terminal 20 sends a query to the reception computer. However, after the robot 10 obtains the identification information of the subject C in step S18, it is also possible to send a query including the facial image of the subject C to the reception computer. In response to the query, the reception computer sends a reception number to the robot 10, and the robot 10 displays the reception number in step S19.

[0168] Alternatively, instead of the reception number, other information that allows the on-site event participant to identify themselves, such as their pseudonym or initials, may be used. The robot 10 may display such information on the display panel 134 as textual information alone, or may display it together with the actual facial image or processed image of the subject C. Furthermore, the display processing of the reception number and the like on the display panel 134 described above can also be similarly implemented in the robot 10 shown in Embodiment 2 by sending a query including facial information to the reception computer after the robot 10 has determined the subject C, and receiving personal information such as the reception number from the reception computer.

[0169] Figure 9 This is a block diagram showing an example of the hardware configuration of the information processing device that constitutes the robot 10, the operation terminal 20, and the management server 30 that execute the processing of each embodiment shown above. Figure 9 The information processing device 90 includes a signal processing circuit 91, an I / O unit (Input / Output) 92, a processor 93, and a memory 94.

[0170] The signal processing circuit 91 is an arbitrary circuit for processing and outputting a signal under the control of the processor 93. The I / O unit 92 is a hardware interface for inputting and outputting information with a remote device of the information processing apparatus 90.

[0171] The processor 93 performs the processing of the device described in the above embodiment by reading and executing software (computer programs) from the memory 94. As an example of the processor 93, one of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), or an ASIC (Application Specific Integrated Circuit) can be used, or multiple of these can be used. The processor 93 reads and executes software from the memory 94 to control the signal processing circuit 91, which performs the functions of each component in the robot 10, the operation terminal 20, and the management server 30, except for the memory.

[0172] The memory 94 is composed of a volatile memory, a non-volatile memory, or a combination thereof. The memory 94 is not limited to one, and multiple memories may be provided. In addition, the volatile memory may be, for example, a RAM (Random Access Memory) such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). The non-volatile memory may be, for example, a ROM (Random Only Memory) such as a PROM (Programmable Random Only Memory) or an EPROM (Erasable Programmable Read Only Memory), or an SSD (Solid State Drive).

[0173] In addition, the memory 94 may be provided outside the processor 93 or may be built into the processor 93. Furthermore, the memory 94 may include a storage device that is arranged away from the processor constituting the processor 93. In this case, the processor can access the memory 94 via the I / O unit 92, for example.

[0174] exist Figure 9 In the example of FIG, the memory 94 is used to store the software module group. The processor 93 can perform the processing described in the above embodiment by reading and executing these software module groups from the memory 94.

[0175] As described above, one or more processors included in each device in the above embodiments execute one or more programs including a group of instructions for causing a computer to perform the algorithms described using the drawings. This processing can implement the signal processing method described in each embodiment.

[0176] Any type of non-transitory computer-readable medium can be used to store and provide the program shown above to a computer. Non-transitory computer-readable media include any type of tangible storage medium. Examples of non-transitory computer-readable media include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (such as magneto-optical disks), CD-ROMs (Compact Disc Read-Only Memory), CD-Rs (Recordable Compact Discs), CD-R / Ws (Rewritable Compact Discs), and semiconductor memories (such as mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), Flash ROMs, RAMs (Random Access Memory), etc.). Any type of temporary computer-readable medium can be used to provide the program to a computer. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can provide the program to a computer via a wired communication line (such as electrical wires and optical fibers) or a wireless communication line.

[0177] From the disclosure thus described, it will be apparent that the embodiments of the present disclosure may be modified in various ways. Such variations are not to be regarded as a departure from the spirit and scope of the present disclosure, and all such modifications obvious to those skilled in the art are intended to be included within the scope of the claims.

Claims

1. A robot that communicates with people in its surroundings, the robot comprising: a determination unit that determines a communication partner who is a communication partner from among people existing around the robot; an acquiring unit that acquires a facial image of the communication partner identified by the identifying unit; a processing unit that processes the facial image of the communication partner to obtain a processed image of the facial image; as well as The display unit displays a processed image of the facial image on the communication partner.

2. The robot according to claim 1, wherein: After displaying the processed image of the facial image on the display unit, the processing unit executes a process of deleting the data of the facial image from the robot.

3. The robot according to claim 1 or 2, wherein: The acquisition unit includes an imaging unit that captures the facial image.

4. The robot according to claim 1 or 2, wherein: The determining unit further determines the location information of the communication partner. The processing unit causes the display unit to display the position information together with the processed image of the face image.

5. The robot according to claim 1 or 2, wherein: The identification unit identifies the communication partner based on the distance between the surrounding people and the robot.

6. A robot that communicates with people in its surroundings, the robot comprising: a camera unit for capturing images of people around the robot; a sending unit, configured to send the image data obtained by the image pickup unit to a remote device; a receiving unit receiving, from the remote device, identification information for identifying a communication partner who is a communication partner from among the people present in the surroundings; a processing unit configured to process the facial image of the communication partner identified by the identification information to obtain a processed image of the facial image; as well as The display unit displays a processed image of the facial image on the communication partner.

7. The robot according to claim 6, wherein: The apparatus further comprises an extraction unit that extracts at least a part of a person as a person region from the image data. The transmitting unit transmits information indicating the human region together with the captured image data to the remote device.

8. The robot according to claim 6, wherein: The robot further comprises an extraction unit configured to extract at least a portion of a person who is a candidate for the communication partner from the camera data as a person region based on the distance between the person existing in the surrounding area and the robot. The transmitting unit transmits information indicating the human region together with the captured image data to the remote device.

9. A communication system comprising: The robot communicates with the characters around it; a remote device for remotely operating the robot; and A camera device captures images of people around the robot to obtain image data. The robot has: a receiving unit that receives, from the remote device, identification information for identifying a communication partner who is a communication partner from among people existing around the robot; an acquiring unit configured to acquire a facial image of the communication partner determined by the receiving unit; a processing unit that processes the facial image of the communication partner to obtain a processed image of the facial image; as well as A display unit displays a processed image of the face image on the communication partner, The remote device has: a second display unit for displaying the transmitted imaging data to an operator; and The specific information transmitting unit transmits the specific information related to the communication partner specified by the operator to the robot.

10. The communication system according to claim 9, wherein: The imaging device or the robot extracts at least a portion of a person as a person region from the imaging data, and transmits information indicating the person region together with the imaging data to the remote device, and the second display unit of the remote device displays the imaging data and the data indicating the person region to the operator, or The remote device extracts at least a portion of a person as a person region from the captured image data, and the second display unit of the remote device displays the captured image data and information indicating the person region to the operator.

11. The communication system according to claim 9, wherein: The camera device or the robot extracts at least a portion of a person who is a candidate for the communication partner from the camera data as a person region based on the distance between the person existing in the surrounding area and the robot, and transmits information indicating the person region together with the camera data to the remote device, and the second display unit of the remote device displays the camera data and the information indicating the person region to the operator, or The remote device extracts at least a portion of a person who will be a candidate for the communication partner as a person area from the camera data based on the distance between the people existing in the surrounding area and the robot, and the second display unit of the remote device displays the camera data and information representing the person area to the operator.

12. A communication method performed by a robot that communicates with a person present nearby, the robot performing: a determination step of determining a communication partner who is a communication partner from among persons existing around the robot; an acquisition step of acquiring a facial image of the determined communication partner; a processing step of processing the facial image of the communication partner to obtain a processed image of the facial image; and A display step of displaying the processed image of the face image on a display unit of the robot.

13. A communication method performed by a robot communicating with a person present in the surrounding area, the robot performing: a photographing step of photographing people around the robot; a sending step of sending the video data obtained by the video recording to a remote device; a receiving step of receiving, from the remote device, identification information for identifying a communication partner who is a communication partner from among people existing around the robot; a processing step of processing the face image of the communication partner identified by the identification information to obtain a processed image of the face image; as well as A display step of displaying the processed image of the face image on a display unit of the robot.

14. A computer-readable medium storing a program for causing a robot communicating with a person existing in the surrounding to: a determination step of determining a communication partner who is a communication partner from among persons existing around the robot; an acquisition step of acquiring a facial image of the determined communication partner; a processing step of processing the facial image of the communication partner to obtain a processed image of the facial image; and A display step of displaying the processed image of the face image on a display unit of the robot.

15. A computer-readable medium storing a program for causing a robot communicating with a person existing in the surrounding to: a photographing step of photographing people around the robot; a sending step of sending the video data obtained by the video recording to a remote device; a receiving step of receiving, from the remote device, identification information for identifying a communication partner who is a communication partner from among people existing around the robot; a processing step of processing the face image of the communication partner identified by the identification information to obtain a processed image of the face image; as well as A display step of displaying the processed image of the face image on a display unit of the robot.

Images