Robot remote control system, robot remote control method and program

Abstract

To provide a robot remote control system, a robot remote control method and a program which can improve controllability of a robot, while reducing communication volumes (traffic) of data during remote control of the robot.SOLUTION: A robot remote control system comprises a control device that controls a robot, and a terminal device that is utilized by an operator when remote controlling the robot. The control device makes communication with the terminal device and transmits a video of a space where the robot exists to the terminal device. The terminal device displays, on a display, the video received from the control device, and transmits visual line data which are data on a visual line of the operator to the control device. Further, the control device identifies an area of interest on the video, on the basis of the visual line data, and sets a video quality of the area of interest to be higher than a video quality of an area of non-interest.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a robot remote control system, a robot remote control method, and a program. [Background technology]

[0002] In recent years, expectations for remote-controlled robots have been rising due to the effects of labor shortages, etc. In relation to this, techniques for an operator to remotely control a robot are known (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-43533 A [Patent Document 2] JP 2018-110397 A Summary of the Invention [Problem to be solved by the invention]

[0004] When an operator controls a robot from a remote location, data is sent and received via a network such as the Internet, which can limit the communication bandwidth and cause a time lag (also known as latency) in the data, resulting in a deterioration in the controllability of the robot.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a robot remote control system, a robot remote control method, and a program that can improve the maneuverability of a robot while reducing the amount of data communication (traffic) when remotely controlling a robot. [Means for solving the problem]

[0006] The robot remote control system, the robot remote control method, and the program according to the present invention employ the following configuration. (1) A first aspect of the present invention is a robot remote control system comprising: a control device for controlling a robot; and a terminal device used by an operator to remotely control the robot, wherein the control device has a first communication interface for communicating with the terminal device; and a first communication control unit for transmitting an image of a space in which the robot is present, captured by a camera, to the terminal device via the first communication interface, and the terminal device has a display visible to the operator, a second communication interface for communicating with the control device, a display control unit for displaying the image received from the control device via the second communication interface on the display, an acquisition unit for acquiring gaze data which is data of the operator's gaze, and a second communication control unit for transmitting the gaze data acquired by the acquisition unit to the control device via the second communication interface, and the control device further has an identification unit for identifying a region of interest which is an area that should be noted on the image, based on the gaze data received from the terminal device via the first communication interface, and a setting unit for setting the image quality of the region of interest to be higher than the image quality of a region of non-interest which is the remaining area of ​​the space excluding the region of interest.

[0007] (2) A second aspect of the present invention is that in the first aspect, the identification unit predicts a future direction of movement of the operator's gaze based on the gaze data, and expands the area of ​​interest in the future direction of movement.

[0008] (3) A third aspect of the present invention is the second aspect, wherein the identification unit identifies an object operated by an end effector of the robot on the image, and includes the area in which the object is present in the area of ​​interest.

[0009] (4) A fourth aspect of the present invention is a method for remotely controlling a robot that uses a control device that controls a robot and a terminal device that is used by an operator to remotely control the robot, the method including: the control device communicating with the terminal device; and transmitting an image of a space in which the robot is present, captured by a camera, to the terminal device; the terminal device communicating with the control device; displaying the image received from the control device on a display visible to the operator; acquiring gaze data that is data of the operator's gaze; and transmitting the gaze data to the control device; and the control device further including: identifying a region of interest that is an area that should be noted on the image, based on the gaze data received from the terminal device; and setting the image quality of the region of interest to be higher than the image quality of a region of non-interest that is the remaining area of ​​the space excluding the region of interest.

[0010] (5) A fifth aspect of the present invention is a program to be executed by a control device that controls a robot and a terminal device used by an operator to remotely operate the robot, the program causing the control device to communicate with the terminal device and transmit an image of the space in which the robot is present, captured by a camera, to the terminal device, causing the terminal device to communicate with the control device, display the image received from the control device on a display visible to the operator, acquire gaze data that is data on the operator's line of sight, and transmit the gaze data to the control device, and further causing the control device to identify a region of interest, which is an area that should be noted on the image, based on the gaze data received from the terminal device, and set the image quality of the region of interest to be higher than the image quality of a region of non-interest, which is the remaining area of ​​the space excluding the region of interest. Effect of the Invention

[0011] According to the above aspect, it is possible to reduce the amount of data communication (traffic) during remote control of the robot, while improving the controllability of the robot. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a scene in which the robot remote control system 1 according to the first embodiment is used. [Diagram 2] FIG. 1 is a functional configuration diagram of a robot remote control system 1 according to a first embodiment. [Diagram 3] FIG. 2 is a sequence diagram showing a flow of a series of processes in the robot remote control system 1 according to the first embodiment. [Figure 4] 4 is a flowchart showing the flow of a series of processes of the HMI 100 according to the first embodiment. [Diagram 5] 4 is a flowchart showing the flow of a series of processes of the control device 300 according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 7] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 8] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 9] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 10] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 11] FIG. 2 is a diagram illustrating an example of a region of interest. [Figure 12] 10 is a flowchart showing the flow of a series of processes of the HMI 100 according to the second embodiment. [Figure 13] 10 is a flowchart showing the flow of a series of processes of a control device 300 according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, with reference to the drawings, an embodiment of a robot remote control system, a robot remote control method, and a program according to the present invention will be described.

[0014] First Embodiment [Scenes where the robot remote control system can be used] Fig. 1 is a diagram showing an example of a usage scene of the robot remote operation system 1 according to the first embodiment. As shown in Fig. 1, the robot remote operation system 1 includes, for example, a human machine interface 100 (hereinafter referred to as HMI 100) used by an operator OP, a robot 200 remotely operated by the operator OP, and a control device 300 for controlling the robot 200. These are connected via a network NW. The network NW includes a LAN (Local Area Network) and a WAN (Wide Area Network), etc.

[0015] The HMI 100 is, for example, an interface used to remotely operate the robot 200, and includes a wearable display 110 (also called a wearable device) worn by an operator OP, a remote controller 120, and the like. The HMI 100 is an example of a "terminal device."

[0016] Wearable display 110 is a display for displaying still images and moving images (video) captured by camera 210 or the like mounted on robot 200 under remote control. For example, wearable display 110 is goggles, a headset, a head-mounted display, smart glasses, or the like that are worn on the head. Wearable display 110 may be a device that uses technology such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and projection mapping. Wearable display 110 is an example of a "display."

[0017] The remote controller 120 is a controller for moving the end effectors 230 (right end effector 230R, left end effector 230L) corresponding to the hands of the robot 200, and the arms 220 (right arm 220R, left arm 220L) connecting the torso of the robot 200 and the end effectors 230. The remote controller 120 may be a controller on which switches, buttons, joysticks, etc. are arranged as shown in the figure, or may be a glove-type controller. For example, the operator OP operates the remote controller 120 with his / her hand while checking the image displayed on the wearable display 110, thereby remotely operating the robot 200. The remote controller 120 is an example of an "input interface".

[0018] The robot 200 is typically a humanoid robot capable of grasping or manipulating an object OB by the end effector 230, but is not limited to this and may be any type of robot as long as it can grasp or manipulate an object OB. For example, the robot 200 may be a quadrupedal animal-type robot, an industrial robot, a military robot, or any other type of robot.

[0019] The end effector 230 may also be referred to as a robot hand. The end effector 230 may be provided with, for example, several fingers F (such as a thumb F1, an index finger F2, a middle finger F3, and a ring finger F4) as grippers.

[0020] The robot 200 is equipped with, for example, a camera 210 for imaging the external environment or working space as seen by the robot 200, and a control device 300 for controlling the operation of the robot 200, and performs a desired task according to the actions determined by the control device 300.

[0021] The task is, for example, gripping an object OB with the end effector 230, transferring the object OB to the other end effector 230, or moving the object OB. Note that the tasks are not limited to these, and any task can be set.

[0022] The camera 210 is installed in a part of the body (typically the head) of the robot 200. The camera 210 may be, for example, a depth camera (3D camera). For example, the camera 210 captures an image of a scene in which an object OB is grasped or operated by the end effector 230, and transmits video data of the scene to the control device 300 or transmits the video data to the HMI 100 via the control device 300.

[0023] Furthermore, the above-mentioned video data may be video data generated by an external camera (not shown) installed in the working space of the robot 200 in addition to or instead of the video data from the camera 210 installed in the robot 200. The external camera installed in the working space of the robot 200 may be used to perform image analysis such as pattern matching. The image analysis is for extracting an object OB from the video, and will be described in detail later.

[0024] The control device 300 is typically mounted on the robot 200 and controls the robot 200 based on an operation command received from the HMI 100 (i.e., an operation command input to the remote controller 120). Instead of being mounted on the robot 200, the control device 300 may be installed at a location far away from the robot 200 and control the robot 200 remotely via a network NW.

[0025] [Functional configuration of the robot remote control system] 2 is a functional configuration diagram of the robot remote operation system 1 according to the first embodiment. In addition to the above-mentioned wearable display 110 and remote controller 120, the HMI 100 may further include a stationary display 130, an eye tracking sensor 140, a storage unit 150, a communication interface 160, a processing unit 170, and the like.

[0026] The stationary display 130 displays images generated by the processing unit 170, a GUI (Graphical User Interface) for receiving various input operations from an operator OP, and the like. For example, the stationary display 130 is an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, or the like. The stationary display 130 is another example of a "display."

[0027] The eye tracking sensor 140 detects the line of sight or viewpoint of the operator OP. For example, the eye tracking sensor 140 may be built into the wearable display 110 and configured integrally with the wearable display 110. In this case, the eye tracking sensor 140 detects the line of sight, etc. of the operator OP wearing the wearable display 110 on his head. The eye tracking sensor 140 may also be installed in a part of the bezel or frame of the stationary display 130. In this case, the eye tracking sensor 140 detects the line of sight, etc. of the operator OP facing the stationary display 130.

[0028] The storage unit 150 is realized by, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. The storage unit 150 stores firmware, application programs, calculation results of the processing unit 170, etc.

[0029] The communication interface 160 communicates with the control device 300 via the network NW. The communication interface 160 includes, for example, a wireless communication module including a receiver and a transmitter, a network interface card (NIC), etc. The communication interface 160 is an example of a "second communication interface."

[0030] The processing unit 170 displays images on the wearable display 110 or the stationary display 130, and transmits operation commands for the robot 200 (end effector 230) input by the operator OP to the remote controller 120 to the control device 300 via the communication interface 160.

[0031] The processing unit 170 includes, for example, a display control unit 171 and a communication control unit 172. These components may be realized by a central processing unit (CPU), a graphics processing unit (GPU), or the like executing a program or instructions stored in the storage unit 150. The components may also be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or may be realized by a combination of software and hardware.

[0032] The display control unit 171 causes the wearable display 110 and the stationary display 130 to display moving images (video) and still images.

[0033] The communication control unit 172 transmits information or data to an external device such as the control device 300 via the communication interface 160. The communication control unit 172 is an example of a "second communication control unit."

[0034] The robot 200 further includes an actuator 240, a state sensor 250, and a processing unit 260, in addition to the above-mentioned camera 210, arm 220 (220R, 220L), and end effector 230 (230R, 230L).

[0035] The actuator 240 drives each part of the robot 200 (such as the arm 220 and each finger F of the end effector 230) under the control of the processing unit 260. The actuator 240 includes, for example, an electromagnetic motor, a gear, an artificial muscle, and the like.

[0036] The status sensor 250 is a sensor that detects the status (for example, joint angle, angular velocity, torque, etc.) of the robot 200. The status sensor 250 includes, for example, a rotary encoder that detects the degree of rotation of the joints of the robot 200 and the end effector 230, a tension sensor that detects the tension of a wire for rotating the joint, a torque sensor that detects the torque applied to the joint shaft, an acceleration sensor or a gyro sensor that detects the posture of the robot 200, etc.

[0037] The processing unit 260 controls the actuator 240 based on the control command generated by the control device 300. The processing unit 260 may be realized by a CPU, a GPU, or the like. The processing unit 260 may also be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0038] The control device 300 includes, for example, a communication interface 310, a processing unit 320, and a storage unit 330.

[0039] The communication interface 310 communicates with an external device via a network NW, and with the robot 200 via a communication line such as a bus. The external device is, for example, the HMI 100. The communication interface 310 includes, for example, a wireless communication module including a receiver and a transmitter, and a NIC. The communication interface 310 is an example of a "first communication interface."

[0040] The processing unit 320 includes, for example, an acquisition unit 321, a command generation unit 322, a specification unit 323, a setting unit 324, and a communication control unit 325.

[0041] The above components of the processing unit 320 are realized, for example, by a CPU, a GPU, or the like executing a program or instructions stored in the storage unit 330. Some or all of these components may be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0042] The storage unit 330 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, a RAM, etc. The storage unit 330 stores firmware, application programs, the results of calculations by the processing unit 320, and the like.

[0043] The acquisition unit 321 acquires operation data, eye tracking data, and video data from the HMI 100 via the communication interface 310.

[0044] The operation data is signal data indicating the operation content input to the remote controller 120, that is, data indicating the finger F instructed to be moved by the operator OP. The operation data may include, for example, identification data that can identify the finger F instructed to be moved by the operator OP among the multiple fingers F provided on the end effector 230, and the position and posture of the finger F.

[0045] The eye tracking data is data indicating the line of sight or viewpoint of the operator OP detected by the eye tracking sensor 140. The eye tracking data is an example of "line of sight data".

[0046] The video data is data showing a video of the external environment or the working space seen from the robot 200, captured by the camera 210 of the robot 200. The video data may be not only a video (moving image) but also a still image.

[0047] The command generation unit 322 generates a control command for controlling each actuator 240 of the robot 200 based on the operation data and / or eye tracking data acquired by the acquisition unit 121.

[0048] The identification unit 323 identifies an area to be noted (hereinafter, referred to as an area of ​​interest) on the image data acquired by the acquisition unit 121, that is, on the image of the external environment or the working space seen by the robot 200.

[0049] For example, the identification unit 323 may identify a region of interest on the video based on the operation data acquired by the acquisition unit 121, or may identify a region of interest on the video based on eye-tracking data acquired by the acquisition unit 121. Furthermore, the identification unit 323 may identify a region of interest on the video based on both the operation data and the eye-tracking data.

[0050] The setting unit 324 sets the image quality of the region of interest identified by the identification unit 323 to be higher than the image quality of the remaining region on the image excluding the region of interest (hereinafter referred to as a region of non-interest).

[0051] The video quality may be, for example, image quality. For example, the setting unit 324 sets the compression rate of the region of interest to be lower than that of the region of non-interest. In other words, the setting unit 324 sets the compression rate of the region of interest to be low, and sets the compression rate of the region of non-interest to be high. By setting the compression rates in this manner, video data (hereinafter referred to as compressed video data) can be generated in which the region of interest has high image quality (high resolution) and the region of non-interest has low image quality (low resolution).

[0052] The communication control unit 325 transmits the compressed video data to the HMI 100 via the communication interface 310 .

[0053] Furthermore, the communication control unit 325 transmits the control command generated by the command generation unit 322 to the robot 200 via the communication interface 310. When the processing unit 260 of the robot 200 receives the control command from the control device 300, it controls the actuator 240 based on the control command. In this way, the robot 200 executes a task such as grasping and manipulating an object OB. The communication control unit 325 is an example of a "first communication control unit."

[0054] [Overall processing flow of the robot remote control system] Hereinafter, a series of process flows of the robot remote control system 1 according to the first embodiment will be described with reference to a sequence diagram. Fig. 3 is a sequence diagram showing a series of process flows of the robot remote control system 1 according to the first embodiment.

[0055] First, the HMI 100 waits until the operator OP wearing the wearable display 110 inputs an operation for the end effector 230 to the remote controller 120 (step S100).

[0056] When an operation for the end effector 230 is input to the remote controller 120, the HMI 100 transmits operation data indicating the input operation content to the control device 300 (step S102).

[0057] When the control device 300 receives the operation data from the HMI 100, the control device 300 identifies an area of ​​interest on the image showing the workspace of the robot 200 based on the operation data (step S104).

[0058] Next, the control device 300 sets the compression ratio for each of the regions of interest and non-interest on the image (step S106).

[0059] Next, the control device 300 encodes the video data with the compression ratio set for each region (step S108). This generates compressed video data in which the region of interest is low-compressed and the region of non-interest is highly-compressed.

[0060] Next, the control device 300 transmits the compressed video data to the HMI 100 (step S110).

[0061] When the HMI 100 receives the compressed video data from the control device 300, the HMI 100 decodes the compressed video data and displays the video data on the wearable display 110 or the stationary display 130 (step S112).

[0062] In response to this, the operator OP, while visually checking the images displayed on the wearable display 110 or the stationary display 130, searches for the object OB to be grasped by the end effector 230 on the image, and repeats a series of processes using the remote controller 120 until the object OB is grasped by the end effector 230.

[0063] [HMI processing flow] Hereinafter, the process flow of each of the HMI 100 and the control device 300 included in the robot remote operation system 1 will be described with reference to flowcharts. First, the process of the HMI 100 will be described, and then the process of the control device 300 will be described. Fig. 4 is a flowchart showing the process flow of a series of processes of the HMI 100 according to the first embodiment.

[0064] First, the communication control unit 172 of the HMI 100 acquires operation data from the remote controller 120 (step S200).

[0065] Next, the communication control unit 172 transmits the operation data to the control device 300 via the communication interface 160 (step S202).

[0066] Next, the display control unit 171 waits until the communication interface 160 receives the compressed video data from the control device 300 (step S204).

[0067] Next, when communication interface 160 receives the compressed video data, display control unit 171 decodes the compressed video data (step S206).

[0068] Next, display control unit 171 causes wearable display 110 and stationary display 130 to display the decoded video data (step S208). This ends the processing of this flowchart.

[0069] In the process of this flowchart, the process of transmitting operation data to the control device 300 (step S202) and the process of receiving compressed video data from the control device 300 (step S204) have been described as being processed sequentially, but this is not limiting. For example, the process of transmitting operation data to the control device 300 (step S202) and the process of receiving compressed video data from the control device 300 (step S204) may be processed in parallel.

[0070] [Control device processing flow] FIG. 5 is a flowchart showing the flow of a series of processes of the control device 300 according to the first embodiment.

[0071] First, the command generating unit 322 of the processing unit 320 determines whether the acquiring unit 321 has acquired operation data from the HMI 100 via the communication interface 310 (step S300).

[0072] When operation data is acquired by the acquisition unit 321, the command generation unit 322 determines a target position and posture of the end effector 230 based on the operation data, and generates a control command for bringing the current position and posture of the end effector 230 closer to the target position and posture (step S302).

[0073] On the other hand, if the operation data is not acquired by the acquisition unit 321, the process of S302 is omitted.

[0074] Next, the identifying unit 323 determines whether or not the acquiring unit 321 has acquired video data from the camera 210 of the robot 200 via the communication interface 310 (step S304).

[0075] If the video data is not acquired by the acquisition unit 321, the process returns to S300. As a result, generation of a control command is repeated every time operation data is acquired during the period until the video data is acquired.

[0076] On the other hand, when the image data is acquired by the acquisition unit 321, the identification unit 323 calculates a target point on the image shown by the acquired image data, based on the operation data (step S306).

[0077] The target point is a point that the operator OP is expected to particularly want to pay attention to during remote operation. For example, the target point may be a point where a finger F that the operator OP intends to move using the remote controller 120, among multiple fingers F provided as grippers on the end effector 230, or another finger F that is predetermined to be moved together with the finger F as a gripping taxonomy, is present.

[0078] The grasping taxonomy is a classification system of patterns of fingers F used when the end effector 230 grasps an object OB, or a database of the classification system. For example, the grasping taxonomy includes medium-wrap, power-sphere, tripod, palmar-pinch, prismatic-2fingers, prismatic-3fingers, and the like.

[0079] A medium-wrap is a taxonomy in which an object OB is a horizontal cylinder, and the horizontal cylinder is grasped using all of the fingers F, including the thumb, index finger, middle finger, and ring finger.

[0080] Power-sphere is a taxonomy in which an object OB is a sphere, and the sphere is grasped with all fingers F so that the force (contact force) applied to the sphere from each finger F is directed toward the center point of the sphere.

[0081] Tripod is a taxonomy in which an object OB is a vertical cylinder that is grasped with three fingers F: the thumb, index finger, and middle finger.

[0082] A palmar pinch is a taxonomy in which an object OB is a rectangular parallelepiped, and the rectangular parallelepiped is grasped by pinching it between two fingers F, the thumb and the index finger.

[0083] Prismatic-2fingers is a taxonomy in which the object OB is a rectangular prism that is wider than that of the palmer pinch, and the ends of the rectangular prism are pinched together using three fingers F: the thumb, index finger, and middle finger.

[0084] Prismatic-3fingers is a taxonomy in which an object OB is a rectangular prism that is wider than that of prismatic-2fingers, and the ends of the rectangular prism are grasped by pinching them with four fingers F: the thumb, index finger, middle finger, and ring finger.

[0085] In this way, for each grip taxonomy, there are fingers F that are used to grip an object OB and fingers F that are not used. The identification unit 323 may include the position of a finger F that is determined in advance to be used by the grip taxonomy in the target point.

[0086] It should be noted that the grip taxonomy is not limited to the above examples, and finger F patterns may also be organized for objects OB of a certain shape, such as a triangular prism or a cone.

[0087] Furthermore, the target point may be the position where the object OB that is to be grasped by the end effector 230 is located, or it may be a position specified by the operator OP (for example, the position of the right end effector 230, the position of the left effector 230, the midpoint between the left and right end effectors 230, etc.).

[0088] For example, the identification unit 323 may extract an object OB from the video using an image analysis method such as pattern matching, and identify an area in the video that is occupied by the extracted object OB as the target point.

[0089] Furthermore, the determination unit 323 calculates a future moving direction of the end effector 230 on the image represented by the acquired image data based on the operation data (step S308).

[0090] For example, the determination unit 323 may use linear prediction or the like to calculate the future movement direction by differential calculation from the history of the positions of the end effector 230 that have been measured up to now.

[0091] Next, the identification unit 323 identifies a region of interest on the image based on either or both of the target point and the future moving direction of the end effector 230 (step S310). A method for identifying a region of interest will be described later.

[0092] Next, the setting unit 324 sets the compression rate for each of the regions of interest and non-interest on the video (step S312).

[0093] Next, the communication control unit 325 encodes the video data with the compression ratio set for each region (step S314), thereby generating compressed video data in which the region of interest is compressed at a low compression rate and the region of non-interest is compressed at a high compression rate.

[0094] Next, the communication control unit 325 transmits the compressed video data to the HMI 100 via the communication interface 310 (step S316). This ends the process of this flowchart.

[0095] 6 and 7 are diagrams showing an example of a region of interest. In the example of FIG. 6, the region in which the end effector 230 exists is the first target point R A Furthermore, an object OB (pliers or pincers in the illustrated example) to be grasped by the end effector 230 exists in the area surrounded by the dashed line. In such a case, as shown in FIG. 7, the area in which the object OB exists is located within the second target point R B The identification unit 323 identifies these first target points R A and the second target point R B is identified as the region of interest.

[0096] As a result, the area in which the end effector 230 exists (i.e., R A ) and the area where the object OB exists (i.e. R B ) is compressed at a low level and displayed at a high resolution on the HMI 100. A and R B The remaining area excluding the above is highly compressed and has low image quality (low resolution) when displayed on the HMI 100. In this way, instead of transmitting an image with high image quality for all areas to the HMI 100, an image with high image quality for only the area that is expected to be particularly important for the operator OP to pay attention to during remote operation is transmitted to the HMI 100. As a result, the maneuverability of the robot 200 can be improved while reducing the amount of data communication (traffic) during remote operation of the robot 200.

[0097] 8 and 9 are diagrams showing other examples of the region of interest. The arrow D in Fig. 8 indicates the future moving direction of the end effector 230. In such a case, the identification unit 323 may determine the first target point R A may be extended in the moving direction D. As a result, the first target point R A# is the region of interest.

[0098] 10 and 11 are diagrams showing other examples of the region of interest. As shown in the figures, it is assumed that an object OB to be grasped by the end effector 230 is present on the image, and furthermore, a future moving direction D of the end effector 230 has been calculated. In such a case, the identification unit 323 identifies a first target point R where the end effector 230 is present. A is extended in the moving direction D, and a second target point R where an object OB exists is obtained. B and the first target point R whose area is expanded in the direction of movement D. A The area combined with # is identified as the region of interest.

[0099] According to the first embodiment described above, the control device 300 acquires operation data indicating an operation command of the operator OP to the end effector 230 from the HMI 100 (an example of a "terminal device"). The control device 300 identifies a region of interest, which is a region to be noted on the image of the workspace of the robot 200, based on the operation data. The control device 300 sets a compression rate of the region of interest lower than that of the regions of non-interest, encodes video data with different compression rates according to each region, and transmits the encoded video data to the HMI 100. As a result, the amount of data communication (traffic) during remote operation of the robot 200 can be reduced, while improving the maneuverability of the robot 200.

[0100] <Second embodiment> The second embodiment will be described below. In the above-mentioned first embodiment, the region of interest is mainly specified on the video based on operation data. In contrast, the second embodiment differs from the above-mentioned first embodiment in that the region of interest is specified on the video based on eye tracking data. Hereinafter, the differences from the first embodiment will be mainly described, and the points in common with the first embodiment will be omitted. In the description of the second embodiment, the same parts as those in the first embodiment will be described with the same reference numerals.

[0101] [HMI processing flow] FIG. 12 is a flowchart showing the flow of a series of processes of the HMI 100 according to the second embodiment.

[0102] First, the communication control unit 172 of the HMI 100 acquires eye tracking data indicating the line of sight or viewpoint of the operator OP from the eye tracking sensor 140 (step S400).

[0103] Next, the communication control unit 172 transmits the eye tracking data to the control device 300 via the communication interface 160 (step S402).

[0104] Next, the display control unit 171 waits until the communication interface 160 receives the compressed video data from the control device 300 (step S404).

[0105] Next, when communication interface 160 receives the compressed video data, display control unit 171 decodes the compressed video data (step S406).

[0106] Next, display control unit 171 causes wearable display 110 and stationary display 130 to display the decoded video data (step S408). This ends the processing of this flowchart.

[0107] In the process of this flowchart, the process of transmitting eye-tracking data to the control device 300 (step S402) and the process of receiving compressed video data from the control device 300 (step S404) have been described as being processed sequentially, but this is not limited thereto. For example, the process of transmitting eye-tracking data to the control device 300 (step S402) and the process of receiving compressed video data from the control device 300 (step S404) may be processed in parallel.

[0108] [Control device processing flow] FIG. 13 is a flowchart showing the flow of a series of processes of the control device 300 according to the second embodiment.

[0109] First, the identification unit 323 of the processing unit 320 waits until the acquisition unit 321 acquires eye tracking data from the HMI 100 via the communication interface 310 and acquires video data from the camera 210 of the robot 200 (step S500).

[0110] Next, the identification unit 323 calculates a target point on the video based on the acquired eye tracking data (step S502).

[0111] The target point is a point that the operator OP is expected to particularly want to pay attention to during remote operation, as in the first embodiment described above. For example, the target point in the second embodiment may be a position where the line of sight or viewpoint of the operator OP is concentrated.

[0112] Also, as in the first embodiment, the target point may be the position where the object OB to be grasped by the end effector 230 is located, or it may be a position specified by the operator OP (for example, the position of the right end effector 230, the position of the left effector 230, the midpoint between the left and right end effectors 230, etc.).

[0113] Furthermore, the specifying unit 323 calculates the future moving direction of the line of sight or the viewpoint of the operator OP based on the eye tracking data (step S504).

[0114] For example, the determination unit 323 may use linear prediction or the like to calculate the future movement direction by differential calculation from the history of the line of sight or the viewpoint of the operator OP that has been measured so far.

[0115] Next, the specifying unit 323 specifies a region of interest on the video based on either or both of the target point and the future moving direction of the line of sight or the viewpoint of the operator OP (step S506).

[0116] Next, the setting unit 324 sets the compression rate for each of the regions of interest and non-interest on the video (step S508).

[0117] In the above-mentioned examples of FIG. 6 and FIG. 7, the first target point R A and a second target point R where the object OB is located. B The image is compressed at a low level and displayed at a high resolution on the HMI 100. A and R B The remaining area excluding the above is highly compressed and has low image quality (low resolution) when displayed on the HMI 100.

[0118] 8 and 9, the arrow D represents the future moving direction of the line of sight or the viewpoint of the operator OP. In such a case, the specifying unit 323 specifies the first target point R, which is a point where the line of sight or the viewpoint of the operator OP is concentrated. A may be extended in the moving direction D. As a result, the first target point R A # is the region of interest.

[0119] Next, the communication control unit 325 encodes the video data with the compression ratio set for each region (step S510). This generates compressed video data in which the region of interest is low-compressed and the region of non-interest is highly compressed.

[0120] Next, the communication control unit 325 transmits the compressed video data to the HMI 100 via the communication interface 310 (step S512). This ends the process of this flowchart.

[0121] According to the second embodiment described above, the control device 300 acquires eye tracking data indicating the line of sight or viewpoint of the operator OP from the HMI 100 (an example of a "terminal device"). The control device 300 identifies an area of ​​interest, which is an area to be noted on the image of the workspace of the robot 200, based on the eye tracking data. The control device 300 sets a compression rate for the area of ​​interest lower than that for areas of non-interest, encodes video data with different compression rates depending on the area, and transmits the encoded video data to the HMI 100. As a result, similar to the first embodiment, it is possible to reduce the amount of data communication (traffic) during remote operation of the robot 200, while improving the maneuverability of the robot 200.

[0122] <Other embodiments> Other embodiments will be described below. In the above-described embodiment, the compression rate of the video data generated by the camera 210 is different between the region of interest and the region of non-interest, but this is not limited to the above. For example, when the camera 210 captures an image of the workspace of the robot 200, the camera 210 may be controlled to capture the region of interest with high resolution and the region of non-interest with low resolution.

[0123] In addition, the HMI 100 and the control device 300 included in the robot remote operation system 1 have been described as being separate devices independent of each other, but this is not limited thereto. For example, some or all of these devices may be integrated into one device. For example, the processing unit 170 of the HMI 100 may include the specifying unit 323 and the setting unit 324 included in the processing unit 320 of the control device 300.

[0124] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0125] The above-described embodiment can also be expressed as follows. [Appendix 1] A control device for controlling the robot; a terminal device used by an operator when remotely operating the robot; The control device includes: A first communication interface for communicating with the terminal device; a first communication control unit that transmits an image of a space in which the robot exists, captured by a camera, to the terminal device via the first communication interface; The terminal device a display viewable by the operator; and A second communication interface for communicating with the control device; a display control unit that causes the video received from the control device via the second communication interface to be displayed on the display; an input interface through which an operation command from the operator to the robot is input; a second communication control unit that transmits operation data indicating the operation command input to the input interface to the control device via the second communication interface; The control device further comprises: an identification unit that identifies a region of interest that is a region that should be noted on the video based on the operation data received from the terminal device via the first communication interface; A setting unit that sets the image quality of the region of interest higher than the image quality of a region of non-interest, which is a remaining region of the space excluding the region of interest. Robot remote control system.

[0126] [Appendix 2] In the robot remote operation system described in Supplementary Note 1, the identification unit may identify the region of interest based on the operation data and a grasping taxonomy.

[0127] [Appendix 3] In the robot remote operation system described in Supplementary Note 1 or 2, the identification unit may predict a future movement direction of an end effector of the robot based on the operation data, and expand the region of interest in the future movement direction.

[0128] [Appendix 4] In the robot remote operation system described in Supplementary Note 3, the identification unit may identify an object operated by the end effector on the image, and include an area in which the object exists in the area of ​​interest.

[0129] [Appendix 5] A robot remote control method using a control device that controls a robot and a terminal device that is used by an operator when remotely controlling the robot, comprising: The control device, communicating with said terminal device; transmitting an image of a space in which the robot exists, captured by a camera, to the terminal device; The terminal device, communicating with the controller; displaying the image received from the control device on a display viewable by the operator; transmitting operation data indicating an operation command of the operator for the robot inputted to an input interface to the control device; The control device further comprises: Identifying a region of interest that is a region to be noted on the video based on the operation data received from the terminal device; Setting the image quality of the region of interest higher than the image quality of a non-interest region, which is a remaining region of the space excluding the region of interest. A method for remotely controlling a robot.

[0130] [Appendix 6] A program to be executed by a control device that controls a robot and a terminal device that is used by an operator when remotely operating the robot, The control device includes: communicating with said terminal device; and transmitting an image of the space in which the robot exists, captured by a camera, to the terminal device; The terminal device, communicating with the controller; displaying the image received from the control device on a display viewable by the operator; transmitting operation data, which indicates an operation command of the operator for the robot inputted to an input interface, to the control device; The control device further includes: Identifying a region of interest that is a region to be noted on the video based on the operation data received from the terminal device; setting the image quality of the region of interest higher than the image quality of a region of non-interest, which is a remaining region of the space excluding the region of interest; program. [Explanation of symbols]

[0131] 1...robot remote operation system, 100...HMI (terminal device), 110...wearable display, 120...remote controller, 130...stationary display, 140...eye tracking sensor, 150...storage unit, 160...communication interface, 170...processing unit, 171...display control unit, 172...communication control unit, 200...robot, 210...camera, 220...arm, 230...end effector, 240...actuator, 250...status sensor, 260...processing unit, 300...control device, 310...communication interface, 320...processing unit, 321...acquisition unit, 322...command generation unit, 323...identification unit, 324...setting unit, 325...communication control unit, 330...storage unit, NW...network

Claims

1. A control device for controlling the robot; a terminal device used by an operator when remotely operating the robot; The control device includes: A first communication interface for communicating with the terminal device; a first communication control unit that transmits an image of a space in which the robot exists, captured by a camera, to the terminal device via the first communication interface; The terminal device a display viewable by the operator; and A second communication interface for communicating with the control device; a display control unit that causes the video received from the control device via the second communication interface to be displayed on the display; An acquisition unit that acquires gaze data, which is data on the operator's gaze; a second communication control unit that transmits the line of sight data acquired by the acquisition unit to the control device via the second communication interface; The control device further comprises: an identification unit that identifies a region of interest that is a region that should be noted on the video based on the line-of-sight data received from the terminal device via the first communication interface; A setting unit that sets the image quality of the region of interest higher than the image quality of a region of non-interest, which is a remaining region of the space excluding the region of interest. Robot remote control system.

2. The identification unit is predicting a future direction of the operator's gaze based on the gaze data; Expanding the region of interest in the direction of future movement. The robot teleoperation system according to claim 1 .

3. The identification unit is Identifying an object to be manipulated by an end effector of the robot on the video; The region in which the object exists is included in the region of interest. The robot teleoperation system according to claim 2 .

4. A robot remote control method using a control device that controls a robot and a terminal device that is used by an operator when remotely controlling the robot, comprising: The control device, communicating with said terminal device; transmitting an image of a space in which the robot exists, captured by a camera, to the terminal device; The terminal device, communicating with the controller; displaying the image received from the control device on a display viewable by the operator; acquiring gaze data, the gaze data being data of the operator's gaze; transmitting the gaze data to the control device; The control device further comprises: Identifying a region of interest that is a region that should be noted on the video based on the line of sight data received from the terminal device; Setting the image quality of the region of interest higher than the image quality of a non-interest region, which is a remaining region of the space excluding the region of interest. A method for remotely controlling a robot.

5. A program to be executed by a control device that controls a robot and a terminal device that is used by an operator when remotely operating the robot, The control device includes: communicating with said terminal device; and transmitting an image of the space in which the robot exists, captured by a camera, to the terminal device; The terminal device, communicating with the controller; displaying the image received from the control device on a display viewable by the operator; acquiring gaze data, the gaze data being data of the operator's gaze; transmitting the gaze data to the control device; The control device further includes: Identifying a region of interest that is a region that should be noted on the video based on the line of sight data received from the terminal device; setting the image quality of the region of interest higher than the image quality of a region of non-interest, which is the remaining region of the space excluding the region of interest; program.

Images