robot

Abstract

To provide a robot which allows both maintenance of operability and reduction of erroneous operation due to a network time difference.SOLUTION: The present invention is directed to a robot characterized by having: moving image acquiring means which can capture surrounding environment as a moving image; a reception unit for receiving operation instructions; a movement destination estimation unit for estimating a movement destination of the robot based on the operation instructions received by the reception unit; and an autonomous control unit for voluntarily correcting movement to the movement destination due to the operation instructions based on the information of the environment obtained from the moving image at the reception of the operation instructions.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to robots To .

Background Art

[0002] Robots called telepresence robots, which are operated remotely via a network, are already known. In such robots, since they communicate via a network, it is inevitable that a delay occurs in the transmission and reception of data. Further, when an operator performs remote operation, there is concern that the real-time moving image captured by the robot and the moving image that the operator views as a clue for a movement instruction may deviate due to the delay, causing an erroneous operation. Also known are robots that are autonomously judged by the robot and then automatically operated. However, in such autonomous robots, there is little room to accept operations from the operator, and there arises a problem that the characteristics of the telepresence robot, namely, "the robot operates while the operator operates according to his / her own will to the place where the operator wants to go at a remote location," are lost.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The present invention has been made in view of the above problems, and an object thereof is to provide a robot capable of reducing erroneous operations due to network time difference while maintaining operability.

Means for Solving the Problems

[0004] In order to solve the above-described problems, the robot of the present invention includes a receiving unit for receiving an operation instruction and an autonomous control unit that automatically corrects the movement to the movement destination according to the operation instruction based on the moving image captured when the operation instruction is received among the moving images captured of the surroundings of the robot . , The time when the video image transmitted to the operator was captured, and the time when the operation instruction was received. The autonomous control unit has a delay measuring means for measuring the time difference, and the autonomous control unit responds to the amount of the time difference Then the amount of the correction is changed. It is characterized by the following: [Effects of the Invention]

[0005] According to the present invention, it is possible to maintain operability while simultaneously reducing errors caused by network time differences. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows an example of the overall configuration of a control system for a telepresence robot according to an embodiment of the present invention. [Figure 2] This block diagram shows an example of the configuration of the movement instruction unit shown in Figure 1. [Figure 3] Figure 1 is a block diagram showing an example of the configuration of the control system. [Figure 4] This figure shows an example of the delay relationship between network latency and the video footage acquired by the robot and the operation instructions. [Figure 5] This figure shows an example of what happens when autonomous control in this embodiment is added to the delay relationship shown in Figure 4. [Figure 6] This figure shows a first example of the operation of the autonomous control unit in the present invention. [Figure 7] This figure shows an example of the control operation of the control system in the present invention. [Figure 8] This figure shows a second example of the operation of the autonomous control unit in the present invention. [Figure 9] This figure shows a third example of the operation of the autonomous control unit in the present invention. [Figure 10] This figure shows a fourth example of the operation of the autonomous control unit in the present invention. [Modes for carrying out the invention]

[0007] As an example of an embodiment of the present invention, Figure 1 shows a conceptual diagram of the overall configuration of a control system 100 for a telepresence robot TR, which is a remotely operated robot operated by operator P using an operating unit 10 via a network 9.

[0008] The telepresence robot TR includes a camera 20 which is a means for acquiring moving images, a mobile unit 21 which is configured using wheels or an endless belt or the like to make it movable, a receiving unit 22 for receiving operation instructions from the operation unit 10, and a control unit 30 for controlling each part of the telepresence robot TR based on the operation instructions received by the receiving unit 22.

[0009] The operation unit 10 includes a video display unit 11 that displays images or videos for the operator P to refer to when operating the device, and a movement instruction unit 12 equipped with multiple buttons that indicate the direction of movement, as shown in Figure 2. The control unit 10 communicates with the telepresence robot TR via a wireless or wired network 9 to send and receive data. Specifically, the video captured by the camera 20 of the telepresence robot TR is transmitted to the video display unit 11, and the operation instructions given by the operator P using the movement instruction unit 12 while viewing the video display unit 11 are transmitted to the receiver unit 22 of the telepresence robot TR. The movement instruction unit 12 functions as a movement direction instruction unit, for example, by having four buttons: a forward instruction button 12a, a right turn instruction button 12b, a reverse instruction button 12c, and a left turn instruction button 12d. In this embodiment, the receiving unit 22 is described separately from the control unit 30, but the receiving unit 22 may also be provided as a function within the control unit 30, and the configuration is not limited to this.

[0010] As shown in Figure 3, the control unit 30 includes a destination estimation unit 31 that estimates the destination of the telepresence robot TR based on the operation instructions received by the receiving unit 22, and an autonomous control unit 32 that spontaneously corrects the movement to the destination based on the operation instructions, based on information about the surrounding environment. The control unit 30 also has a delay measurement unit 33 for measuring a delay time td, which is the time from when a moving image is acquired from the camera 20 until an operation instruction of the operator P based on the moving image is received by the receiving unit 22, as will be described later. Based on the operation instruction received from the receiving unit 22, the moving destination estimation unit 31 determines the direction of movement in any of the front, rear, left, or right directions, and estimates a provisional moving destination Q, as will be described later. Based on the moving destination Q estimated by the moving destination estimation unit 31 and the information on the surrounding environment obtained from the camera 20 for the movement to the moving destination Q, the autonomous control unit 32 controls the movement of the moving unit 21. For example, when transmitting a moving image frame F1, which will be described later, to the operation unit 10, the delay measurement unit 33 transmits a moving image with a time stamp, and measures the time difference from the time when the operation unit 10 receives the moving image with the time stamp, thereby measuring the delay time. Note that such measurement of the delay time is merely an example, and for example, data that reciprocates through the network 9 may be transmitted and received, and the measurement may be performed from the time required for the transmission and reception, or any means capable of measuring the delay on other known networks may be used.

[0011] Now, when using such a control system 100, due to the time required for the transmission and reception of the network 9, as shown in FIG. 4, there arises a problem that the frame of the moving image acquired by the camera 20 and the frame of the moving image that is delayed by the transmission and reception and displayed on the moving image display unit 11 are delayed by the moving image transmission and reception time t1. Furthermore, the time until the operation instruction of the operator P reaches the receiving unit 22 of the telepresence robot TR is delayed by the total delay time td, which is the sum of the moving image transmission and reception time t1, the operation determination time t2, which is the time from when the operator P views the moving image until actually giving an operation instruction using the movement instruction unit 12, and the operation instruction transmission and reception time t3, which is the time for the operation instruction to reach the receiving unit 22 from the operation unit 10.

[0012] That is, when the operator P attempts to remotely operate the telepresence robot TR, as shown in FIG. 4, a delay of at least the delay time td will necessarily occur between the frame of the moving image captured by the camera 20 and the time when the operation instruction is actually received and movement starts. When using a general current network 9, it is known that a delay of approximately td = 0.3 seconds to 0.4 seconds occurs. That is, when the operator P views the frame F1 displayed on the moving image display unit 11 and transmits an operation instruction, and the telepresence robot TR actually operates, the situation often differs from the frame F13 acquired by the camera 20 representing the actual surrounding environment. There may be cases where an operation instruction unfavorable to the surrounding environment of the telepresence robot TR in the frame F13 acquired by the camera 20 is made.

[0013] Such a delay time td cannot be made zero when remotely operating the telepresence robot TR, and when operating the telepresence robot TR, it can cause unintended accidents or malfunctions, such as passing by the path that should be advanced or colliding when people or objects suddenly appear as obstacles on the path. <00OO100> Also, to solve such problems, a method of determining the path by the robot itself without operation can be considered. However, in the case of movement by simply programmed autonomous control, it cannot be moved as the operator P desires. For example, there is a problem that it is difficult to correctly perform operations such as remotely looking around an arbitrary location.

[0015] To solve such problems, the present invention has an autonomous control unit 32 that spontaneously corrects the movement to the movement destination Q based on the operation instruction based on the moving image obtained from the camera 20. The control unit 30 also has a delay measurement unit 33 for measuring the delay time td, which is the time from when the moving image is acquired from the camera 20 until the operation instruction of the operator P based on the moving image is received by the reception unit 22, as will be described later.

[0016] The operation of the autonomous control unit 32 will be explained in detail using Figures 5 to 7. First, let's assume that there is a delay of the same amount as the delay time td shown in Figure 4 between the operator P viewing the moving image on the moving image display unit 11 and actually issuing an operation instruction. The autonomous control unit 32, based on the video frame of the camera 20 at the time the receiving unit 22 receives operation instruction A (particularly frame F13 in Figure 5), instructs the moving unit 21 to perform autonomous operation 13A, which is an adjusted version of operation instruction A based on frame F13. Furthermore, if operation instructions B to R are received subsequently, for example, autonomous operations 14B to 25M are sequentially executed based on the video frames of the camera 20 at the time operation instructions B to R are received (F14 to F25), as shown in Figure 5.

[0017] In this way, by correcting the operation instruction A, which was made based on the video frame F1, based on the video frame F13, autonomous operation 13A can be performed, which allows for more accurate control that takes into account the surrounding environment obtained from the latest video frame F13 than by directly following the movement control A, which was made based only on the video frame F1. Furthermore, compared to simply performing movement control A, correction is performed using a later video frame equal to the delay time td, allowing for control that takes delay time into account. Furthermore, the correction applied when moving to the destination Q may vary, for example, depending on the delay time td. Alternatively, if the delay time td can be considered sufficiently small, such a correction value may be set to 0, and the telepresence robot TR may execute the instructions given by the operator P directly.

[0018] Specifically, we will describe the operation when operator P presses the forward command button 12a, which means moving forward on the movement instruction unit 12, after obtaining a moving image frame F1 as shown in Figure 6(a). When camera 20 acquires a video frame F1, the telepresence robot TR transmits the video frame F1 to the video display unit 11 (step S101). Based on the received video frame F1, operator P presses the forward command button 12a on the movement command unit 12 to indicate that he wants to move straight ahead (step S102). The receiving unit 22 receives the pressing of the forward instruction button 12a as an operation instruction (step S103). When the receiving unit 22 receives such an operation instruction, the destination estimation unit 31 estimates the direction in which the operator P would want to move the telepresence robot TR, according to the button pressed on the movement instruction unit 12 (step S104). The destination estimation unit 31 estimates the destination Q in the direction corresponding to the forward direction along the passage 41, based on the pre-set or distinguished passable passage 41 and the area outside the passage 42 which is the area outside the passage where passage is impossible, from the video frame F13 as shown in Figure 6(b) acquired by the camera 20 when an operation instruction is received (step S105). At this time, as is clear from Figure 6(b), the button pressed was the forward command button 12a, but in reality the passage 41 curves gently to the left and forward, so the autonomous control unit 32, based on the video frame F13, estimates that the forward command button 12a and the indicated movement "forward" are actually movements to the destination Q, which is "diagonally to the left and forward".

[0019] Furthermore, once the destination Q is estimated, the autonomous control unit 32 controls the moving unit 21 along the curvature of the passage 41 to move toward the destination Q by autonomous operation 13A (step S106). In this way, the autonomous control unit 32 performs corrections based on the video frame F13 when the receiving unit 22 receives the operation instruction, thereby moving to the destination Q.

[0020] At this time, the autonomous operation 13A of the autonomous control unit 32 is preferably to continue moving toward the estimated destination Q when the telepresence robot TR receives no movement instruction (no button pressed or no new operation instruction) (step S107). Furthermore, if no new operation instructions are received, a determination is made as to whether the state of no operation instructions has continued for a predetermined period of time or longer (step S108). In this embodiment, the autonomous control unit 32 continues moving toward the destination Q if it stops receiving operation instructions for a predetermined period of time during the process of moving toward the destination Q. In this way, even when there is no movement instruction, the system continues to move towards the previous destination Q for a predetermined time. This allows the system to continue moving even if an unintended interruption of movement instructions occurs, such as a momentary disconnection of network 9, and enables smooth operation to resume when network 9 is restored.

[0021] Furthermore, if the operation instructions are stopped for a predetermined period of time or longer (Yes in step S108), the movement of the telepresence robot TR may be stopped (step S109). In this way, by controlling the system to stop if no operation instructions are received for a predetermined period of time, accidents such as collisions can be prevented even if a full-scale disconnection of network 9 occurs, rather than just a momentary interruption.

[0022] Furthermore, when moving based on the autonomous operation 13A in step S106, in order for the operator P to accurately understand the direction of movement, when transmitting the video frame F13 to the video display unit 11, the destination Q and the predicted trajectory that the autonomous operation 13A will likely trace may be displayed as a dashed line in Figure 6(b). In this way, by displaying the predicted direction of movement, operator P can operate the robot while confirming whether the movement commands they have performed are being correctly transmitted to the telepresence robot TR, thereby contributing to improved accuracy of operation instructions.

[0023] Furthermore, in this embodiment, the destination Q was estimated simply based on the fact that it was on the passageway 41. However, as a method for estimating the destination Q, for example, one could link indoor / outdoor location information with map information in a remote location and select one of several destination candidates listed or predefined in the map information. Alternatively, the telepresence robot TR may be pre-programmed with training data consisting of a set of "captured video footage" as shown in Figures 6(a) and 6(b), a "movement command" that has been pressed, and the "desired location (destination)" that the operator P actually wants to move to, and the autonomous control unit 32 may be trained using machine learning. According to this machine learning method, the autonomous control unit 32 can take "movement instructions" and environmental information obtained from "moving images" as input, and output the optimal operation of the movement unit 21 from various patterns input in the training data. Furthermore, in addition to preparing training data, it is also possible to perform environmental learning by repeatedly operating the telepresence robot TR, using the video data obtained from the camera 20 as environmental information and the operation instructions.

[0024] When the receiving unit 22 receives different operation instructions from the operation unit 10 in step S107, the moving destination estimation unit 31 and the autonomous control unit 32 continue to operate by repeatedly performing the operations in steps S101 to S108 based on those operation instructions. Furthermore, if no new operation instructions are received for a predetermined period of time, the telepresence robot TR will stop as shown in step S109.

[0025] Now, examples of different autonomous operations of the autonomous control unit 32 are shown in Figures 8 to 10, along with schematic diagrams of the moving images and operation instructions. It should be noted that the autonomous operation 13A in this embodiment is not limited to this operation, but is merely shown as an example.

[0026] Figure 8 shows a schematic diagram of a situation where a person 43, acting as an obstacle to be avoided, is located on the passageway 41. In the video frame F13 shown in Figure 8, when operator P presses the forward command button 12a, the destination estimation unit 31 estimates the destination Q to be in front of person 43 in video frame F13, and stops in front of person 43 even if the forward command button 12a is held down. In this way, by identifying the person 43 based on the video frame F13, which is a moving image at the time the operation instruction is received, if the person 43 is in the direction of movement of the telepresence robot TR, the autonomous control unit 32 stops moving and autonomously operates to avoid a collision with the person 43. Furthermore, this autonomous operation can be applied not only to the person 43, but also to situations where the vehicle makes a decision to stop if there is any obstacle on the passageway 41.

[0027] Figure 9 is a schematic diagram showing a case where the direction of movement is unclear because the forward command button 12a is pressed. In the video frame F13 shown in Figure 9, the passageway 41 is interrupted in the forward direction, making it unclear whether to turn left or right at the end of the passage. In the video frame F13, the destination estimation unit 31 estimates the vicinity of the end of the passage 41 as the destination Q, and the autonomous control unit 32 instructs the movement unit 21 to move toward the destination Q and then stop. Thus, in situations where it is unclear whether to turn left or right at a T-junction, the autonomous control unit 32 moves toward the destination Q and then stops the telepresence robot TR. In such cases, when transmitting the video frame F13 to the operator P, a message indicating that a left / right turn instruction is required may be displayed, and the system may wait for the "movement instruction from operator P" to be received. When the receiving unit 22 receives the movement instruction B, it may then perform the left / right turn control again. Furthermore, when operation instruction B is given, the autonomous control unit 32 refers to the video frame F26 obtained from the camera 20 at the time the receiving unit 22 receives operation instruction B, and transmits it to the moving unit 21 as autonomous operation 26B.

[0028] Figure 10 shows an example of autonomous movement control when the vehicle has passed the right-turn point 44. Figure 10(a) shows an image of a moving image frame F1. Consider the case where operator P sees this moving image frame F1 displayed on the moving image display unit 11, finds a right turn location 44, and after passing through it, presses the right turn instruction button 12b on the movement instruction unit 12.

[0029] As already mentioned, due to the network 9's delay time td, by the time the right-turn instruction is received by the receiver 22, in the video frame F13, the telepresence robot TR has already passed the right-turn point 44, as shown in Figure 10(b). The autonomous control unit 32 recognizes and stores in advance that there is a point in the passage 41 where a right turn is possible, at the time of the video frame F1 shown in Figure 10(a). When the receiving unit 22 receives a right-turn instruction, the destination estimation unit 31 tentatively sets destination Q as the destination to the right. When the autonomous control unit 32 receives a right-turn instruction within a delay time td after remembering that there was a right-turnable location 44, it determines that the right-turn instruction was for a right turn to the right-turnable location 44 and corrects the destination Q to the right-turnable location 44. In other words, based on two pieces of environmental information—that there was a right-turnable point and that the vehicle had passed that point in the video frame F13 shown in Figure 10(b)—the autonomous control unit 32 corrects the destination Q to the right-turnable point 44 in the passage 41, and then selects autonomous control 13A to move backward by the distance it had moved too far forward, as shown in Figure 10(c). After the backward movement is completed, autonomous control 26N is performed to turn right in accordance with the right-turn instruction.

[0030] In this way, the autonomous control unit 32 spontaneously corrects the movement to the destination Q, which is determined by the operation instruction, based on environmental information obtained from the video frame F13. This configuration makes it possible to maintain ease of operation for the operator P while simultaneously reducing errors caused by network time differences.

[0031] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the spirit of the invention as described in the claims, unless otherwise specifically limited in the above description.

[0032] For example, in this embodiment, a robot in which the mobile unit 21 is driven by wheels or an endless belt has been described as a telepresence robot TR, but the present invention may also be applied to robots having other drive configurations.

[0033] Furthermore, in this embodiment, we have described a case where the autonomous control unit 32 corrects the destination Q by using machine learning. However, any method that takes into account the surrounding environment at the time the telepresence robot TR receives a movement instruction from the receiver 22 to make the correction may simply involve adding a predetermined correction value equal to the delay time td. Furthermore, although the destination estimation unit 31 and the autonomous control unit 32 are described as a single component of a control unit with separate functions in this embodiment, the system is not limited to this configuration, and the autonomous control unit may also have the function of estimating the destination.

[0034] The effects described in the embodiments of the present invention are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments of the present invention. [Explanation of Symbols]

[0035] 10 Control section 11. Moving image display unit 12 Movement instruction section 20. Means for acquiring moving images (camera) 21 Mobile unit 22 Receiving section 30 Control Unit 31 Movement destination estimation section 32 Autonomous Control Unit 100 control systems TR Robot (Telepresence Robot) P Operator Q Moving destination [Prior art documents] [Patent Documents]

[0036] [Patent Document 1] Japanese Patent Publication No. 2017-102705 [Patent Document 2] Patent No. 5503052

Claims

1. A robot that can move based on operator instructions via a network, A receiving unit for receiving the aforementioned operation instructions, The movement to the destination according to the aforementioned operation instructions is captured by video footage of the robot's surroundings. Among these is an autonomous control unit that spontaneously corrects based on the video image when the aforementioned operation instruction is received, The time when the video image transmitted to the operator was captured, and the time when the operation instruction was received. It has a delay measuring means for measuring the difference, The autonomous control unit is characterized by changing the amount of the correction according to the amount of the time difference. A robot.

2. A robot according to claim 1, The aforementioned destination can be selected from a predetermined list of candidate locations. robot.

3. A robot according to claim 1 or 2, The autonomous control unit uses the previously collected video footage, the operation instructions, and the destination to move to. A robot characterized by performing machine learning based on a set of datasets.

4. A robot according to any one of claims 1 to 3, The autonomous control unit performs the operation only for a predetermined time during the process of moving to the destination. A robot characterized by continuing the aforementioned movement when it stops receiving instructions.

5. The robot according to claim 4, The system is characterized by stopping if the aforementioned operation instructions are not received for a predetermined period of time or longer. A robot.

6. A robot according to any one of claims 1 to 5, The autonomous control unit, upon receiving the operation instruction, displays the video image of the surrounding environment of the robot. A robot characterized by its ability to input information.

7. A robot according to any one of claims 1 to 6, The robot is characterized by having a motion image acquisition means capable of capturing the surrounding environment as a motion image. A robot that serves as a symbol.

Images