Remote operation system, remote operation method, and program

Abstract

To provide a remote operation system, and a remote operation method and program, capable of enabling a robot to operate on the basis of situations that are unexpected or unrecognized by the system.SOLUTION: A remote operation system 1 is equipped with an operation terminal 30. The operation terminal 30 generates a display image at least on the basis of instruction proposal information and a position of a robot 10, in response to receiving the instruction proposal information proposing an instruction of operation of the robot 10 from a monitoring terminal 20 located in a predetermined range based on the robot 10, displays the display image, and transmits instruction information to the robot 10 in response to receiving input of the instruction information for instructing the robot 10 about the operation from an operator.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a remote control system, a remote control method, and a program, and more particularly to a remote control system, a remote control method, and a program for remotely controlling a robot. [Background technology]

[0002] In recent years, methods have been proposed for operators to appropriately remotely control work robots. For example, Patent Document 1 discloses a robot control system that, in a robot remote control system, identifies a task required for an object to be worked on by the robot based on an image captured by the robot of a work site, and transmits information about the task to an operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-160072 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the system described in the above-mentioned Patent Document 1, the robot is limited to work within the range that the system can expect or recognize, for example, work within the robot's visual field or work within the range that the operator can see through a screen from a remote location. Therefore, the robot cannot operate in light of situations that are beyond the expectation or recognition of the system, particularly situations that are beyond the expectation or recognition of the operator.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide a remote control system, a remote control method, and a program that enable a robot to operate in response to situations unexpected or unrecognized by the system. [Means for solving the problem]

[0006] A remote operation system according to an aspect of the present disclosure includes an operation terminal. In response to receiving instruction proposal information proposing an instruction for an operation of a robot from a monitoring terminal located within a predetermined range of the robot, the operation terminal generates a display image based at least on the instruction proposal information and a position of the robot, displays the display image, and in response to receiving input of instruction information for instructing the robot to operate from an operator, transmits the instruction information to the robot. This allows the operator to instruct the robot to operate after anticipating or recognizing a situation that the system cannot anticipate or recognize.

[0007] In the remote control system, the operation terminal may superimpose on the display image an option regarding whether or not to adopt the suggestion indicated by the instruction suggestion information, receive a selection of whether or not to adopt the instruction suggestion information, and transmit instruction information according to the selection to the robot. Since the operator only needs to select whether or not to adopt the suggestion, the instruction can be quickly executed.

[0008] The remote operation system may further include the monitoring terminal. In response to receiving a handwritten input image from a monitor for an image showing the surrounding environment of the robot, the monitoring terminal may generate instruction proposal information based on the handwritten input image and transmit the instruction proposal information to the operation terminal. This allows the monitor to easily provide instruction proposals in real time. Furthermore, the proposed instructions are not limited to predetermined content, so the monitor can propose dynamic and flexible instructions.

[0009] The remote operation system may further include the robot. The robot may have, as operation modes, a normal mode in which the robot operates based on instruction information received from the operation terminal, and an intervention mode in which the robot operates based on an operation plan generated by the robot or instruction information received from the monitoring terminal. The robot may switch from the normal mode to the intervention mode when it does not receive instruction information from the operation terminal within a predetermined time after the monitoring terminal transmits the instruction proposal information. This allows the robot to flexibly respond on site and avoid danger even when the operator does not notice the instruction proposal or when the operator takes time to make a decision.

[0010] In the above remote control system, the monitoring terminal may transmit instruction information to the robot in response to receiving the instruction information from a supervisor in the intervention mode, thereby enabling the supervisor at the site to flexibly respond and avoid danger.

[0011] In the remote control system, the operation terminal may superimpose mode information on the display image in response to the robot switching to the intervention mode, thereby enabling the operator to immediately recognize that the robot has switched to the intervention mode.

[0012] A remote control method according to one aspect of the present disclosure generates a display image based at least on the instruction proposal information and a position of the robot in response to receiving instruction proposal information proposing instructions for an operation of a robot from a monitoring terminal located within a predetermined range of the robot, displays the display image, and transmits instruction information to the robot in response to receiving input of instruction information for instructing the robot to operate from an operator, thereby enabling the operator to instruct the robot to operate after anticipating or recognizing a situation that the system cannot anticipate or recognize.

[0013] A program according to one aspect of the present disclosure causes a computer to realize a generation function of generating a display image based at least on the position of the robot and instruction proposal information that proposes instructions for a robot to move, in response to receiving the instruction proposal information from a monitoring terminal located within a predetermined range based on the robot, a display function of displaying the display image, and a transmission function of transmitting instruction information to the robot in response to receiving input of instruction information from an operator for instructing the robot to move. This allows the operator to instruct the robot to move after anticipating or recognizing a situation that the system cannot anticipate or recognize. Effect of the Invention

[0014] The present disclosure makes it possible to provide a remote control system, a remote control method, and a program that enable a robot to operate in response to situations unexpected or unrecognized by the system. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a configuration of a remote control system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram showing a usage state of a remote control system according to an embodiment of the present invention. [Diagram 3] FIG. 1 is an external perspective view showing an example of the external configuration of a robot according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the robot according to the present embodiment. [Diagram 5] 5 is a flowchart showing an example of the operation of the robot according to the present embodiment. [Figure 6] 5 is a flowchart showing an example of the operation of the robot according to the present embodiment. [Figure 7] FIG. 2 is a block diagram showing a functional configuration of a monitoring terminal according to the present embodiment. [Figure 8] 5 is a flowchart showing an example of an operation of the monitoring terminal according to the present embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a handwritten input image according to the embodiment; [Figure 10] FIG. 2 is a block diagram showing a functional configuration of an operation terminal according to the present embodiment. [Figure 11] 6 is a flowchart showing an example of an operation of the operation terminal according to the present embodiment. [Figure 12] FIG. 4 is a diagram showing an example of a display on a display unit according to the embodiment; [Figure 13] 10 is a flowchart showing an example of the operation of a robot according to a first modified example of the present embodiment. [Figure 14] 13 is a flowchart showing an example of the operation of a robot according to a second modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated descriptions will be omitted as necessary for clarity of explanation.

[0017] <Embodiment 1> First, a first embodiment of the present disclosure will be described. Fig. 1 is a block diagram showing a configuration of a remote control system 1 according to this embodiment. The remote control system 1 is a computer system for remotely controlling a robot.

[0018] The remote control system 1 includes a robot 10, a monitoring terminal 20, and an operation terminal 30, which are configured to be able to communicate with each other via a network N.

[0019] The network N is a wired or wireless network, and may be at least one of a local area network (LAN), a wide area network (WAN), the Internet, and other lines, or a combination of these.

[0020] The robot 10 is an example of a moving object that is remotely operated by the operation terminal 30. The robot 10 periodically transmits its own position information and sensing data of sensors mounted on the robot 10 to the operation terminal 30 via the network N. The robot 10 receives instructions from the operation terminal 30 via the network N and operates according to the instructions. The robot 10 is also configured to be able to operate autonomously depending on the operation mode.

[0021] The monitoring terminal 20 is a terminal device located around the robot 10. In this embodiment, the monitoring terminal 20 is a terminal device carried and used by a monitor located around the robot 10. Note that being located around the robot 10 may mean being located within a predetermined range based on the robot 10. The monitoring terminal 20 is, for example, a smartphone or tablet terminal having a touch panel.

[0022] The monitor using the monitoring terminal 20 grasps the situation around the robot from a different field of view than the robot 10. In other words, the monitor can grasp dynamic environmental changes and situations in a range that the robot 10 cannot recognize using sensors or the like. Depending on the situation around the robot 10, the monitor uses the monitoring terminal 20 to suggest to the operation terminal 30 the action to be instructed to the robot 10. Hereinafter, information related to the suggestion of an instruction for an action of the robot 10 is referred to as instruction proposal information.

[0023] The operation terminal 30 is a terminal device used by an operator to issue instructions for remotely operating the robot 10. The operation terminal 30 is a personal computer, a smartphone, or a tablet terminal. The operation terminal 30 receives position information and sensing data of the robot 10, and displays the environment within the range that the robot 10 can recognize on a display unit (not shown) based on the position information, sensing data, and map information. Then, the operator who has viewed the display uses the operation terminal 30 to instruct the robot 10 regarding the operation of the robot 10. This allows the operator to appropriately instruct the robot 10 to operate based on information within the range that the robot 10 can recognize. Note that, hereinafter, information for instructing the robot regarding an operation is referred to as instruction information.

[0024] Furthermore, when the operation terminal 30 receives instruction proposal information from the monitoring terminal 20, it visually displays that fact on the display unit. Then, the operator who has viewed the display uses the operation terminal 30 to transmit instruction information regarding the movement of the robot 10 to the robot 10. This allows the operator to dynamically and flexibly instruct the robot 10 to move based on information that the robot 10 cannot recognize.

[0025] 2 is a diagram showing a usage state of the remote control system 1 according to the present embodiment. For example, the robot 10 is used to introduce stores to passersby in a shopping district or to assist passersby in carrying luggage. The operator can converse with passersby through the display screen of the robot 10.

[0026] An observer G is present around the robot 10, and monitors the situation around the robot 10 while overlooking it. For example, when the observer G recognizes that multiple passersby are approaching from the front, which is outside the visual field of the robot 10, he / she inputs information by handwriting into the monitoring terminal 20 indicating that the robot 10 should move to the right. The handwriting may be a line drawn with a touch pen or the like, or a stamp indicating the content of the instruction placed at a position designated by the observer G.

[0027] Also, for example, if the supervisor G predicts that when the robot 10 moves its arm, the arm will hit a small child passing nearby, the supervisor G hand-writes information into the monitoring terminal 20 indicating that the robot 10's arm movement should be prohibited.

[0028] The monitoring terminal 20 that has received the input transmits to the operation terminal 30 instruction suggestion information corresponding to the input information.

[0029] FIG. 3 is an external perspective view showing an example of the external configuration of the robot 10 according to the present embodiment. FIG. 3 shows the external configuration of the robot 10 equipped with an end effector having a gripping function as an example of the robot 10. The robot 10 is roughly divided into a cart unit 110 and a main body unit 120. The cart unit 110 is a movable unit that contributes to the movement of the robot 10 in the direction of travel. The cart unit 110 supports two drive wheels 111 and one caster 112, each of which is in contact with the running surface, in a cylindrical housing. The two drive wheels 111 are arranged so that their rotation axes coincide with each other. Each drive wheel 111 is independently rotated by a motor (not shown). The caster 112 is a driven wheel, and a rotating shaft extending vertically from the cart unit 110 is provided to pivotally support the wheel away from the rotation axis of the wheel, and follows the moving direction of the cart unit 110.

[0030] The cart unit 110 is provided with a laser scanner 133 on the periphery of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane for each step angle, and outputs whether or not an obstacle exists in each direction. Furthermore, if an obstacle exists, the laser scanner 133 outputs the distance to the obstacle.

[0031] The main body 120 includes a movable part that exerts an action different from the movement in the advancing direction of the robot 10. Specifically, the main body 120 mainly includes a trunk 121 mounted on the upper surface of the cart 110, a head 122 placed on the upper surface of the trunk 121, an arm 123 supported on the side of the trunk 121, and a hand 124 installed at the tip of the arm 123. The arm 123 and the hand 124 are driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 can rotate around a vertical axis with respect to the cart 110 by the driving force of the motor (not shown). A hand camera 135 is disposed near the hand 124.

[0032] Head 122 mainly includes stereo camera 131 and display unit 141. Stereo camera 131 has a configuration in which two camera units having the same angle of view are arranged apart from each other, and outputs image signals captured by each camera unit.

[0033] The display unit 141 is, for example, a liquid crystal panel, and displays the face of a set character in animation, and displays information about the robot 10 in the form of text or icons.

[0034] Head 122 can rotate about a vertical axis relative to torso 121 by the driving force of a motor (not shown). Therefore, stereo camera 131 can capture images in any direction, and display unit 141 can present display content in any direction.

[0035] 4 is a block diagram showing the functional configuration of the robot 10 according to this embodiment. The robot 10 includes a control unit 150, a cart driving unit 145, an upper body driving unit 146, a display unit 141, a stereo camera 131, a laser scanner 133, a memory 180, a hand camera 135, and a communication unit 190. Note that the upper body driving unit 146, the display unit 141, the stereo camera 131, the laser scanner 133, and the hand camera 135 may be omitted.

[0036] The control unit 150 is a processor such as a CPU, and is stored in, for example, a control unit provided in the torso 121. The control unit 150 executes a control program read from the memory 180 to control the entire robot 10 and perform various arithmetic processing.

[0037] Here, the control unit 150 executes different controls depending on the operation mode. In this embodiment, the robot 10 has a first mode and a second mode as operation modes. The first mode is a mode in which the control unit 150 controls the cart driving unit 145 and the upper body driving unit 146 based on instruction information transmitted by the operation terminal 30. The first mode is also referred to as a normal mode. The second mode is a mode in which the control unit 150 controls the cart driving unit 145 and the upper body driving unit 146 based on an operation plan generated by the control unit 150 itself.

[0038] For example, the control unit 150 executes rotation control of the drive wheels by sending a drive signal to the cart drive unit 145 according to instruction information from the operation terminal 30 in the first mode, or according to the latest operation plan P stored in the memory 180 in the second mode. The control unit 150 also receives a feedback signal from an encoder or the like from the cart drive unit 145 to grasp the moving direction and moving speed of the cart unit 110.

[0039] The carriage driving unit 145 includes the driving wheels 111 , and a driving circuit and a motor for driving the driving wheels 111 .

[0040] Upper body driver 146 includes arm 123, hand 124, torso 121, head 122, and drive circuits and motors for driving these. Control unit 150 realizes an extension motion, a grasping motion, and a gesture by sending a drive signal to upper body driver 146. Control unit 150 also receives a feedback signal from an encoder or the like from upper body driver 146 to grasp the position and moving speed of arm 123 and hand 124, and the orientation and rotation speed of torso 121 and head 122.

[0041] The display unit 141 receives the image signal generated by the control unit 150 and displays it.

[0042] The stereo camera 131 captures images of the surrounding environment in which the robot 10 exists in response to a request from the control unit 150, and transfers the captured image signal to the control unit 150. The control unit 150 performs image processing using the captured image signal, and converts the captured image signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether or not an obstacle exists in the moving direction in response to a request from the control unit 150, and transfers a detection signal that is the detection result to the control unit 150.

[0043] The hand camera 135 is, for example, a distance image sensor, and is used to recognize the distance, shape, direction, etc. of the object to be grasped. The hand camera 135 includes an image sensor in which pixels that photoelectrically convert an optical image incident from the target space are arranged two-dimensionally, and outputs the distance to the object for each pixel to the control unit 150. Specifically, the hand camera 135 includes an irradiation unit that irradiates the target space with pattern light, receives the reflected light with the image sensor, and outputs the distance to the object captured by each pixel from the distortion and size of the pattern in the image. The control unit 150 grasps the state of a wider surrounding environment with the stereo camera 131, and grasps the state near the object to be grasped with the hand camera 135.

[0044] The memory 180 is a non-volatile storage medium, such as a solid state drive. The memory 180 stores various parameter values, functions, lookup tables, and the like used for control and calculation, in addition to a control program for controlling the robot 10. In particular, the memory 180 stores an environmental map M and a motion plan P.

[0045] The communication unit 190 is a communication interface with the network N, and is, for example, a wireless LAN unit. The communication unit 190 receives instruction information transmitted from the operation terminal 30 and passes it to the control unit 150. Furthermore, the communication unit 190 transmits position information of the robot 10 and various detection results acquired from a GPS receiver (not shown) to the monitoring terminal 20 and the operation terminal 30 according to the control of the control unit 150.

[0046] 5 and 6 are flow charts showing an example of the operation of the robot 10 according to this embodiment. Fig. 5 shows an example of the operation of the robot 10 when the robot 10 is set to the first mode.

[0047] First, when the control unit 150 sets the operation mode to the first mode (S10), it determines whether or not instruction information has been received from the operation terminal 30 (S11). If instruction information has been received from the operation terminal 30 (Yes in S11), the control unit 150 controls the cart driving unit 145 and the upper body driving unit 146 based on the instruction information, thereby operating the cart unit 110 and the main body unit 120 of the robot 10 (S12). Then, the control unit 150 advances the process to S13.

[0048] On the other hand, if no instruction information has been received from the operation terminal 30 (No in S11), the process proceeds directly to S13.

[0049] In S13, the control unit 150 judges whether or not to end the operation. Examples of cases where the operation is ended include when instruction information to end the operation is received from the operation terminal 30, and when the power supply to the robot 10 is stopped. The control unit 150 repeats the processes shown in S11 to S12 until it is judged to end the operation (No in S13).

[0050] FIG. 6 shows an example of the operation of the robot 10 when it is set to the second mode.

[0051] First, when the control unit 150 sets the operation mode to the second mode (S14), it creates an operation plan P based on the environmental map M stored in the memory 180 and its own position information (S15). The control unit 150 stores the created operation plan P in the memory 180. Then, the control unit 150 controls the cart driving unit 145 and the upper body driving unit 146 based on the operation plan P, thereby operating the cart unit 110 and the main body unit 120 of the robot 10 (S16).

[0052] Next, the control unit 150 judges whether or not to end the operation (S17). The control unit 150 repeats the processes shown in S15 to S16 until it judges to end the operation (No in S17).

[0053] 7 is a block diagram showing the functional configuration of the monitoring terminal 20 according to this embodiment. The monitoring terminal 20 includes a memory 200, a communication unit 210, an input unit 220, a display unit 230, and a monitoring control unit 240.

[0054] The memory 200 is a non-volatile storage medium, such as a solid state drive. The memory 200 stores various parameter values, functions, lookup tables, and the like used for control and calculation, in addition to a control program for controlling the monitoring terminal 20. In particular, the memory 200 stores an environmental map M.

[0055] The communication unit 210 is a communication interface with the network N. The communication unit 210 receives position information of the robot 10 and various detection results from the robot 10, and passes them to the monitoring control unit 240 for display. The communication unit 210 may receive instruction information addressed to the robot 10 from the operation terminal 30, and passes them to the monitoring control unit 240 for display. The communication unit 210 also cooperates with the monitoring control unit 240 to transmit instruction proposal information to the operation terminal 30.

[0056] The input unit 220 includes a touch panel superimposed on the display unit 230, push buttons provided on the periphery of the display unit 230, etc. The input unit 220 accepts a handwritten input image input by the monitor by touching the touch panel to specify the content of an instruction to be proposed to the operation terminal 30, and passes it to the monitor control unit 240.

[0057] The display unit 230 is, for example, a liquid crystal panel, and displays a display image showing the surrounding environment of the robot 10. The surrounding environment of the robot 10 may be an environment within a predetermined range based on the robot 10. The display unit 230 also displays the input handwritten input image superimposed on the display image.

[0058] The monitoring control unit 240 is a processor such as a CPU, and executes control programs read from the memory 200 to control the entire monitoring terminal 20 and perform various arithmetic processing. Specific control by the monitoring control unit 240 will be described with reference to FIG.

[0059] 8 is a flowchart showing an example of the operation of the monitoring terminal 20 according to this embodiment. First, the monitoring control unit 240 of the monitoring terminal 20 receives position information of the robot 10 from the robot 10 via the communication unit 210 (S20). In addition to the position information, the monitoring control unit 240 may receive various detection results from the robot 10 via the communication unit 210. Furthermore, the monitoring control unit 240 may receive instruction information addressed to the robot 10 from the operation terminal 30 via the communication unit 210.

[0060] Next, the monitoring control unit 240 generates a display image showing the surrounding environment of the robot 10 based on at least the environmental map M stored in the memory 200 and the position information of the robot 10 (S21). Note that, if the monitoring control unit 240 has acquired various detection results of the robot 10, the monitoring control unit 240 may further use the various detection results as a basis for generating the display image.

[0061] Next, the monitoring control unit 240 causes the display unit 230 to display a display image (S22). Next, the monitoring control unit 240 determines whether input of a handwritten input image has been received from the monitor (S23). If input has been received (Yes in S23), in response to the input being received, the monitoring control unit 240 generates instruction suggestion information based on the handwritten input image (S24).

[0062] The instruction proposal information includes the moving direction, moving amount or trajectory of the carriage, arm 123 or hand 124 of the robot 10, movable prohibited parts of the robot 10, or position information of the moving destination of the robot 10. In the present embodiment 1, the instruction proposal information is information including a handwritten input image and an input position on the environmental map M. However, if the monitoring control unit 240 has an image recognition function, the instruction proposal information may be information including a recognition result of the handwritten input image and an input position on the environmental map M. Alternatively, if the display image displayed on the display unit 230 is synchronized with the display image displayed on the display unit 330 of the operation terminal 30, the instruction proposal information may be the display image itself on which the handwritten input image is superimposed.

[0063] The monitoring control unit 240 transmits the instruction proposal information to the operation terminal 30 via the communication unit 210 (S25).

[0064] Next, the monitoring control unit 240 judges whether or not to end the series of processes (S26). The series of processes may be ended when the operation of the robot 10 ends or when the operation mode of the robot 10 is switched to the second mode. The monitoring control unit 240 repeats the processes shown in S20 to S25 until it judges to end the processes (No in S26).

[0065] FIG. 9 is a diagram showing an example of a handwritten input image 600 according to the present embodiment. The display unit 230 of the monitoring terminal 20 displays a display image 500 showing the surrounding environment of the robot 10. For example, the display image 500 may include an image area showing the robot 10 and its surrounding environment as seen from a predetermined field of view. The display image 500 shown in FIG. 9 is three-dimensional, but the display image 500 may be a two-dimensional image in which the position of the robot 10 is illustrated on a two-dimensional environmental map M. Furthermore, the display image may include a movement path 501 of the robot 10 on an image area showing the surrounding environment. The movement path 501 may be generated based on instruction information. Furthermore, an obstacle 502 estimated based on the detection result of the robot 10 may be included on the image area showing the surrounding environment. Such a display image 500 may be generated by the monitoring control unit 240 using computer graphics.

[0066] The observer inputs a handwritten input image 600 on the displayed display image 500. A method of inputting a handwritten input image includes directly inputting the image by touching a relevant portion on a touch panel using a user's finger, a touch pen, or the like. However, the method of inputting a handwritten input image is not limited to this. For example, a handwritten input image may be input by selecting a predetermined figure using a mouse or the like and specifying the position and size. A handwritten input image may be input as a two-dimensional line or figure, or as a three-dimensional object.

[0067] In this figure, the observer inputs a left arrow indicating the movement direction as a handwritten input image 600 to suggest moving the robot 10 to the left. The observer may draw a trajectory by overwriting the movement route 501. The drawn trajectory is the movement route proposed by the observer. The observer may also specify the movement location of the robot 10 with a marker.

[0068] Furthermore, if the observer wishes to suggest a movement of the arm 123 or hand 124 of the robot 10, the display unit 230 may display the movable area and movable axis in three dimensions. The observer may then specify the direction and amount of movement with a marker. Conversely, the observer may be able to specify a prohibited movement area, in which movement is prohibited, with a marker.

[0069] In this way, by allowing the observer to intuitively input a handwritten input image, it is possible to easily provide suggested instructions in real time. Furthermore, the suggested instructions are not limited to predetermined contents. Therefore, the observer can propose dynamic and flexible instructions.

[0070] 10 is a block diagram showing the functional configuration of the operation terminal 30 according to this embodiment. The operation terminal 30 includes a memory 300, a communication unit 310, an input unit 320, a display unit 330, and an operation control unit 340.

[0071] The memory 300 is a non-volatile storage medium, for example, a solid state drive. The memory 300 stores various parameter values, functions, lookup tables, etc. used for control and calculation, in addition to a control program for controlling the operation terminal 30. In particular, the memory 300 stores an environmental map M.

[0072] The communication unit 310 is a communication interface with the network N. The communication unit 310 receives position information and various detection results of the robot 10 from the robot 10, and passes them over to the operation control unit 340. The communication unit 310 also receives instruction proposal information from the monitoring terminal 20, and passes them over to the operation control unit 340. The communication unit 310 also cooperates with the operation control unit 340 to transmit instruction information to the robot 10.

[0073] The input unit 320 includes a mouse, a keyboard, a joystick, a touch panel superimposed on the display unit 330, and push buttons provided on the periphery of the display unit 330. The input unit 320 accepts input of instruction information for the robot 10 by the operator clicking the mouse, inputting a command, touching the touch panel, or tilting the lever of the joystick, and passes it to the operation control unit 340.

[0074] The display unit 330 is, for example, a liquid crystal panel, and displays a display image showing the surrounding environment of the robot 10. When the display unit 330 receives instruction proposal information, the display unit 330 displays a display image further including the received instruction proposal information. The display unit 330 also displays instruction information input by the operator superimposed on the display image.

[0075] The operation control unit 340 is a processor such as a CPU, and executes control programs read from the memory 300 to control the entire operation terminal 30 and perform various arithmetic processing. Specific control by the operation control unit 340 will be described with reference to FIG.

[0076] 11 is a flowchart showing an example of the operation of the operation terminal 30 according to the present embodiment. The operation terminal 30 may operate when the robot 10 is set to the first mode.

[0077] First, the operation control unit 340 of the operation terminal 30 receives position information of the robot 10 from the robot 10 via the communication unit 310 (S30). Next, the operation control unit 340 determines whether or not it has received instruction proposal information from the monitoring terminal 20 via the communication unit 310 (S31).

[0078] When the operation control unit 340 has not received instruction proposal information from the monitoring terminal 20 (No in S31), it generates a display image showing the surrounding environment of the robot 10 based on the environmental map M stored in the memory 300 and the position information of the robot 10 (S32). The method of generating the display image may be similar in part or in whole to the method of generating the display image displayed on the display unit 230 of the monitoring terminal 20. The operation control unit 340 advances the process to S34.

[0079] On the other hand, when the operation control unit 340 receives the instruction proposal information from the monitoring terminal 20 (Yes in S31), the operation control unit 340 generates a display image based on the environmental map M, the instruction proposal information, and the position information of the robot 10 in response to the reception (S33). The display image is an image that visually displays the instruction proposal in the surrounding environment of the robot 10. The operation control unit 340 advances the process to S34.

[0080] In S34, the operation control unit 340 causes the display unit 330 to display the display image generated in S32 or S33.

[0081] The operation control unit 340 then determines whether the input unit 320 has received input of instruction information from the operator (S35). If input has been received (Yes in S35), the operation control unit 340 transmits the instruction information to the robot 10 via the communication unit 310 in response to the received input (S36). The operation control unit 340 then advances the process to S37. On the other hand, if there has been no input (No in S35), the operation control unit 340 advances the process directly to S37.

[0082] Next, the operation control unit 340 judges whether or not to end the series of processes (S37). The series of processes may be ended when the operation of the robot 10 ends or when the operation mode of the robot 10 is switched to the second mode. The operation control unit 340 repeats the processes shown in S30 to S36 until it judges to end the processes (No in S37).

[0083] 12 is a diagram showing an example of a display on the display unit 330 according to this embodiment. The display unit 330 of the operation terminal 30 displays a display image 500 showing the surrounding environment of the robot 10. The display unit 330 also displays a captured image 510 captured by the stereo camera 131 of the robot 10. The display image 500 and the captured image 510 allow the operator to grasp the surrounding environment of the robot 10.

[0084] The display unit 330 also displays operation units 520, 530 for giving operation instructions to the robot 10. This allows the operator to specifically specify the direction and amount of movement of the robot 10.

[0085] Here, when the operation terminal 30 receives the instruction proposal information, the display unit 330 displays the handwritten input image 600 as the instruction proposal information by superimposing it on the display image 500. Note that, when the instruction proposal information includes the trajectories of the cart unit 110 or the arm 123, the display unit 330 may reproduce the trajectories.

[0086] Then, the display unit 330 displays options 610 regarding whether or not to adopt the suggestion indicated by the instruction suggestion information, superimposed on the display image 500. This display may be a pop-up type.

[0087] The operator who views the display checks the surrounding environment indicated by the instruction suggestion information, and moves the robot 10 little by little in the direction indicated by the instruction suggestion information, thereby checking the safety of moving in the direction indicated by the instruction suggestion information.

[0088] Then, the operator selects the option 610 of "ACCEPT" or "REJECT". In response to this, the input unit 320 receives from the operator a selection of whether or not to adopt the instruction proposal information. In response to receiving the selection, the operation control unit 340 transmits instruction information according to the selected option to the robot 10 via the communication unit 310.

[0089] By displaying the instruction proposal in an easy-to-understand manner, the operator can easily understand the contents of the proposal. Since the instruction proposal information is displayed visually, even if the operator is talking to a person facing the robot 10 through the robot 10, the conversation is not interrupted. Furthermore, the operator can easily adopt the action proposed by the monitoring terminal 20 with a single click. Therefore, the robot 10 can quickly carry out the action proposed by the monitoring terminal 20.

[0090] The embodiment has been described above. According to the present embodiment, a monitor having a different field of view from the robot 10 uses the monitoring terminal 20 to grasp situations that cannot be recognized from the robot's field of view or situations that the operator cannot recognize based on map information, that is, situations that are unexpected or unrecognized by the system, and then proposes instructions. Therefore, the operator can instruct the robot 10 to operate in light of situations that are unexpected or unrecognized by the system. This allows the robot to operate in light of situations that are unexpected or unrecognized by the system.

[0091] The embodiment can also be modified as follows.

[0092] <First Modification of the Present Embodiment> In the first modified example, when the operation terminal 30 does not respond to the instruction proposed by the monitoring terminal 20, the robot 10 performs an autonomous operation. Specifically, when the operation terminal 30 does not transmit instruction information within a predetermined time after the monitoring terminal 20 transmits the instruction proposal information, the robot 10 switches the operation mode from the first mode to the second mode. In this case, when the monitoring terminal 20 transmits the instruction proposal information to the operation terminal 30, the monitoring terminal 20 may also transmit the instruction proposal information to the robot 10. The second mode switched to in the above case may be referred to as an intervention mode. In the intervention mode, even if the operation terminal 30 transmits instruction information to the robot 10, the robot 10 may operate autonomously regardless of the instruction information.

[0093] 13 is a flowchart showing an example of the operation of the robot 10 according to the first modified example of this embodiment. The steps shown in FIG. 13 include S40 to S45 in addition to the steps shown in FIG.

[0094] When the control unit 150 determines in S11 that it has not received instruction information from the operation terminal 30 (No in S11), it determines whether a predetermined time has elapsed since the monitoring terminal 20 transmitted the instruction proposal information (S40). When the predetermined time has not elapsed (No in S40), the control unit 150 returns the process to S11. On the other hand, when the predetermined time has elapsed (Yes in S40), the control unit 150 switches the operation mode to the second mode (S41). Then, the control unit 150 notifies the operation terminal 30 via the communication unit 190 that it has switched to the second mode (S42).

[0095] In response to switching to the second mode, the control unit 150 executes S43 to S44 similar to S15 to S16 in FIG. 6, and operates the cart unit 110 and the main body unit 120 based on the operation plan created by the control unit 150 itself.

[0096] Then, the control unit 150 repeats S43 to S44 until a predetermined time has elapsed since switching to the second mode (No in S45), and when the predetermined time has elapsed (Yes in S45), the second mode is cancelled and the process proceeds to S13.

[0097] In this way, even if the operator is unaware of the instruction suggestions or takes time to make a decision, the system can respond flexibly on-site to avoid danger.

[0098] In addition, the display unit 330 of the operation terminal 30 may superimpose mode information on the display image in response to the robot 10 switching to the second mode, i.e., in response to receiving a switching notification. This allows the operator to immediately know that the robot 10 has switched to the second mode, and the operator can take appropriate action.

[0099] <Second Modification of the Present Embodiment> In the second modified example, when the operation terminal 30 does not respond to the instruction proposed by the monitoring terminal 20, the robot 10 switches to a third mode in which the robot 10 operates according to the instruction of the monitoring terminal 20. Specifically, when the operation terminal 30 does not transmit instruction information within a predetermined time after the monitoring terminal 20 transmits the instruction proposal information, the robot 10 switches the operation mode from the first mode to the third mode. In the third mode, the monitoring terminal 20 transmits instruction information to the robot 10 in response to receiving input of instruction information from the monitor. The third mode switched to in the above case may be referred to as an intervention mode. In the intervention mode, even if the operation terminal 30 transmits instruction information to the robot 10, the robot 10 may operate based on the instruction information from the monitoring terminal 20.

[0100] 14 is a flowchart showing an example of the operation of the robot 10 according to the second modified example of this embodiment. The steps shown in FIG. 14 include S50 to S52 instead of S43 to S45 shown in FIG.

[0101] In response to notifying the operation terminal 30 of the switching to the third mode, the control unit 150 determines whether or not instruction information has been received from the monitoring terminal 20 (S50). If instruction information has been received from the monitoring terminal 20 (Yes in S50), the control unit 150 operates the cart unit 110 and the main body unit 120 based on the instruction information (S51), and proceeds to processing at S52. On the other hand, if instruction information has not been received from the monitoring terminal 20 (No in S50), the processing proceeds directly to S52.

[0102] Then, the control unit 150 repeats S50 to S51 until a predetermined time has elapsed since switching to the third mode (No in S52), and when the predetermined time has elapsed (Yes in S52), it cancels the third mode and proceeds to the process at S13. In this way, when the operator does not notice the instruction proposal or when the operator is taking time to make a decision, the supervisor at the site can flexibly respond and avoid danger at his / her discretion.

[0103] In addition, the display unit 330 of the operation terminal 30 may superimpose mode information on the display image in response to the robot 10 switching to the third mode, i.e., in response to receiving a switching notification. This allows the operator to immediately know that the robot 10 has switched to the third mode, and the operator can take appropriate action.

[0104] The present disclosure is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present disclosure. For example, in the present embodiment, the operation terminal 30 determines whether to adopt the instruction proposal information from the monitoring terminal 20, and issues an operation instruction to the robot 10 based on the decision. However, if the robot 10 receives the instruction proposal information from the monitoring terminal 20 before receiving the instruction information from the operation terminal 30, the robot 10 may operate based on the instruction proposal information, and the operation terminal 30 may later confirm the operation based on the instruction proposal information. In this case, the operation terminal 30 transmits the instruction information to the robot 10 by including information indicating confirmation in the instruction information.

[0105] In the present embodiment, the monitoring terminal 20 is carried by the monitor, but the monitoring terminal 20 may not be carried by the monitor and may be installed on a moving object such as a drone. In this case, the operator of the moving object may be the monitor. The monitoring terminal 20 may be installed in a plurality of locations within the movement range of the robot 10. The monitoring terminals 20 around the robot 10 may take pictures of the surrounding environment of the robot 10 from angles different from those of the robot 10, and may transmit instruction proposal information to the operation terminal 30 when a danger is predicted by image recognition.

[0106] In the present embodiment, the monitor inputs a handwritten input image, but the input image does not have to be handwritten.

[0107] In the above first or second modified example, the robot 10 transitions to the second mode or the third mode when the operation terminal 30 does not transmit instruction information within a predetermined time after the monitoring terminal 20 transmits the instruction proposal information. However, the robot 10 may switch to the second mode or the third mode when a request to switch the operation mode is received from the operation terminal 30. In this case, a help mark is displayed on the display unit 330 of the operation terminal 30, and the operation terminal 30 may transmit a request to switch the operation mode to the robot 10 in response to the operator selecting the help mark. [Explanation of symbols]

[0108] 1 Remote Control System 10. Robot 20 Monitoring terminal 30 Operation terminal 110 Bogie section 111 Drive wheel 112 Caster 120 Main body 121 Torso 122 Head 123 Arm 124 Hands 131 Stereo Camera 133 Laser Scanner 135 Hand Camera 141 Display section 145 Truck drive unit 146 Upper body drive unit 150 Control section 180 Memory 190 Communications Department 200 Memory 210 Communications Department 220 Input section 230 Display section 240 Monitoring and control section 300 Memory 310 Communications Department 320 Input section 330 Display section 340 Operation control section 500 Display images 501 Travel Route 502 Obstacles 510 Photographed Images 520 Operation section 530 Operation section 600 handwritten input images 610 Choice G Observer M Environmental Map P Motion Planning

Claims

1. generating a display image based at least on the position of the robot in response to receiving instruction suggestion information proposing an instruction for an operation of the robot from a monitoring terminal located within a predetermined range based on the robot; Displaying the display image; In response to receiving, from an operator, instruction information for instructing the robot regarding an operation, the instruction information is transmitted to the robot; superimposing an option regarding whether or not to adopt the suggestion indicated by the instruction suggestion information on the display image; Accept a selection of whether or not to adopt the instruction proposal information; A remote control system comprising an operation terminal that transmits instruction information according to the selection to the robot.

2. generating a display image based at least on the position of the robot in response to receiving instruction suggestion information proposing an instruction for an operation of the robot from a monitoring terminal located within a predetermined range based on the robot; Displaying the display image; In response to receiving, from an operator, input of instruction information for instructing the robot regarding an operation, the instruction information is transmitted to the robot. An operation terminal; The robot, The robot comprises: The robot has, as operation modes, a normal mode in which the robot operates based on instruction information received from the operation terminal, and an intervention mode in which the robot operates based on an operation plan generated by the robot or instruction information received from the monitoring terminal, If the monitoring terminal does not receive instruction information from the operation terminal within a predetermined time after transmitting the instruction proposal information, the monitoring terminal switches from the normal mode to the intervention mode. Remote control system.

3. The monitoring terminal is further provided. The monitoring terminal includes: generating instruction suggestion information based on a handwritten input image in response to receiving, from a supervisor, input of a handwritten input image for an image showing a surrounding environment of the robot; The instruction proposal information is transmitted to the operation terminal. The remote control system according to claim 1 or 2.

4. The robot further comprises: The robot comprises: The robot has, as operation modes, a normal mode in which the robot operates based on instruction information received from the operation terminal, and an intervention mode in which the robot operates based on an operation plan generated by the robot or instruction information received from the monitoring terminal, If the monitoring terminal does not receive instruction information from the operation terminal within a predetermined time after transmitting the instruction proposal information, the monitoring terminal switches from the normal mode to the intervention mode. The remote control system according to claim 1 .

5. The monitoring terminal is further provided. The monitoring terminal transmits instruction information to the robot in response to receiving input of instruction information from a monitor in the intervention mode. The remote control system according to claim 2 or 4.

6. The operation terminal displays mode information superimposed on the display image in response to the robot being switched to the intervention mode. The remote control system according to claim 2 or 4.

7. generating a display image based at least on the position of the robot in response to receiving instruction suggestion information proposing an instruction for an operation of the robot from a monitoring terminal located within a predetermined range based on the robot; Display the display image, In response to receiving, from an operator, instruction information for instructing the robot regarding an operation, the instruction information is transmitted to the robot; superimposing an option regarding whether or not to adopt the suggestion indicated by the instruction suggestion information on the display image; Accept a selection of whether or not to adopt the instruction proposal information; Transmitting instruction information according to the selection to the robot Remote operation method.

8. The operation terminal comprises: generating a display image based at least on the position of the robot in response to receiving instruction suggestion information proposing an instruction for an operation of the robot from a monitoring terminal located within a predetermined range based on the robot; Display the display image, In response to receiving, from an operator, instruction information for instructing the robot regarding an operation, the instruction information is transmitted to the robot; The robot comprises: The robot has, as operation modes, a normal mode in which the robot operates based on instruction information received from the operation terminal, and an intervention mode in which the robot operates based on an operation plan generated by the robot or instruction information received from the monitoring terminal, If the monitoring terminal does not receive instruction information from the operation terminal within a predetermined time after transmitting the instruction proposal information, the monitoring terminal switches from the normal mode to the intervention mode. Remote operation method.

9. a generation function of generating a display image based at least on the position of the robot in response to receiving instruction suggestion information proposing an instruction for an operation of the robot from a monitoring terminal located within a predetermined range based on the robot; a display function for displaying a display image; a transmission function of transmitting instruction information to the robot in response to receiving, from an operator, instruction information for instructing the robot on an operation; A display function of displaying, on the display image, options regarding whether or not to adopt the suggestion indicated by the instruction suggestion information in a superimposed manner; A reception function for receiving a selection of whether or not to adopt the instruction proposal information; a transmission function for transmitting instruction information to the robot according to the selection; A program that enables a computer to achieve the above.

Images