Remote operation system, remote operation method, and control program

Abstract

To provide a remote operation system enabling a maneuvered object to execute high precision processing while enabling a user to perform an intuitive operation.SOLUTION: A remote operation system includes: a recognition part for recognizing a graspable article, a movable zone of an operation-target article, and an immovable zone, on the basis of a pickup image of an environment where the operation-target article exists; an operation terminal for receiving an input of hand-written information into the pickup image; and an estimation part for estimating a grasp-target article required to grasp from among graspable objects on the basis of the graspable articles and hand-written information, and estimating contents of action, on a grasp-target article, required to the operation-target article. In a case where the hand-written information contains information on a transportation route of the grasp-target article, the estimation part estimates a transportation route located in the movable zone of the operation-target article, among transportation routes, as a moving route of the operation-target article, and estimates a transportation route located in the immovable zone of the operation-target article as a moving route of an end effector that grasps the grasp-target article.SELECTED DRAWING: Figure 22

Description

[Technical field]

[0001] The present disclosure relates to a remote control system, a remote control method, and a control program. [Background technology]

[0002] There is known a technique for remotely controlling an operated object equipped with an end effector (e.g., a robot equipped with a gripping unit (e.g., a hand or an adsorption unit) that is an end effector at the tip of an arm) to perform a gripping operation or the like. For example, Patent Document 1 discloses a system that determines how to control a robot based on handwritten information of a user received at an operation terminal that displays captured images of the robot's surroundings, and remotely controls the robot. This system can remotely control a robot based on intuitive handwritten instructions from a user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-094605 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the system disclosed in Patent Document 1, when handwritten information includes information on the transport path of the object to be grasped, it is necessary to determine whether the transport path represents the movement path (travel path) of the robot or the movement path of the end effector caused by the arm operation of the robot. Refuse There was a problem in that it was not possible to

[0005] The present disclosure has been made to solve such problems, and provides a method for controlling a device while allowing the user to perform an intuitive operation. Automatically distinguish whether the transport path information included in the handwritten information is the movement path of the end effector or the movement path of the robot. The present invention provides a remote control system, a remote control method, and a control program that can realize the remote control system. [Means for solving the problem]

[0006] A remote operation system according to the present disclosure is a remote operation system for remotely operating an operated object having an end effector, comprising: an imaging unit for imaging an environment in which the operated object exists; a recognition unit for recognizing a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an unmovable area of ​​the operated object based on the captured image of the environment; an operation terminal for displaying the captured image and accepting input of handwritten information for the captured image; and an estimation unit for estimating a graspable object that is requested to be grasped by the end effector from the graspable object based on the graspable object and the handwritten information, and estimating a content of an action requested of the operated object with respect to the graspable object, Within a range in which either the operated object or the end effector can move When information on the transport path of the grasped object is included, the transport path located in the movable area of ​​the operated object is estimated to be the movement path of the operated object, and the transport path located in the immovable area of ​​the operated object is estimated to be the movement path of the end effector gripping the grasped object. In this remote control system, the user can perform a desired process, such as making the operated object transport the grasped object, by performing a more intuitive handwritten input without having to perform handwritten input while remembering a preset instruction figure. Furthermore, in this remote control system, when handwritten information includes information on the transport path of the grasped object, it is automatically determined whether the transport path represents the movement path (travel path) of the operated object or the movement path of the end effector. do It is possible.

[0007] When the handwritten information includes information on a transport route of the object to be grasped, and an unmovable area sandwiched between two of the movable areas exists on the transport route, the estimation unit estimates a detour route for the object to be grasped such that there is no unmovable area sandwiched between the two movable areas, and the remote control system may further include an output unit that outputs information on the detour route estimated by the estimation unit.

[0008] The output unit may cause the operation terminal to display information about the detour route.

[0009] The recognition unit may recognize, among the graspable objects, a graspable object other than the grasp target object as an immovable area of ​​the operated object.

[0010] The target object may be a robot capable of moving independently.

[0011] When the handwritten information includes information on the transport path of the object to be grasped, the estimation unit may estimate the transport path that is located in a movable area of ​​the operated object as the movement path of the robot that is the operated object, and may estimate the transport path that is located in an immovable area of ​​the operated object as the movement path of the end effector attached to the tip of the arm of the robot by operating the arm.

[0012] The imaging unit may be attached to the target object.

[0013] The imaging unit may be provided in a location separate from the operated object in the environment in which the operated object exists.

[0014] The operation terminal may complete input of the handwritten information by accepting a predetermined handwriting end operation by a user.

[0015] The handwritten information may include a first image simulating the content of the action of the operated object relative to the grasp object.

[0016] The estimation unit may use a trained model such as a neural network trained by deep learning to estimate the action content requested of the operated subject with respect to the object to be grasped from the first image of the handwritten information.

[0017] A remote operation method according to the present disclosure is a remote operation method using a remote operation system that remotely operates an operated object having an end effector, comprising the steps of: capturing an image of an environment in which the operated object exists; recognizing a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an unmovable area of ​​the operated object based on the captured image of the environment; accepting input of handwritten information for the displayed captured image on an operation terminal that displays the captured image; estimating a graspable object that is requested to be grasped by the end effector from the graspable object based on the graspable object and the handwritten information; and estimating the operation content requested of the operated object with respect to the graspable object, Within a range in which either the operated object or the end effector can move When information on the transport path of the grasped object is included, the transport path located in the movable area of ​​the operated object is estimated to be the movement path of the operated object, and the transport path located in the immovable area of ​​the operated object is estimated to be the movement path of the end effector gripping the grasped object. In this remote control method, the user can perform a desired process, such as making the operated object transport the grasped object, by performing a more intuitive handwritten input without having to perform handwritten input while remembering a preset instruction figure. Furthermore, in this remote control method, when handwritten information includes information on the transport path of the grasped object, it is automatically determined whether the transport path represents the movement path (travel path) of the operated object or the movement path of the end effector. do It is possible.

[0018] A control program according to the present disclosure is a control program for causing a computer to execute a remote operation process by a remote operation system that remotely operates an operated object having an end effector, the control program causing a computer to execute the following processes: capturing an image of an environment in which the operated object exists; recognizing a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an unmovable area of ​​the operated object based on the captured image of the environment; accepting input of handwritten information for the displayed captured image on an operation terminal that displays the captured image; and estimating a graspable object that is requested to be grasped by the end effector from the graspable object based on the graspable object and the handwritten information, and estimating the operation content requested of the operated object with respect to the graspable object. In the process of estimating the operation content requested of the operated object with respect to the graspable object, Within a range in which either the operated object or the end effector can move When information on the transport path of the grasped object is included, the transport path located in the movable area of ​​the operated object is estimated to be the movement path of the operated object, and the transport path located in the immovable area of ​​the operated object is estimated to be the movement path of the end effector gripping the grasped object. This control program allows the user to perform a desired process, such as having the operated object transport the grasped object, by performing more intuitive handwritten input without having to perform handwritten input while remembering a preset instruction figure. Furthermore, when the handwritten information includes information on the transport path of the grasped object, this control program automatically determines whether the transport path represents the movement path (travel path) of the operated object or the movement path of the end effector. do It is possible. Effect of the Invention

[0019] According to the present disclosure, while allowing intuitive operation by the user, Automatically distinguish whether the transport path information included in the handwritten information is the movement path of the end effector or the movement path of the robot. It is possible to provide a remote control system, a remote control method, and a control program. [Brief description of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of an overall environment in which a remote control system according to a first embodiment is used. [Diagram 2] FIG. 2 is a diagram illustrating an example of a first environment in which a robot exists. [Diagram 3] FIG. 11 is a diagram showing an example of handwritten information. [Figure 4] FIG. 11 is a diagram showing an example of handwritten information. [Diagram 5] FIG. 11 is a diagram showing an example of handwritten information. [Figure 6] FIG. 11 is a diagram showing an example of handwritten information. [Figure 7] FIG. 2 is an external perspective view showing an example of the external configuration of the robot. [Figure 8] FIG. 2 is a block diagram showing an example of a block configuration of a robot. [Figure 9] FIG. 11 is a diagram showing an example of a captured image acquired by a robot. [Figure 10] A figure showing an example of a graspable area output by the first trained model. [Figure 11] FIG. 11 is a diagram showing an example of handwritten information. [Figure 12] FIG. 13 is a diagram showing an example of training data for a second trained model. [Figure 13] FIG. 13 is a diagram showing an example of training data for a second trained model. [Figure 14] FIG. 2 is a block diagram showing an example of a block configuration of a remote terminal; [Figure 15] 4 is a flowchart showing an example of an overall processing flow of the remote control system according to the first embodiment. FIG. [Figure 16] FIG. 2 is a diagram illustrating an example of a first environment in which a robot exists. [Figure 17] 11 is a diagram for explaining a flow of input of handwritten information for a captured image displayed on a remote terminal. FIG. [Figure 18] 13 is a diagram showing an example of handwritten information finally input to a captured image displayed on a remote terminal. FIG. [Figure 19] 16 is a flowchart showing an example of a detailed flow of the process in step S15 of FIG. 15. [Figure 20]4 is a diagram showing an example of a movable area and an immovable area of ​​a robot recognized by a recognition unit; FIG. [Figure 21] 4 is a diagram showing an example of a movable area and an immovable area of ​​a robot recognized by a recognition unit; FIG. [Figure 22] 11 is a diagram for explaining a method for estimating the robot's operation content using the robot's movable area and immovable area recognized by the recognition unit. FIG. [Figure 23] FIG. 13 is a diagram showing a detour path for an object to be grasped that is transported by a robot. [Figure 24] FIG. 11 is a diagram showing an example of handwritten information. [Diagram 25] FIG. 13 is a diagram showing an example in which a plurality of pieces of handwritten information are input; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. Also, in the following embodiments, a robot having a hand at the tip of an arm as an end effector will be described as an example of an operated object, but the operated object is not limited to this.

[0022] <Embodiment 1> 1 is a conceptual diagram showing an example of an overall environment in which a remote operation system 10 according to embodiment 1 is used. A robot 100 that performs various actions in a first environment is remotely controlled by a user who is a remote operator in a second environment away from the first environment, by operating a remote terminal 300 (operation terminal) via a system server 500 connected to the Internet 600.

[0023] In a first environment, the robot 100 is connected to the Internet 600 via a wireless router 700. In a second environment, the remote terminal 300 is connected to the Internet 600 via the wireless router 700. A system server 500 is connected to the Internet 600. In accordance with the operation of the remote terminal 300 by a user, the robot 100 performs an operation of grasping an object to be grasped with the hand 124, an operation of transporting the object to be grasped from its current location to a destination, and the like.

[0024] In this embodiment, the gripping operation of the gripping object by the hand 124 is not limited to the operation of simply gripping (grabbing) the gripping object, but also includes, for example, the following operations. - Grasping and lifting an object If the object to be grasped is the knob of a door or drawer of a dresser, etc., the action of grabbing the knob to open or close the door or drawer If the object to be grasped is a doorknob, the action of grabbing the doorknob and opening or closing the door

[0025] In addition, in this embodiment, the transport operation of the grasped object from its current location to the destination includes an operation of transporting the grasped object by moving the robot 100, and an operation of transporting the grasped object grasped by the hand 124, for example, by operating the arm, without moving the robot 100.

[0026] The robot 100 uses the stereo camera 131 (imaging unit) to capture an image of the first environment in which the robot 100 exists, and transmits the captured image to the remote terminal 300 via the Internet 600. The robot 100 also recognizes a graspable object that the hand 124 can grasp, based on the captured image.

[0027] Fig. 2 is a diagram showing an example of a first environment in which the robot 100 exists. In the example of Fig. 2, the first environment includes a table 400, chests of drawers 410, 420, and 430, and a door 440. Furthermore, the graspable objects that exist in the first environment are objects 401 and 402 placed on the table 400, knobs 411 of the chest of drawers 410, knobs 421 and 422 of the chest of drawers 420, knobs 431 and 432 of the chest of drawers 430, and a doorknob 441 of the door 440.

[0028] The remote terminal 300 is, for example, a tablet terminal, and has a display panel 341 on which a touch panel is superimposed. The captured image received from the robot 100 is displayed on the display panel 341, and the user can indirectly view the first environment in which the robot 100 exists. In addition, the user can handwrite handwritten information, which is an image (first image) simulating the operation of the robot 100 with respect to a grasp target, on the captured image displayed on the display panel 341.

[0029] A method of inputting handwritten information includes, but is not limited to, a method of touching a corresponding part of a captured image on a touch panel arranged so as to be superimposed on the display panel 341 using a user's finger or a touch pen. FIGS. 3 to 6 are diagrams showing examples of handwritten information input to a captured image 310. The example in FIG. 3 shows handwritten information 901 simulating grabbing a rectangular parallelepiped object 401 placed on a table 400 from above. The example in FIG. 4 shows handwritten information 902 simulating grabbing a cylindrical object 402 placed on the table 400 from the side. The example in FIG. 5 shows handwritten information 903 simulating grabbing a knob 411 of a chest of drawers 410 to open the door. The example in FIG. 6 shows handwritten information 904 simulating grabbing a cylindrical object 402 placed on the table 400 from the side and moving it to another position on the table 400. The handwritten information 904 is composed of handwritten information 904a simulating grasping the cylindrical object 402 from the side, handwritten information 904b simulating the transport path of the object 402, and handwritten information 904c simulating the destination (destination) of the object 402. As shown in Figs. 3 to 6, the image of the handwritten information may be an image consisting of only figures such as lines, or an image consisting of a combination of figures such as lines and characters. The handwritten information input by the user for the captured image is transmitted to the robot 100 via the Internet 600.

[0030] Note that the remote terminal 300 is not limited to a tablet terminal, and the handwritten information input to the remote terminal 300 is not limited to being input by touching a touch panel with a user's finger, a touch pen, etc. For example, the remote terminal 300 may be VR (Virtual Reality) goggles, and the handwritten information input to the remote terminal 300 may be input to a captured image displayed on the VR goggles by a user wearing the VR goggles operating a VR controller.

[0031] Based on the graspable objects recognized from the captured image and the handwritten information input by the user for the captured image, the robot 100 estimates the graspable object that is requested to be grasped by the hand 124 from among the graspable objects, and estimates the operation content of the graspable object that is requested of the robot 100. Note that the operation content of the graspable object that is requested of the robot 100 includes, for example, how to grasp the graspable object by the hand 124 and how to transport the graspable object from its current location to its destination.

[0032] FIG. 7 is an external perspective view showing an example of the external configuration of the robot 100. The robot 100 is broadly composed of a cart unit 110 and a main body unit 120. The cart unit 110 supports two drive wheels 111 and one caster 112, each of which is in contact with a running surface, in a cylindrical housing. The two drive wheels 111 are arranged so that their rotational axes coincide with each other. Each drive wheel 111 is independently rotated by a motor (not shown). The caster 112 is a driven wheel, and is provided such that a rotating shaft extending vertically from the cart unit 110 is separated from the rotational axis of the wheel to support the wheel, and follows the moving direction of the cart unit 110.

[0033] The cart unit 110 is provided with a laser scanner 133 on the periphery of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane for each step angle, and outputs whether or not an obstacle exists in each direction. Furthermore, if an obstacle exists, the laser scanner 133 outputs the distance to the obstacle.

[0034] The main body 120 mainly includes a trunk 121 mounted on the upper surface of the cart 110, a head 122 placed on the upper surface of the trunk 121, an arm 123 supported on the side of the trunk 121, and a hand 124 installed at the tip of the arm 123. The arm 123 and the hand 124 are driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 can rotate around a vertical axis relative to the cart 110 by the driving force of the motor (not shown).

[0035] Head 122 mainly includes stereo camera 131 and display panel 141. Stereo camera 131 has a configuration in which two camera units having the same angle of view are arranged apart from each other, and outputs image signals captured by each camera unit.

[0036] The display panel 141 is, for example, a liquid crystal panel, and displays the face of a set character as an animation, and displays information about the robot 100 as text or icons. If a character's face is displayed on the display panel 141, it is possible to give people around the display panel 141 the impression that the display panel 141 is a pseudo-face.

[0037] Head 122 can rotate about a vertical axis relative to torso 121 by the driving force of a motor (not shown). Therefore, stereo camera 131 can capture images in any direction, and display panel 141 can present display content in any direction.

[0038] 8 is a block diagram showing an example of a block configuration of the robot 100. Here, main elements related to estimation of a grasp target object and estimation of an action to be taken by the robot 100 with respect to the grasp target object will be described, but the configuration of the robot 100 also includes other elements, and other elements that contribute to estimation of a grasp target object and estimation of an action to be taken by the robot 100 with respect to the grasp target object may be added.

[0039] The control unit 150 is, for example, a CPU, and is stored in a control unit provided in, for example, the body 121. The cart drive unit 145 includes the drive wheels 111, and a drive circuit and a motor for driving the drive wheels 111. The control unit 150 executes rotation control of the drive wheels by sending a drive signal to the cart drive unit 145. The control unit 150 also receives a feedback signal from an encoder or the like from the cart drive unit 145 to grasp the moving direction and moving speed of the cart unit 110.

[0040] Upper body drive unit 146 includes arm 123, hand 124, torso 121, head 122, and drive circuits and motors for driving these. Control unit 150 realizes gripping motions and gestures by sending drive signals to upper body drive unit 146. Control unit 150 also receives feedback signals from an encoder or the like from upper body drive unit 146 to grasp the positions and movement speeds of arm 123 and hand 124, and the orientations and rotation speeds of torso 121 and head 122.

[0041] The display panel 141 receives and displays the image signal generated by the control unit 150. In addition, the control unit 150 generates image signals of characters and the like and causes the display panel 141 to display them, as described above.

[0042] The stereo camera 131 captures an image of the first environment in which the robot 100 exists in response to a request from the control unit 150, and transfers the captured image signal to the control unit 150. The control unit 150 performs image processing using the captured image signal, and converts the captured image signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether or not an obstacle exists in the moving direction in response to a request from the control unit 150, and transfers a detection signal that is the detection result to the control unit 150.

[0043] The hand camera 135 is, for example, a distance image sensor, and is used to recognize the distance, shape, direction, etc. of the object to be grasped. The hand camera 135 includes an image sensor in which pixels that photoelectrically convert an optical image incident from the target space are arranged two-dimensionally, and outputs the distance to the object for each pixel to the control unit 150. Specifically, the hand camera 135 includes an irradiation unit that irradiates the target space with pattern light, receives the reflected light with the image sensor, and outputs the distance to the object captured by each pixel from the distortion and size of the pattern in the image. The control unit 150 grasps the state of a wider surrounding environment with the stereo camera 131, and grasps the state near the object to be grasped with the hand camera 135.

[0044] The memory 180 is a non-volatile storage medium, and for example, a solid state drive is used. The memory 180 stores various parameter values, functions, lookup tables, etc. used for control and calculation, in addition to a control program for controlling the robot 100. In particular, the memory 180 stores a first trained model 181 that takes a captured image as an input image and outputs a graspable object shown in the captured image, a second trained model 182 that takes an image of handwritten information as an input image and outputs the meaning of the operation content of the robot 100 simulated by the handwritten information, and a third trained model 183 that takes a captured image as an input image and outputs a movable area and an immovable area of ​​the robot 100 shown in the captured image.

[0045] The communication unit 190 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 190 receives handwritten information sent from the remote terminal 300 and passes it to the control unit 150. The communication unit 190 also transmits images captured by the stereo camera 131 to the remote terminal 300 under the control of the control unit 150.

[0046] The control unit 150 executes the control program read from the memory 180 to control the entire robot 100 and various arithmetic processing. The control unit 150 also serves as a function execution unit that executes various calculations and controls related to the control. As such function execution units, the control unit 150 includes a recognition unit 151, an estimation unit 152, and an image adjustment unit 153.

[0047] (Details of the recognition unit 151) The recognition unit 151 mainly extracts a graspable area that can be grasped by the hand 124 and is shown in the captured image captured by the stereo camera 131, and recognizes it as a graspable object. This will be specifically described below.

[0048] First, the recognition unit 151 uses the first trained model 181 read out from the memory 180 to extract a graspable area that can be grasped by the hand 124 and is shown in the image captured by the stereo camera 131, and recognizes it as a graspable object.

[0049] Fig. 9 is a diagram showing an example of a captured image 310 of the first environment captured by the robot 100 using the stereo camera 131. A chest of drawers 410 having a knob 411 and a chest of drawers 420 having knobs 421 and 422 are shown in the captured image 310 in Fig. 9. The recognition unit 151 provides the captured image 310 to the first trained model 181 as an input image.

[0050] Fig. 10 is a diagram showing an example of a graspable area output by first trained model 181 when captured image 310 in Fig. 9 is used as an input image. Specifically, an area surrounding knob 411 is detected as graspable area 801, an area surrounding knob 421 as graspable area 802, and an area surrounding knob 422 as graspable area 803. Therefore, recognition unit 151 recognizes knobs 411, 421, and 422, which are respectively surrounded by graspable areas 801, 802, and 803, as graspable objects.

[0051] The first trained model 181 is a neural network trained using training data that is a combination of an image showing a graspable object that can be grasped by the hand 124 and a correct answer value for which area of ​​the image is the graspable object. In this case, the first trained model 181 becomes a trained model that can extract not only the graspable object but also information about the distance and direction of the graspable object from the captured image by training using training data that includes information about the distance and direction of the graspable object in the image. The first trained model 181 may be a neural network trained using deep learning. Furthermore, the first trained model 181 may be trained by adding training data as needed.

[0052] Furthermore, the recognition unit 151 not only recognizes a graspable object, but also recognizes a movable area and an immovable area of ​​the robot 100. The flow of the recognition process of the recognition unit 151 for the movable area and the immovable area of ​​the robot 100 is basically the same as the recognition process for a graspable object.

[0053] That is, the recognition unit 151 extracts the movable area and the immovable area of ​​the robot 100 shown in the captured image captured by the stereo camera 131, and recognizes them, using the third trained model 183 read from the memory 180. Here, the third trained model 183 is a neural network that has learned using training data that is a combination of an image showing the movable area of ​​the robot 100 and a correct answer value as to which area of ​​the image is the movable area of ​​the robot 100, and has also learned using training data that is a combination of an image showing the immovable area of ​​the robot 100 and a correct answer value as to which area of ​​the image is the immovable area of ​​the robot 100.

[0054] (Details of the estimation unit 152) The estimation unit 152 estimates a graspable object that is requested to be grasped by the hand 124 from among the graspable objects recognized by the recognition unit 151, and estimates the operation content for the graspable object that is requested of the robot 100. A specific description will be given below.

[0055] First, the estimation unit 152 estimates a graspable object that is requested to be grasped by the hand 124 from among the graspable objects recognized by the recognition unit 151, based on the graspable objects recognized by the recognition unit 151 from the captured image and the handwritten information input by the user for the captured image.

[0056] FIG. 11 is a diagram showing an example of handwritten information input by a user to the captured image 310 of FIG. 9 in the remote terminal 300. In the example of FIG. 11, handwritten information 905 is input at the position of the knob 411 on the captured image 310. Therefore, the estimation unit 152 estimates that the object to be grasped that is requested to be grasped by the hand 124 is the knob 411 among the knobs 411, 421, and 422 recognized by the recognition unit 151 as graspable objects. Note that the estimation unit 152 can recognize the input position of the handwritten information 905 on the captured image 310 by any method. For example, if the remote terminal 300 transmits position information indicating the input position of the handwritten information 905 on the captured image 310 by including it in the handwritten information 905, the estimation unit 152 can recognize the input position of the handwritten information 905 based on the position information. Alternatively, if the remote terminal 300 transmits a captured image 310 that has been processed to include handwritten information 905, the estimation unit 152 can recognize the input position of the handwritten information 905 based on the captured image 310.

[0057] Furthermore, the estimation unit 152 uses the second trained model 182 read from the memory 180 to obtain the meaning of the operation content of the robot 100 simulated by the handwritten information input by the user to the captured image from the image of the handwritten information, and estimates the operation content required of the robot 100 with respect to the object to be grasped.

[0058] The second trained model 182 is a neural network that has been trained using training data that is a combination of an image of handwritten information and the meaning of the operation content of the robot 100 that the handwritten information simulates. FIG. 12 is a diagram showing an example of training data for the second trained model 182. The example of FIG. 12 is training data for making the second trained model 182 learn three images that mean the gripping action of "grab", four images that mean the gripping action of "open", and four images that mean the transport action of "transport". In addition, the second trained model 182 may learn more detailed training data compared to the training data of FIG. 12. FIG. 13 is a diagram showing an example of training data for making the second trained model 182 learn the gripping action of "grab" in more detail. The example of FIG. 13 is training data for making the second trained model 182 learn images that mean the gripping actions of "grab from above", "grab from the side", and "grab from diagonally above". The second trained model 182 may be a neural network trained by deep learning. The second trained model 182 may be additionally trained by adding teacher data as needed.

[0059] In the example of FIG. 11, the estimation unit 152 estimates handwritten information 90 from the second trained model 182. 5 means the grasping action of "open." Therefore, the estimation unit 152 estimates that the action content requested of the robot 100 with respect to the object to be grasped is the action content of grabbing the knob 411, which is the object to be grasped, and opening the door.

[0060] A more detailed estimation method by the estimation unit 152 of the action content required of the robot 100 with respect to the object to be grasped, using the movable and immovable areas of the robot 100 recognized by the recognition unit 151, will be described later.

[0061] (Overview of image adjustment unit 153) Image adjustment unit 153 changes the imaging direction and imaging range of stereo camera 131 in accordance with handwritten information input by the user to the captured image displayed on display panel 341 of remote terminal 300, and causes the captured image captured by stereo camera 131 to be displayed on display panel 341 of remote terminal 300. Details of image adjustment unit 153 will be described later.

[0062] In this manner, the estimation unit 152 in the control unit 150 can estimate the object to be grasped that is requested to be grasped by the hand 124 and the operation content for the object to be grasped that is requested of the robot 100. Depending on the learning content of the first trained model 181, the control unit 150 can obtain the distance and direction of the object to be grasped from the captured image acquired by the stereo camera 131. The control unit 150 may obtain the distance and direction of the object to be grasped by performing image analysis on the captured image of the first environment, or may obtain the distance and direction from the detection result by another sensor. The control unit 150 can also detect whether or not an obstacle exists in the moving direction of the robot 100 based on a detection signal from the laser scanner 133.

[0063] Therefore, the control unit 150 generates a path for the robot 100 to move from the current position to the vicinity of the object to be grasped while avoiding the obstacles, based on the distance and direction of the object to be grasped, and the presence or absence of an obstacle, and transmits a drive signal according to the generated path to the cart drive unit 145. The cart drive unit 145 moves the robot 100 to the vicinity of the object to be grasped in response to the drive signal.

[0064] When the robot 100 moves near the object to be grasped, the control unit 150 prepares to start an operation on the object to be grasped that is requested of the robot 100. Specifically, the control unit 150 first drives the arm 123 to a position where the hand camera 135 can observe the object to be grasped. Next, the control unit 150 causes the hand camera 135 to capture an image of the object to be grasped, and recognizes the state of the object to be grasped.

[0065] Then, based on the state of the object to be grasped and the action requested of the robot 100 with respect to the object to be grasped, the control unit 150 determines the details of the action of each part of the robot 100 to realize the action requested of the robot 100 with respect to the object to be grasped.

[0066] For example, the control unit 150 generates a trajectory of the hand 124 for realizing a gripping operation of the hand 124 to grip an object to be gripped. At this time, the control unit 150 generates the trajectory of the hand 124 so as to satisfy a predetermined gripping condition. The predetermined gripping condition includes a condition when the hand 124 grips an object to be gripped, a condition of a trajectory until the hand 124 grips the object to be gripped, and the like. An example of a condition when the hand 124 grips an object to be gripped is that the arm 123 is not extended too much when the hand 124 grips the object to be gripped. Also, an example of a condition of a trajectory until the hand 124 grips an object to be gripped is that the hand 124 follows a straight trajectory when the object to be gripped is a drawer knob.

[0067] When the control unit 150 generates the trajectory of the hand 124, it transmits a drive signal corresponding to the generated trajectory to the upper body drive unit 146. The hand 124 performs a gripping operation on the gripping object in response to the drive signal.

[0068] Alternatively, the control unit 150 generates a trajectory of the hand 124 and a movement path (travel path) of the robot 100 to realize a conveying operation of the object to be grasped by the robot 100. In this case, the control unit 150 transmits a drive signal corresponding to the generated trajectory of the hand 124 to the upper body drive unit 146, and transmits a drive signal corresponding to the generated movement path of the robot 100 to the cart drive unit 145. The hand 124 and the robot 100 perform a conveying operation of the object to be grasped in response to these drive signals.

[0069] 14 is a block diagram showing an example of the block configuration of the remote terminal 300. Here, the main elements related to the process of inputting handwritten information to the captured image received from the robot 100 will be described, but the remote terminal 300 may also include other elements, and other elements that contribute to the process of inputting handwritten information may also be added.

[0070] The calculation unit 350 is, for example, a CPU, and executes various calculation processes and controls the entire remote terminal 300 by executing a control program read from the memory 380. The display panel 341 is, for example, a liquid crystal panel, and displays, for example, a captured image sent from the robot 100.

[0071] The input unit 342 is a display panel 3 A touch panel or a display panel arranged superimposed on 41 3 The input unit 342 includes a push button etc. provided on the periphery of the input unit 341. The input unit 342 delivers handwritten information, which is an image simulating the operation of the robot 100 with respect to the object to be grasped, input by the user touching the touch panel with a finger or a touch pen, to the calculation unit 350. Examples of the handwritten information are as shown in Figs. 3 to 6.

[0072] Memory 380 is a non-volatile storage medium, such as a solid state drive, and stores a control program for controlling remote terminal 300, as well as various parameter values, functions, lookup tables, and the like used for control and calculation.

[0073] The communication unit 390 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 390 receives captured images sent from the robot 100 and passes them to the calculation unit 350. The communication unit 390 also cooperates with the calculation unit 350 to transmit handwritten information to the robot 100.

[0074] Next, the overall processing of the remote control system 10 according to this embodiment will be described with reference to FIGS.

[0075] Fig. 15 is a flow diagram showing an example of the overall processing flow of the remote operation system 10 according to this embodiment. In Fig. 15, the flow on the left side represents the processing flow of the robot 100, and the flow on the right side represents the processing flow of the remote terminal 300. Also, the dotted arrows represent the transfer of handwritten information and captured images between the robot 100 and the remote terminal 300 via the system server 500.

[0076] FIG. 16 is a diagram showing an example of a first environment in which the robot 100 exists. In the example of FIG. 16, the robot 100 (not shown) is assumed to be placed in an area facing the paper surface in the first environment. Therefore, in the example of FIG. 16, the robot 100 can take an image of the first predetermined area A1 and an image of the second predetermined area A2 using the stereo camera 131. In the following, a case will be described in which a user remotely controls the robot 100 to transport an object 402 placed on a table 400 onto a chest of drawers 420.

[0077] 17 is a diagram for explaining a flow of input of handwritten information for a captured image displayed on the remote terminal 300. Furthermore, FIG. 18 is a diagram showing an example of handwritten information finally input for a captured image displayed on the remote terminal 300.

[0078] In the robot 100, the control unit 150 uses the stereo camera 131 to capture an image of a first predetermined area A1 of a first environment in which the robot 100 exists (step S11), and transmits the captured image to the remote terminal 300 via the communication unit 190 (step S12).

[0079] In the remote terminal 300, when the calculation unit 350 receives the captured image of the first predetermined area A1 transmitted from the robot 100, the calculation unit 350 displays the received captured image on the display panel 341. Thereafter, the calculation unit 350 transitions to a state in which input of handwritten information for the captured image is accepted (step S31). After that, the user starts inputting handwritten information for the captured image via the input unit 342, which is a touch panel (Yes in step S31).

[0080] Here, if a portion of the captured image displayed on the display panel 341 that is specified in the handwritten information is a predetermined peripheral area 311 of the captured image displayed on the display panel 341 (Yes in step S32), the calculation unit 350 transmits the handwritten information to the robot 100 via the communication unit 390 (step S33).

[0081] 17(a), in the captured image of the predetermined area A1 displayed on the display panel 341, a part of the transport path of the grasp target 402 represented in the handwritten information 906 is the predetermined outer peripheral area 311 of the captured image displayed on the display panel 341 (Yes in step S32). In this case, the calculation unit 350 transmits the handwritten information 906 to the robot 100 via the communication unit 390 (step S33).

[0082] At this time, in the robot 100, the image adjustment unit 153 changes the imaging area from the first predetermined area A1 to the second predetermined area A2 using the stereo camera 131 and captures the image (step S13). The captured image is transmitted to the remote terminal 300 via the communication unit 190 (step S14). Here, the image adjustment unit 153 adjusts the imaging direction, imaging range, etc. of the stereo camera 131 so that the second predetermined area A2 is an area that includes at least the partial area specified in the handwritten information and an area adjacent to the first predetermined area A1. The second predetermined area A2 may include all or a part of the first predetermined area A1.

[0083] In the remote terminal 300, when the calculation unit 350 receives the captured image of the second predetermined area A2 transmitted from the robot 100, the calculation unit 350 displays the received captured image on the display panel 341. That is, the calculation unit 350 switches the captured image displayed on the display panel 341 from the captured image of the first predetermined area A1 to the captured image of the second predetermined area A2.

[0084] 17(b), the captured image displayed on the display panel 341 is switched from the captured image of the first predetermined area A1 to the captured image of the second predetermined area A2. The captured image of the second predetermined area A2 is an captured image of an area including at least an area P1 that is a part of the transport path of the grasp target 402 represented in the handwritten information 906 and an area adjacent to the first predetermined area A1.

[0085] In this way, even if a part of the captured image specified in the handwritten information is an area with low visibility such as an edge area of ​​the captured image, the captured image displayed on the display panel 341 switches in accordance with the area specified in the handwritten information, so that high visibility is maintained. As a result, the user can input handwritten information over a wide range, for example, from the first predetermined area A1 to the second predetermined area A2.

[0086] In the example of FIG. 17(c), the first predetermined area A1 is continuously followed by the second predetermined area A2. Handwritten information 906 has been entered.

[0087] In addition, if the area of ​​the captured image displayed on the display panel 341 specified in the handwritten information (for example, the transport path of the object to be grasped indicated in the handwritten information) is not the specified peripheral area 311 of the captured image displayed on the display panel 341 (No in step S32), the captured image displayed on the display panel 341 will not be switched.

[0088] Thereafter, when the user wishes to end the input of handwritten information into the captured image, the user performs a predetermined handwriting end operation. For example, when the user wishes to end the input of handwritten information into the captured image, the user touches any location on the touch panel twice in succession in a manner similar to double-clicking. Alternatively, the user continues to touch the same location on the touch panel for a predetermined period of time or more. Alternatively, when operation buttons are displayed on the display panel 341, the user touches an area of ​​the touch panel above the button that ends the input of handwritten information.

[0089] In the example of FIG. 17(d), the user touches an arbitrary location on the touch panel twice in succession in a double-click manner as a predetermined handwriting end operation, thereby inputting handwritten information 907 into the captured image.

[0090] As a result, the calculation unit 350 stops accepting input of handwritten information for the captured image (Yes in step S34), and transmits handwritten information 906 as shown in FIG. 18 to the robot 100 via the communication unit 390 (step S35).

[0091] In the robot 100, when the recognition unit 151 receives the handwritten information input for the captured image transmitted from the remote terminal 300, it recognizes a graspable object based on the captured image. The estimation unit 152 estimates a graspable object that is requested to be grasped by the hand 124 from among the graspable objects recognized by the recognition unit 151, based on the graspable object recognized from the captured image by the recognition unit 151 and the handwritten information input for the captured image, and estimates the operation content for the graspable object that is requested of the robot 100 (step S15).

[0092] In the example of Figure 18, the estimation unit 152 estimates that the object 402 is the object to be grasped from among the multiple graspable objects recognized by the recognition unit 151, and estimates that the action requested of the robot 100 with respect to the object to be grasped is to transport the object 402, which is the object to be grasped and is placed on the table 400, to the top of the chest of drawers 420.

[0093] Thereafter, in the robot 100, the control unit 150 controls the cart drive unit 145 to move the robot 100 to the vicinity of the object to be grasped, and at the point when the robot 100 moves to the vicinity of the object to be grasped, determines the details of the operation of each part of the robot 100 to realize the operation on the object to be grasped that is required of the robot 100.

[0094] For example, the control unit 150 generates a trajectory of the hand 124 for realizing a gripping operation of the gripping object by the hand 124 (step S16). After generating the trajectory of the hand 124, the control unit 150 controls the upper body drive unit 146 according to the generated trajectory. As a result, the gripping operation of the gripping object by the hand 124 is performed (step S17).

[0095] Alternatively, the control unit 150 generates a trajectory of the hand 124 and a movement path (travel path) of the robot 100 for realizing a conveying operation of the object to be grasped by the robot 100 (step S16). After generating the trajectory of the hand 124 and the movement path of the robot 100, the control unit 150 controls the upper body drive unit 146 and the carriage drive unit 145 in accordance with the information. Thereby, a conveying operation of the object to be grasped by the hand 124 and the robot 100 is performed (step S17). In the example of FIG. 18 as well, a conveying operation of the object to be grasped by the hand 124 and the robot 100 is performed.

[0096] Next, the process of step S15 in Fig. 15 will be described in detail with reference to Fig. 19. Fig. 19 is a flow diagram showing an example of a detailed flow of the process of step S15 in Fig. 15 by the robot 100.

[0097] First, when the recognition unit 151 receives handwritten information input for the captured image transmitted from the remote terminal 300, it uses the first trained model 181 read from the memory 180 to extract the graspable area shown in the captured image and recognizes it as a graspable object (step S151).

[0098] Next, the estimation unit 152 estimates a graspable object that is requested to be grasped by the hand 124 from among the graspable objects recognized by the recognition unit 151 based on the input position of the handwritten information on the captured image (step S152). Note that the input position of the handwritten information on the captured image is recognized using, for example, the method already described.

[0099] Next, the recognition unit 151 extracts the movable area and the immovable area of ​​the robot 100 shown in the captured image to which the handwritten information has been input, using the third trained model 183 read from the memory 180, and recognizes them (step S153). Note that the recognition process (segmentation process) of the movable area and the immovable area of ​​the robot 100 by the recognition unit 151 may be performed simultaneously with the recognition process of the graspable object.

[0100] Fig. 20 is a diagram showing an example of a movable area and an immovable area of ​​the robot 100 recognized by the recognition unit 151. A captured image 310 shown in Fig. 20 is an example in which the recognition unit 151 performs a recognition process on the movable area and the immovable area of ​​the robot 100 for the captured image 310 shown in Fig. 6. In the captured image 310 shown in Fig. 20, a floor 450 is recognized as the movable area of ​​the robot 100, and a table 400 is recognized as the immovable area of ​​the robot 100. Note that in the example of Fig. 20, the object to be grasped is estimated to be an object 402.

[0101] 21 is a diagram showing another example of the movable area and immovable area of ​​the robot 100 recognized by the recognition unit 151. The captured image shown in FIG. 21 is an example in which the recognition unit 151 has performed a recognition process on the movable area and immovable area of ​​the robot 100 for the captured image shown in FIG. 18. In the captured image shown in FIG. 21, a floor 450 is recognized as the movable area of ​​the robot 100, and a table 400 and a chest of drawers 420 are recognized as the immovable areas of the robot 100. Also, an object 40 to be grasped is shown. 2 Other graspable objects (e.g. object 40 121, the object to be grasped is estimated to be an object 402.

[0102] Next, the estimation unit 152 uses the second trained model 182 read from the memory 180 to obtain the meaning of the action content of the robot 100 simulated by the handwritten information from the image of the handwritten information, and estimates the action content required of the robot 100 with respect to the object to be grasped (step S154).

[0103] For example, in the example of Fig. 20, the estimation unit 152 estimates from the image of the handwritten information 904 that the operation content required of the robot 100 is to transport the object 402 to be grasped, which is represented by the handwritten information 904a, to the destination represented by the handwritten information 904c along the transport path represented by the handwritten information 904b. Here, in the example of Fig. 20, all of the transport paths of the object 402 represented by the handwritten information 904 exist in an area where the robot 100 cannot move. In this case, the estimation unit 152 estimates that all of the transport paths of the object 402 represented by the handwritten information 904 are not the travel paths of the robot 100, but the movement paths of the hand 124 attached to the tip of the arm of the robot 100 by the arm operation.

[0104] 21, the estimation unit 152 estimates from the image of the handwritten information 906 that the operation content required of the robot 100 is to transport the object 402 to be grasped, which is represented by the handwritten information 906a, along the transport path represented by the handwritten information 906b to the destination represented by the handwritten information 906c. Here, in the example of FIG. 21, a part of the transport path of the object 402 represented by the handwritten information 906 exists in the movable area of ​​the robot 100, and another part exists in the movable area of ​​the robot 100. Specifically, as shown in FIG. 22, among the transport paths represented by the handwritten information 906b, the transport path R2 exists in the movable area of ​​the robot 100, and the transport paths R1 and R3 exist in the movable area of ​​the robot 100. In this case, the estimation unit 152 estimates that, among the transport paths represented by the handwritten information 906b, the transport path R2 located in the movable area of ​​the robot 100 is the movement path (travel path) of the robot 100, and estimates that the transport paths R1, R3 located in the immovable area of ​​the robot 100 are the movement paths by arm operation of the hand 124 attached to the tip of the arm of the robot 100.

[0105] In addition, when an unmovable area sandwiched between two movable areas exists on the conveying route represented by the handwritten information, the estimation unit 152 may estimate a detour route for the gripping object such that there is no unmovable area sandwiched between the two movable areas. The robot 100 has an output unit that outputs information on this detour route, and may display the information on the display panel 341 of the remote terminal 300, for example. In the example of FIG. 23, an unmovable area (obstacle 451) sandwiched between two movable areas exists on the conveying route represented by the handwritten information 906. In this case, the estimation unit 152 may estimate, for example, a detour route 916 that avoids the obstacle 451 and propose it to the user.

[0106] As described above, in the remote operation system 10 according to this embodiment, in the robot 100, the recognition unit 151 recognizes a graspable object that the hand 124 can grasp, based on a captured image of the environment in which the robot 100 exists. In addition, the estimation unit 152 estimates a graspable object that is requested to be grasped by the hand 124 from among the graspable objects, based on the graspable object recognized by the recognition unit 151 from the captured image and handwritten information input by the user for the captured image, and estimates the operation content (grasping operation or transport operation) of the robot 100 with respect to the graspable object.

[0107] As a result, in the remote control system 10 according to this embodiment, the user can make the robot 100 perform a desired process, such as transporting an object to be grasped, by making a more intuitive handwritten input, without having to remember a pre-set instruction figure and making the handwritten input.

[0108] In addition, in the remote control system 10 according to the present embodiment, the captured image displayed on the display panel 341 of the remote terminal 300 is switched to follow the area specified in the handwritten information, so that high visibility is maintained and, for example, handwritten information can be input over a wide range. In other words, the remote control system 10 can improve the convenience of operation.

[0109] Furthermore, in the remote control system 10 according to the present embodiment, when handwritten information includes information on a transport path of an object to be grasped, it is automatically determined whether the transport path represents a moving path (traveling path) of the robot 100 or a moving path of the hand 124. do It is possible.

[0110] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention.

[0111] In the above embodiment, as an example, the handwritten information is an image simulating the manner of the gripping operation on the gripping object requested by the hand 124, but the present invention is not limited to this. The handwritten information may further include an image indicating the degree of the gripping operation. In this case, the estimation unit 152 may further estimate the degree of the gripping operation on the gripping object by the hand 124 based on the handwritten information. FIG. 24 is a diagram showing an example of handwritten information including an image indicating the degree of the gripping operation. The example of FIG. 24 shows handwritten information 908 in which an image of "30°" indicating the degree of the gripping operation is added to an image similar to the handwritten information 903 in FIG. 5. In the example of FIG. 24, the estimation unit 152 estimates that the gripping operation on the knob 411 requested by the hand 124 is an operation of grabbing the knob 411 and opening the door by 30°. This allows the user to perform more detailed and intuitive operations.

[0112] In the above embodiment, an example in which one piece of handwritten information is input to a captured image has been described, but the present invention is not limited to this. A plurality of pieces of handwritten information may be input to a captured image. When a plurality of pieces of handwritten information are input to a captured image, the estimation unit 152 estimates the motion content of the object to be grasped and the robot for each piece of handwritten information. At this time, the estimation unit 152 may estimate that the motion of the robot with respect to the object to be grasped estimated by the handwritten information input earlier is performed preferentially. Alternatively, each of the plurality of pieces of handwritten information may further include an image indicating the order of the gripping motion. In this case, the estimation unit 152 further estimates the order of the gripping motion based on the handwritten information. FIG. 25 is a diagram showing an example in which a plurality of pieces of handwritten information including an image indicating the order of the gripping motion are input. The example of FIG. 25 shows an example in which two pieces of handwritten information, namely handwritten information 909 for the knob 422 and handwritten information 910 for the knob 411, are input to the captured image 310. Here, handwritten information 909 includes an image of "1" indicating the order of the gripping action, and handwritten information 910 includes an image of "2" indicating the order of the gripping action. Therefore, in the example of Fig. 25, estimation unit 152 estimates that first, hand 124 will perform a gripping action on knob 422 (an action of grabbing knob 422 and opening a drawer), and second, hand 124 will perform a gripping action on knob 411 (an action of grabbing knob 411 and opening a door).

[0113] In addition, in the above embodiment, three learned models, namely, the first learned model 181, the second learned model 182, and the third learned model 183, are used, but the present invention is not limited to this. For example, instead of the first learned model 181 and the second learned model 182, a transfer learned model in which the output of the first learned model 181 is applied to the second learned model 182 may be used. This transfer learned model is, for example, a model that uses a captured image into which handwritten information is input as an input image, and outputs a graspable object shown in the captured image, a graspable object among the graspable objects, and the meaning of the action content of the robot 100 with respect to the graspable object that is simulated by the handwritten information.

[0114] In the above embodiment, the recognition unit 151, the estimation unit 152, and the image adjustment unit 153 are provided in the robot 100, but this is not limiting. All or a part of the recognition unit 151, the estimation unit 152, and the image adjustment unit 153 may be provided in the remote terminal 300 or in the system server 500.

[0115] In the above embodiment, the robot 100 and the remote terminal 300 exchange captured images and handwritten information via the Internet 600 and the system server 500, but this is not limiting. The robot 100 and the remote terminal 300 may exchange captured images and handwritten information via direct communication.

[0116] In the above embodiment, the imaging unit (stereo camera 131) provided in the robot 100 is used, but the present invention is not limited to this. The imaging unit may be any imaging unit provided at any location in the first environment in which the robot 100 exists. The imaging unit is not limited to a stereo camera, and may be a monocular camera or the like.

[0117] In the above embodiment, the example has been described in which the operated object is the robot 100 equipped with the hand 124 at the tip of the arm 123 as an end effector, but the present invention is not limited to this. The operated object may be anything that is equipped with an end effector and executes a gripping operation using the end effector. The end effector may also be a gripping unit other than a hand (for example, an adsorption unit, etc.).

[0118] Furthermore, in the present disclosure, a part or all of the processing of the remote control system 10 can be realized by causing a CPU (Central Processing Unit) to execute a computer program.

[0119] The above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other memory technology, a CD-ROM, a digital versatile disc (DVD), a Blu-ray® disk or other optical disk storage, a magnetic cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes an electrical, optical, acoustic, or other form of propagating signal. [Explanation of symbols]

[0120] 10 Remote Control System 100 Robots 110 Bogie section 111 Drive wheel 112 Caster 120 Main body 121 Torso 122 Head 123 Arm 124 Hands 131 Stereo Camera 133 Laser Scanner 135 Hand Camera 141 Display Panel 145 Truck drive unit 146 Upper body drive unit 150 Control section 151 Recognition part 152 Estimation Department 153 Image Adjustment Section 180 Memory 181 First trained model 182 Second trained model 183 Third trained model 190 Communication Unit 300 Remote Terminal 310 Captured Images 311 Outer Periphery 341 Display Panel 342 Input Unit 350 Arithmetic unit 380 Memory 390 Communication Unit 400 Tables 401,402 objects 410,420,430 Chest of Drawers 411,421,422,431,432 knob 440 Doors 441 Doorknob 450 beds 451 Obstacles 500 System Server 600 Internet 700 Wireless Router 801~803 Gripable area 901~908 Handwritten information A1 Shooting area A2 shooting area

Claims

1. A remote control system for remotely controlling an object to be operated that has an end effector, An imaging unit that captures an image of an environment in which the target object exists; a recognition unit that recognizes a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an immovable area of ​​the operated object based on a captured image of the environment; an operation terminal that displays the captured image and receives input of handwritten information for the captured image; an estimation unit that estimates a graspable object that is requested to be grasped by the end effector from the graspable object based on the graspable object and the handwritten information, and estimates an operation content requested of the operated object with respect to the graspable object, when the handwritten information includes information on a transport path of the grasped object within a range in which either the operated object or the end effector can move, the estimation unit estimates, among the transport paths, a transport path located in a movable area of ​​the operated object as a transport path of the operated object, and estimates a transport path located in an immovable area of ​​the operated object as a transport path of the end effector gripping the grasped object; Remote control system.

2. the estimation unit, when the handwritten information includes information on a transport route of the object to be grasped and when an immovable area sandwiched between two of the movable areas exists on the transport route, estimates a detour route for the object to be grasped such that the immovable area sandwiched between the two movable areas does not exist; The remote control system includes: an output unit that outputs information about the detour route estimated by the estimation unit, The remote control system according to claim 1 .

3. The output unit causes the operation terminal to display information about the detour route. The remote control system according to claim 2 .

4. The recognition unit recognizes a graspable object other than the grasp target object among the graspable objects as an immovable area of ​​the operated object. The remote control system according to claim 1 .

5. The target object is a robot capable of moving independently. The remote control system according to claim 1 .

6. When the handwritten information includes information on a transport path of the object to be grasped, the estimation unit estimates, among the transport paths, a transport path located in a movable area of ​​the operated object as a transport path of a robot that is the operated object, and estimates, among the transport paths, a transport path located in an immovable area of ​​the operated object as a transport path of the end effector attached to the tip of an arm of the robot by operating the arm. The remote control system according to claim 5.

7. The imaging unit is attached to the target to be operated. The remote control system according to claim 1 .

8. The imaging unit is provided in a location separate from the target to be operated in the environment in which the target to be operated exists. The remote control system according to claim 1 .

9. the operation terminal completes the input of the handwritten information by accepting a predetermined handwriting end operation by a user. The remote control system according to claim 1 .

10. The handwritten information includes a first image simulating the operation content of the operated object with respect to the gripping object, The remote control system according to claim 1 .

11. The estimation unit estimates an action content requested of the operated target with respect to the grasp target from the first image of the handwritten information by using a trained model. The remote control system according to claim 10.

12. A remote control method for a remote control system that remotely controls an object to be operated having an end effector, comprising: Taking an image of an environment in which the target object exists; Recognizing a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an immovable area of ​​the operated object based on a captured image of the environment; accepting input of handwritten information for the displayed captured image in an operation terminal on which the captured image is displayed; based on the graspable object and the handwritten information, estimating a graspable object that is requested to be grasped by the end effector from the graspable object, and estimating an operation content requested of the operated object with respect to the graspable object; In estimating the operation content requested of the operated object with respect to the grasped object, if the handwritten information includes information on a transport path of the grasped object within a range in which either the operated object or the end effector can move, a transport path among the transport paths that is located in a movable area of ​​the operated object is estimated to be a movement path of the operated object, and a transport path that is located in an immovable area of ​​the operated object is estimated to be a movement path of the end effector that has grasped the grasped object. Remote operation method.

13. A control program for causing a computer to execute a remote operation process by a remote operation system that remotely operates an operated object having an end effector, A process of capturing an image of an environment in which the target object exists; A process of recognizing a graspable object that can be grasped by the end effector, a movable area of ​​the operated object, and an immovable area of ​​the operated object based on a captured image of the environment; a process of accepting input of handwritten information for the displayed captured image in an operation terminal that displays the captured image; a process of estimating a graspable object that is requested to be grasped by the end effector from the graspable object based on the graspable object and the handwritten information, and estimating an operation content requested of the operated object with respect to the graspable object; on the computer, In the process of estimating the operation content requested of the operated object with respect to the grasped object, if the handwritten information includes information on a transport path of the grasped object within a range in which either the operated object or the end effector can move, a transport path among the transport paths that is located in a movable area of ​​the operated object is estimated to be a transport path of the operated object, and a transport path that is located in an immovable area of ​​the operated object is estimated to be a movement path of the end effector that has grasped the grasped object. Control program.

Images