Control system, control method, and program

Abstract

To provide a control system, a control method and a program which perform interaction with a user regarding correction of handwriting input information.SOLUTION: A control system 10 determines operation of a robot on the basis of handwriting input information input to an interface, and controls the operation of the robot. The control system 10 includes: a handwriting input information reception part 351 for receiving input of the handwriting input information; a determination part 352 for determining whether or not the handwriting input information needs to be corrected, on the basis of execution possibility or efficiency of operation based on the handwriting input information; and a presentation part 353 for presenting information on the correction of the handwriting input information to a user, when the handwriting input information needs to be corrected.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control system, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a remote control system that controls the operation of a robot based on handwritten input information into an interface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-094605 Summary of the Invention [Problem to be solved by the invention]

[0004] There are cases where the robot cannot be controlled based on handwritten input information due to various factors such as restrictions on the robot's movements and the presence of obstacles. There are also cases where there are more efficient and appropriate movements than those based on handwritten input information. In such cases, the robot's movements as intended by the user may differ from the actual movements of the robot, which may reduce operability for the user.

[0005] The present disclosure has been made to solve such problems, and has an object to provide a control system, a control method, and a program that interact with a user regarding correction of handwritten input information. [Means for solving the problem]

[0006] The control system in this embodiment includes: A control system that determines an action of a robot based on handwritten input information input to an interface and controls the action of the robot, a receiving unit that receives the handwritten input information; a determination unit that determines whether the handwritten input information needs to be corrected based on the feasibility or efficiency of the operation based on the handwritten input information; a presentation unit that presents information regarding the correction of the handwritten input information to a user when the handwritten input information needs to be corrected; Equipped with.

[0007] The handwriting input information may include trajectory information input using a finger, a touch pen, a pointing device, an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device.

[0008] When the handwritten input information needs to be corrected, the presentation unit may correct the handwritten input information and present the corrected handwritten input information to the user.

[0009] When the handwritten input information needs to be corrected, the presentation unit may present to the user information regarding a portion of the robot operating based on the handwritten input information that is interfering with an obstacle.

[0010] the determination unit determines that the handwritten input information needs to be corrected when a part or all of the handwritten input information is included in a predetermined area of ​​an input screen; The handwritten input information accepting unit may display the input screen in such a manner that the predetermined area and an area other than the predetermined area can be distinguished from each other.

[0011] The handwritten input information receiving unit may set a thickness of lines included in the handwritten input information based on a size of the robot when displaying the input handwritten input information.

[0012] The presentation unit may correct a position of the handwritten input information using a result of segmentation based on machine learning.

[0013] The control method in this embodiment is as follows: 1. A control method for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: receiving the handwritten input information; determining whether the handwritten input needs to be modified based on the feasibility or efficiency of an operation based on the handwritten input; if the handwritten input information needs to be corrected, presenting the user with information for correcting the handwritten input information; Includes.

[0014] The program in this embodiment is A program for causing a computer to execute a control method for determining an action of a robot based on handwritten input information inputted to an interface and controlling the action of the robot, comprising: The control method includes: receiving the handwritten input information; determining whether the handwritten input needs to be modified based on the feasibility or efficiency of an operation based on the handwritten input; if the handwritten input information needs to be corrected, presenting the user with information for correcting the handwritten input information; Includes. Effect of the Invention

[0015] The present disclosure makes it possible to provide a control system, a control method, and a program for interacting with a user regarding correction of handwritten input information. [Brief description of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of an overall environment in which a control system according to a first embodiment is used. [Diagram 2] FIG. 11 is a diagram showing an example of handwritten input information. [Diagram 3] FIG. 11 is a diagram showing an example of handwritten input information. [Figure 4] FIG. 2 is an external perspective view showing an example of the external configuration of the robot. [Diagram 5] FIG. 2 is a block diagram showing an example of a block configuration of a robot. [Figure 6] FIG. 2 is a block diagram showing an example of a block configuration of a remote terminal; [Figure 7] FIG. 11 is a diagram for explaining an example of a captured image to be displayed. [Figure 8] FIG. 11 is a diagram for explaining an example of a captured image to be displayed. [Figure 9] FIG. 11 is a diagram for explaining an example of presentation information. [Figure 10] FIG. 11 is a diagram for explaining an example of presentation information. [Figure 11] 11 is a diagram for explaining an example of a method for correcting handwritten input information. FIG. [Figure 12] 11 is a diagram for explaining an example of a method for correcting handwritten input information. FIG. [Figure 13] 11 is a diagram for explaining an example of a method for correcting handwritten input information. FIG. [Figure 14] 5 is a flowchart showing an example of the operation of the control system according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] EMBODIMENT 1 The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0018] 1 is a conceptual diagram showing an example of an overall environment in which a control system 10 according to the embodiment 1 is used. A robot 100 that performs various actions in a first environment is remotely controlled by a user who is a remote operator in a second environment away from the first environment, by operating a remote terminal 300 (operation terminal) via a system server 500 connected to the Internet 600.

[0019] In a first environment, the robot 100 is connected to the Internet 600 via a wireless router 700. In a second environment, the remote terminal 300 is connected to the Internet 600 via a wireless router 700. The robot 100 performs a grasping action with the hand 124 in accordance with the operation of the remote terminal 300.

[0020] The robot 100 captures an image of the first environment in which the robot 100 exists by using the stereo camera 131 (imaging unit) and transmits the captured image to the remote terminal 300 via the Internet 600. The robot 100 also recognizes a graspable object that can be grasped by the hand 124 based on the captured image. In the first environment, a graspable object 401 such as a can exists. Note that the shape of the graspable object 401 is not limited to a cylindrical shape.

[0021] The remote terminal 300 is, for example, a tablet terminal, and has a display panel 341 on which a touch panel is superimposed. The captured image received from the robot 100 is displayed on the display panel 341, and the user can indirectly view the first environment in which the robot 100 exists. The user can also input handwritten input information by handwriting on the captured image displayed on the display panel 341. The handwritten input information indicates, for example, the rotation of the imaging unit, the moving path of the robot 100, the object to be grasped by the hand 124, and the grasping direction. The handwritten input information can be input by touching a corresponding part of the captured image on a touch panel superimposed on the display panel 341 using a user's finger or a touch pen, but is not limited to this. The handwritten input information may include trajectory information input using a finger, a touch pen, or a pointing device (e.g., a mouse). The handwriting input information may also include trajectory information of a part of the user's body (e.g., a finger) input into a three-dimensional input space using an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0022] 2 and 3 are diagrams showing examples of handwritten input information input to the captured image 310. The example of FIG. 2 shows handwritten input information 901 simulating grasping the grasp target 401 from the side. The example of FIG. 3 shows handwritten input information 902 simulating the movement path of the robot 100. Although not shown, handwritten input information simulating the rotation of the imaging unit may be input. For example, when a touch pen or the like is moved up, down, left, or right on the captured image 310, the viewpoint of the imaging unit follows that direction. The robot 100 may change the viewpoint by rotating its head, or may change the viewpoint by turning on the spot. For example, when the imaging unit is rotated in the pan direction, the robot 100 turns on the spot, and when the imaging unit is rotated in the tilt direction, the head of the robot 100 may be tilted. The handwritten input information may include handwritten character information. The handwritten input information input by the user to the captured image is transmitted to the robot 100 via the Internet 600.

[0023] The control system 10 determines the operation of the robot based on the handwritten input information input to the interface, and controls the operation of the robot. The control system 10 determines whether or not the handwritten input information needs to be corrected based on the feasibility or efficiency of the operation based on the handwritten input information. If the handwritten input information needs to be corrected, the control system 10 presents information regarding the correction of the handwritten input information to the user. For example, the control system 10 may present the corrected handwritten input information, or may present information indicating the part that needs to be corrected. The control system 10 may operate the robot 100 in response to a user's confirmation operation after presenting the information, or may present the information after operating the robot 100 based on the corrected handwritten input information.

[0024] FIG. 4 is an external perspective view showing an example of the external configuration of the robot 100. The robot 100 is broadly composed of a cart unit 110 and a main body unit 120. The cart unit 110 supports two drive wheels 111 and one caster 112, each installed on a running surface, in a cylindrical housing. The two drive wheels 111 are arranged so that their rotational axes coincide with each other. Each drive wheel 111 is independently rotated by a motor (not shown). The caster 112 is a driven wheel, and is provided such that a rotating shaft extending vertically from the cart unit 110 is separated from the rotational axis of the wheel to support the wheel, and follows the moving direction of the cart unit 110.

[0025] The cart unit 110 is provided with a laser scanner 133 on the periphery of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane for each step angle, and outputs whether or not an obstacle exists in each direction. Furthermore, if an obstacle exists, the laser scanner 133 outputs the distance to the obstacle.

[0026] The main body 120 mainly includes a trunk 121 mounted on the upper surface of the cart 110, a head 122 placed on the upper surface of the trunk 121, an arm 123 supported on the side of the trunk 121, and a hand 124 installed at the tip of the arm 123. The arm 123 and the hand 124 are driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 is driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 can rotate around a vertical axis relative to the cart 110 by the driving force of the motor (not shown).

[0027] Head 122 mainly includes stereo camera 131 and display panel 141. Stereo camera 131 has a configuration in which two camera units having the same angle of view are arranged apart from each other, and outputs image signals captured by each camera unit.

[0028] The display panel 141 is, for example, a liquid crystal panel, and displays the face of a set character as an animation, and displays information about the robot 100 as text or icons. If the face of a character is displayed on the display panel 141, it is possible to give people around the display panel 141 the impression that the display panel 141 is a pseudo face.

[0029] Head 122 can rotate about a vertical axis relative to torso 121 by the driving force of a motor (not shown). Therefore, stereo camera 131 can capture images in any direction, and display panel 141 can present display content in any direction.

[0030] 5 is a block diagram showing an example of the block configuration of the robot 100. Here, the main elements related to the control of the operation based on the handwritten input information will be described, but the configuration of the robot 100 also includes other elements, and other elements that contribute to the control of the operation based on the handwritten input information may also be added.

[0031] The control unit 150 is, for example, a CPU, and is stored in a control unit provided in, for example, the body 121. The cart drive unit 145 includes the drive wheels 111, and a drive circuit and a motor for driving the drive wheels 111. The control unit 150 executes rotation control of the drive wheels by sending a drive signal to the cart drive unit 145. The control unit 150 also receives a feedback signal from an encoder or the like from the cart drive unit 145 to grasp the moving direction and moving speed of the cart unit 110.

[0032] Upper body drive unit 146 includes arm 123, hand 124, torso 121, head 122, and drive circuits and motors for driving these. Control unit 150 realizes gripping motions and gestures by sending drive signals to upper body drive unit 146. Control unit 150 also receives feedback signals from an encoder or the like from upper body drive unit 146 to grasp the positions and movement speeds of arm 123 and hand 124, and the orientations and rotation speeds of torso 121 and head 122.

[0033] The display panel 141 receives and displays the image signal generated by the control unit 150. Furthermore, the control unit 150 may generate an image signal of a character or the like and cause the display panel 141 to display the image, as described above.

[0034] The stereo camera 131 captures an image of the first environment in which the robot 100 exists in response to a request from the control unit 150, and transfers the captured image signal to the control unit 150. The control unit 150 performs image processing using the captured image signal, and converts the captured image signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether or not an obstacle exists in the moving direction in response to a request from the control unit 150, and transfers a detection signal that is the detection result to the control unit 150.

[0035] The hand camera 135 is, for example, a distance image sensor, and is used to recognize the distance, shape, direction, etc. of the object to be grasped. The hand camera 135 includes an image sensor in which pixels that photoelectrically convert an optical image incident from the target space are arranged two-dimensionally, and outputs the distance to the object for each pixel to the control unit 150. Specifically, the hand camera 135 includes an irradiation unit that irradiates the target space with pattern light, receives the reflected light with the image sensor, and outputs the distance to the object captured by each pixel from the distortion and size of the pattern in the image. The control unit 150 grasps the state of a wider surrounding environment with the stereo camera 131, and grasps the state near the object to be grasped with the hand camera 135.

[0036] The memory 180 is a non-volatile storage medium, and may be, for example, a solid state drive. The memory 180 stores various parameter values, functions, lookup tables, and the like used for control and calculation, in addition to a control program for controlling the robot 100. The memory 180 may store a learned model that uses an image of handwritten input information as an input image and outputs the meaning of a gripping motion simulated by the handwritten input information.

[0037] The communication unit 190 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 190 receives handwritten input information sent from the remote terminal 300 and passes it to the control unit 150. The communication unit 190 also transmits images captured by the stereo camera 131 to the remote terminal 300 under the control of the control unit 150.

[0038] The control unit 150 executes a control unit program read from the memory 180 to control the entire robot 100 and perform various arithmetic processing. The control unit 150 also serves as a function execution unit that executes various calculations and controls related to control. As such function execution units, the control unit 150 includes an imaging control unit 151, a movement control unit 152, and a grip control unit 153. At least some of the functions of a determination unit 352 and a presentation unit 353, which will be described later, may be included in the control unit 150.

[0039] The imaging control unit 151 changes the imaging direction of the stereo camera 131 based on the handwritten input information. The imaging control unit 151 may rotate the cart unit 110 or the head unit 122 in the direction in which the touch pen or a finger is moved, for example.

[0040] The movement control unit 152 moves the cart unit 110 based on the handwritten input information. For example, when the handwritten input information includes a line, the movement control unit 152 generates a trajectory passing through points on the line. Then, the movement control unit 152 may send a drive signal to the cart drive unit 145 so that the cart unit 110 moves along the trajectory.

[0041] The grip control unit 153 grips the grip target based on the handwritten input information. The grip control unit 153 recognizes the grip target shown in the captured image using, for example, a trained model for recognition, and generates a trajectory of the hand 124 so as to grip the grip target from a direction according to the handwritten input information. The grip control unit 153 may then transmit a drive signal according to the generated trajectory to the upper body drive unit 146.

[0042] Furthermore, the grip control unit 153 may place the gripping object on a table, a floor, or the like based on the handwritten input information. For example, the grip control unit 153 generates a trajectory of the hand 124 so as to place the gripping object at a position specified by the handwritten input information. Then, the grip control unit 153 may transmit a drive signal according to the generated trajectory to the upper body drive unit 146.

[0043] 6 is a block diagram showing an example of the block configuration of the remote terminal 300. Here, the main elements related to the process of inputting handwritten input information to the captured image received from the robot 100 will be described, but the remote terminal 300 may also include other elements, and other elements that contribute to the process of inputting handwritten input information may also be added.

[0044] Control unit 350 is, for example, a CPU, and executes a control program read from memory 380 to control the entire remote terminal 300 and perform various arithmetic processing. Control unit 350 also serves as a function execution unit that executes various arithmetic operations and controls related to the control. As such a function execution unit, control unit 350 includes handwritten input information receiving unit 351, determination unit 352, and presentation unit 353.

[0045] The handwritten input information accepting unit 351 displays the captured image on the display panel 341 and accepts input of handwritten input information from the input unit 342. The handwritten input information accepting unit 351 may accept input into a three-dimensional space. The captured image may be a three-dimensional image. Furthermore, the handwritten input information accepting unit 351 superimposes an image corresponding to the input handwritten input information on the captured image.

[0046] The handwritten input information receiving unit 351 may display a predetermined area included in the captured image in a distinguishable manner from an area other than the predetermined area. The predetermined area may be, for example, an obstacle (e.g., a desk or a floor) that impedes the operation of the robot 100 or an area in which the robot 100 is captured. When a part or the whole of the handwritten input information is included in the predetermined area, it is determined that the handwritten input information needs to be corrected.

[0047] FIG. 7 is a diagram for explaining an example of the captured image 310 displayed by the handwritten input information receiving unit 351. The captured image 310 shows an obstacle 402 that hinders the movement of the robot 100. The captured image 310 also includes an area 311 in which a part of the robot 100 is shown. The above-mentioned predetermined area includes the area in which the obstacle 402 is shown and the area 311. If a part or all of the handwritten input information indicating the movement path of the robot 100 is included in the predetermined area, the robot 100 cannot operate based on the handwritten input information. The area 312 other than the predetermined area is an area in which the handwritten input information can be input. Therefore, the handwritten input information receiving unit 351 displays the area 312 and the area other than the area 312 in different colors. This can encourage the user to input the handwritten input information in the appropriate area. The handwritten input information receiving unit 351 may, for example, add a specific light color to the area 312. The handwritten input information receiving unit 351 may improve the identification accuracy of the predetermined area and the area 312 based on the segmentation information. The segmentation information is information indicating a result of performing segmentation on the captured image 310 using a learning model or the like. Note that the handwritten input information accepting unit 351 may change the imaging direction if the imaging unit faces above or below a certain range when the user starts to input handwritten input information.

[0048] Furthermore, when displaying the handwritten input information in the captured image, the handwritten input information receiving unit 351 may set the thickness of the lines included in the handwritten input information based on the size of the robot 100. This allows the user to input appropriate handwritten input information taking into account the size of the robot 100. FIG. 8 is a diagram for explaining an example of the captured image 310 displayed by the handwritten input information receiving unit 351. The captured image 310 displays handwritten input information 902a representing the moving path of the robot 100. The thickness of the lines in the handwritten input information 902a is set according to the size of the robot 100. Since an object far from the robot 100 appears small in the captured image 310, the thickness of the lines corresponding to positions far from the current position of the robot 100 is set to be thin. This allows appropriate handwritten input information to be input even for positions far from the current position of the robot 100.

[0049] Returning to FIG. 6, the determination unit 352 determines whether or not the handwritten input information needs to be corrected based on the feasibility or efficiency of the operation based on the handwritten input information. For example, when a part or all of the handwritten input information indicating the movement path of the robot 100 is included in a predetermined area, the operation based on the handwritten input information is not executable. Also, when the stability of the gripping operation of the robot 100 according to the handwritten input information is low, the operation based on the handwritten input information is not executable. In such a case, the determination unit 352 determines that the handwritten input information needs to be corrected. Also, the determination unit 352 may determine that the handwritten input information needs to be corrected not only when the operation is not executable, but also when the efficiency of the operation is low.

[0050] When the handwritten input information needs to be corrected, the presentation unit 353 presents information regarding the correction of the handwritten input information to the user. This allows the remote terminal 300 to execute an interaction with the user regarding the correction of the handwritten input information. The presentation unit 353 may present the information to the user to prompt the user to correct the handwritten input information. The correction of the handwritten input information may be inputting appropriate handwritten input information. Furthermore, the presentation unit 353 may correct the handwritten input information and present the corrected handwritten input information to the user.

[0051] FIG. 9 is a diagram for explaining an example of information presented by the presenting unit 353. The handwritten input information 9021 indicated by a dotted line represents a moving path where the robot 100 and the obstacle 402 interfere with each other. In this case, the determining unit 352 determines that the handwritten input information 902 needs to be corrected. The presenting unit 353 determines a contact point P1 where the robot 100 and the obstacle 402 come into contact based on segmentation information of the captured image 310 and contact point information between the virtual bumper of the robot 100 and the obstacle 402. Then, the presenting unit 353 presents handwritten input information 9022 representing a moving path passing through an adjacent point P2 adjacent to the contact point P1. When a confirmation operation such as pressing a confirmation button is input, the presenting unit 353 may cause the robot 100 to execute an action based on the handwritten input information 9022.

[0052] 10 is a diagram for explaining an example of information presented by the presenting unit 353. The captured image 310 includes handwritten input information 901 representing a gripping motion by the robot 100. An area 314 represents a portion of the hand 124 performing the motion represented by the handwritten input information 901 that interferes with an obstacle (floor). The presenting unit 353 displays the area 314 in a distinguishable manner from an area other than the area 314. The user can check the captured image 310 and input appropriate handwritten input information. The presenting unit 353 may further present the hand 124 performing the motion based on the handwritten input information.

[0053] The presentation unit 353 may correct the handwritten input information and cause the robot 100 to perform an action based on the corrected handwritten input information. The presentation unit 353 presents the corrected handwritten information to the user.

[0054] 11, the movement path represented by the handwritten input information 9021 shown by a dotted line passes through a space in which the grasp target 401 exists, and therefore the handwritten input information 9021 needs to be corrected. In this case, the presentation unit 353 corrects the handwritten input information 9021 to handwritten input information 9022 including a detour that detours around the grasp target 401. The detour includes, for example, points on the trajectory represented by the handwritten input information 9021 and an intersection point between a straight line passing through the viewpoint of the stereo camera 131 and the floor.

[0055] 12, handwritten input information 9021 indicated by a dotted line represents a movement route that passes near an obstacle 402 such as a table or a panel, and the handwritten input information 9021 needs to be corrected. In this case, the presenting unit 353 corrects the handwritten input information 9021 to handwritten input information 9022 that represents a detour route that goes around the obstacle 402.

[0056] Furthermore, when the handwritten input information indicating the movement path of the robot 100 passes through a predetermined area in which the robot 100 is shown, the handwritten input information needs to be corrected. For example, when the trajectory represented by the handwritten input information includes a point passing through an area within a predetermined radius from the robot 100, the presentation unit 353 may generate a new trajectory by removing the point from the trajectory. Then, the presentation unit 353 presents the handwritten input information corresponding to the generated trajectory to the user.

[0057] 13 is a diagram for explaining an example of a method for correcting handwritten input information. When correcting the handwritten input information 901, the presenting unit 353 first corrects the position of the handwritten input information 901 until a gripping point 9011 included in the U-shaped handwritten input information 901 is located in an area corresponding to the gripping target 401. Then, the presenting unit 353 moves the handwritten input information 901 in a stepwise manner in a direction away from the gripping target 401 until an operation based on the handwritten input information 901 can be executed. The presenting unit 353 may determine a gripping position / posture that is closest to the gripping position / posture estimated from the handwritten input information 901 and allows the gripping target 401 to be stably gripped by any other method.

[0058] The presentation unit 353 may use segmentation information based on machine learning to correct the position of the handwritten input information 901. For example, the segmentation information may be used when positioning the gripping point 9011 on the gripping object 401. The segmentation information includes information indicating a region corresponding to the gripping object 401.

[0059] Returning to FIG. 6, the display panel 341 is, for example, a liquid crystal panel, and displays, for example, a captured image sent from the robot 100.

[0060] The input unit 342 includes a touch panel superimposed on the display panel 141, a push button provided on the periphery of the display panel 141, etc. The input unit 342 passes handwritten input information to the control unit 350. Examples of the handwritten input information are as shown in, for example, FIGS. 2 and 3.

[0061] Memory 380 is a non-volatile storage medium, such as a solid state drive, and stores a control program for controlling remote terminal 300, as well as various parameter values, functions, lookup tables, and the like used for control and calculation.

[0062] The communication unit 390 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 390 receives captured images sent from the robot 100 and passes them to the control unit 350. The communication unit 390 also cooperates with the control unit 350 to transmit handwritten input information to the robot 100.

[0063] An example of the operation of the control system 10 according to the first embodiment will be described with reference to Fig. 14. First, the handwritten input information accepting unit 351 of the remote terminal 300 displays a captured image and accepts input of handwritten input information (step S101). At this time, the handwritten input information accepting unit 351 displays the captured image and the handwritten input information so that appropriate handwritten input information is input.

[0064] Next, the user completes input of the handwritten input information (step S102). Next, the determination unit 352 of the remote terminal 300 estimates the motion of the robot 100 based on the handwritten input information (step S103). Next, the determination unit 352 determines whether or not the handwritten input information needs to be corrected based on the feasibility or efficiency of the estimated motion (step S104). If no correction is required (NO in step S104), the remote terminal 300 transmits the handwritten input information to the robot 100, and step S108 is executed.

[0065] If the handwritten input information needs to be corrected (YES in step S104), the presentation unit 353 of the remote terminal 300 presents information regarding the correction of the handwritten input information to the user (step S105). The presentation unit 353 may present information regarding the part of the handwritten input information that should be corrected, or may present the corrected handwritten input information.

[0066] Next, the handwritten input information receiving unit 351 determines whether or not a correction instruction for the handwritten input information has been input by the user (step S106). The correction instruction may be, for example, input of appropriate handwritten input information. The user may also instruct to correct the presented handwritten input information. If a correction instruction has been input by the user (YES in step S106), step S101 is executed.

[0067] If the user does not input a correction instruction (NO in step S106), the presentation unit 353 automatically corrects the handwritten input information (step S107), and the remote terminal 300 transmits the corrected handwritten input information to the robot 100. Then, the presentation unit 353 presents the corrected handwritten input information to the user. Then, the robot 100 executes an action based on the handwritten input information input by the user or the corrected handwritten input information (step S108).

[0068] Next, the effects of the control system 10 according to the first embodiment will be described. When instructing a robot by handwritten input characters, the user can freely write the instructions, but there is a problem that the robot may not be able to operate according to the instructions. For example, when the user's instructions are not accurate or when the robot cannot accurately understand the user's intention, the handwritten input characters need to be corrected.

[0069] The control system 10 according to the first embodiment can improve operability by performing interactions with the user regarding correction of handwritten input information.

[0070] The present invention is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In the above embodiment, the robot 100 and the remote terminal 300 exchange captured images and handwritten input information via the Internet 600 and the system server 500, but the present invention is not limited to this. The robot 100 and the remote terminal 300 may exchange captured images and handwritten input information through direct communication.

[0071] In the above embodiment, the stereo camera 131 is used, but the present invention is not limited to this. The robot 100 may include an arbitrary imaging unit provided at an arbitrary location in the first environment. The imaging unit is not limited to a stereo camera, and may be a monocular camera or the like.

[0072] In the above embodiment, the robot 100 is described as having the hand 124 at the tip of the arm 123 as an end effector, but the present invention is not limited to this. The robot 100 may be any robot that is equipped with an end effector and performs a gripping operation using the end effector. The end effector may be a gripping unit other than a hand (for example, a suction unit, etc.).

[0073] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0074] 10. Control System 100 Robots 110 Bogie section 111 Drive wheel 112 Caster 120 Main body 121 Torso 122 Head 123 Arm 124 Hands 131 Stereo Camera 133 Laser Scanner 135 Hand Camera 141 Display Panel 145 Truck drive unit 146 Upper body drive unit 150 Control section 180 Memory 190 Communication Unit 300 Remote Terminal 341 Display Panel 342 Input Unit 350 Control section 351 Handwritten input information reception unit 352 Judgment section 353 Presentation part 380 Memory 390 Communication Unit 401 Grasped object 402 Obstacles 500 System Server 600 Internet 700 Wireless Router 901, 902, 902a, 9021, 9022 Handwritten input information 9011 Grip point 310 Captured Images 311, 312, 313, 314 area

Claims

1. A control system that determines an action of a robot based on handwritten input information input to an interface and controls the action of the robot, a handwritten input information receiving unit that receives the handwritten input information; a determination unit that determines whether the handwritten input information needs to be corrected based on the feasibility or efficiency of the operation based on the handwritten input information; a presentation unit that presents information regarding the correction of the handwritten input information to a user when the handwritten input information needs to be corrected; A control system comprising:

2. The handwriting input information includes trajectory information input using a finger, a touch pen, a pointing device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device. The control system of claim 1 .

3. The presentation unit, when it is necessary to correct the handwritten input information, corrects the handwritten input information and presents the corrected handwritten input information to the user. A control system according to claim 1 or 2.

4. When the handwritten input information needs to be corrected, the presenting unit presents to the user information regarding a portion of the robot operating based on the handwritten input information that is interfering with an obstacle. A control system according to claim 1 or 2.

5. the determination unit determines that the handwritten input information needs to be corrected when a part or all of the handwritten input information is included in a predetermined area of ​​an input screen; The handwritten input information receiving unit displays the input screen in such a manner that the predetermined area and an area other than the predetermined area can be distinguished from each other. A control system according to claim 1 or 2.

6. The handwritten input information receiving unit sets a thickness of lines included in the handwritten input information based on a size of the robot when displaying the input handwritten input information. A control system according to claim 1 or 2.

7. The presenting unit corrects a position of the handwritten input information using segmentation information based on machine learning. A control system according to claim 1 or 2.

8. 1. A control method for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: receiving the handwritten input information; determining whether the handwritten input needs to be modified based on the feasibility or efficiency of an operation based on the handwritten input; if the handwritten input information needs to be corrected, presenting the user with information for correcting the handwritten input information; A control method comprising:

9. A program for causing a computer to execute a control method for determining an action of a robot based on handwritten input information inputted to an interface and controlling the action of the robot, comprising: The control method includes: receiving the handwritten input information; determining whether the handwritten input needs to be modified based on the feasibility or efficiency of an operation based on the handwritten input; if the handwritten input information needs to be corrected, presenting the user with information for correcting the handwritten input information; Including, the program.

Images