Control system, control method, and program

Abstract

To provide a control system, a control method and a program which allow a robot to perform a plurality of kinds of operations using handwriting input information.SOLUTION: A control system 10 allows a robot 100 to execute operation on the basis of handwriting input information input to an interface. The control system 10 includes: a handwriting input information reception part 351 for displaying a photographic image of an environment that the robot 100 exists, and receiving input of the handwriting input information for the displayed photographic image; and a switching part 352 for switching a plurality of input modes of the handwriting input information for allowing the robot 100 to perform different kinds of operations.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control system, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a remote control system that controls the operation of a robot based on handwritten input information into an interface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-094605 Summary of the Invention [Problem to be solved by the invention]

[0004] Robot motions include multiple types of motions, such as a motion to move the robot's viewpoint, a motion to move the robot to a destination, a motion to grasp an object to be grasped, etc. It is difficult to distinguish the type of motion using handwritten input information, and there is a risk that the usability of the system will be impaired.

[0005] The present disclosure has been made to solve such problems, and has an object to provide a control system, a control method, and a program for switching the input mode of handwritten input information. [Means for solving the problem]

[0006] The control system in this embodiment includes: A control system for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: a handwritten input information receiving unit that receives the handwritten input information; a switching unit for switching between a plurality of input modes of the handwritten input information for causing the robot to execute different types of operations; Equipped with.

[0007] a notification section for notifying a current input mode to a user who inputs the handwritten input information; The device may further include:

[0008] The handwriting input information may include trajectory information input using a finger, a touch pen, a pointing device, an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device.

[0009] The handwritten entry information is input to a captured image of an environment in which the robot exists; The plurality of input modes may include a first mode for changing an imaging direction of the captured image, a second mode for moving the robot, and a third mode for performing a grasping operation by an end effector.

[0010] The switching unit may switch to a mode other than the second mode when the captured image does not show an area in which the robot can move, and may switch to a mode other than the third mode when the end effector is grasping an object to be grasped.

[0011] The switching unit may switch between the plurality of input modes in response to a selection operation.

[0012] The handwritten entry information is input to a captured image of an environment in which the robot exists; The switching unit may input the captured image to a learning model and switch to a mode represented by information output from the learning model.

[0013] The control method in this embodiment is as follows: A control method for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: receiving handwritten input information for the captured image; switching between a plurality of input modes of the handwritten input information for causing the robot to perform different types of actions; Includes.

[0014] The program in this embodiment is A program for causing a computer to execute a control method for determining an action of a robot based on handwritten input information inputted to an interface and controlling the action of the robot, comprising: The control method includes: receiving handwritten input information for the captured image; switching between a plurality of input modes of the handwritten input information for causing the robot to perform different types of actions; Includes. Effect of the Invention

[0015] The present disclosure makes it possible to provide a control system, a control method, and a program for switching the input mode of handwritten input information. [Brief description of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of an overall environment in which a control system according to a first embodiment is used. [Diagram 2] FIG. 11 is a diagram showing an example of handwritten input information. [Diagram 3] FIG. 11 is a diagram showing an example of handwritten input information. [Figure 4] FIG. 2 is an external perspective view showing an example of the external configuration of the robot. [Diagram 5] FIG. 2 is a block diagram showing an example of a block configuration of a robot. [Figure 6] FIG. 2 is a block diagram showing an example of a block configuration of a remote terminal; [Figure 7] FIG. 13 is a diagram illustrating an example of a display screen of a remote terminal. [Figure 8]5 is a flowchart showing an example of the operation of the control system according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] EMBODIMENT 1 The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0018] 1 is a conceptual diagram showing an example of an overall environment in which a control system 10 according to the embodiment 1 is used. A robot 100 that performs various actions in a first environment is remotely controlled by a user who is a remote operator in a second environment away from the first environment, by operating a remote terminal 300 (operation terminal) via a system server 500 connected to the Internet 600.

[0019] In a first environment, the robot 100 is connected to the Internet 600 via a wireless router 700. In a second environment, the remote terminal 300 is connected to the Internet 600 via a wireless router 700. The robot 100 performs a grasping action with the hand 124 in accordance with the operation of the remote terminal 300.

[0020] The robot 100 captures an image of the first environment in which the robot 100 exists by using the stereo camera 131 (imaging unit) and transmits the captured image to the remote terminal 300 via the Internet 600. The robot 100 also recognizes a graspable object that can be grasped by the hand 124 based on the captured image. In the first environment, a graspable object 401 such as a can exists. Note that the shape of the graspable object 401 is not limited to a cylindrical shape.

[0021] The remote terminal 300 is, for example, a tablet terminal, and has a display panel 341 on which a touch panel is superimposed. The captured image received from the robot 100 is displayed on the display panel 341, and the user can indirectly view the first environment in which the robot 100 exists. The user can input handwritten input information by handwriting on the captured image displayed on the display panel 341. The handwritten input information indicates, for example, the rotation of the imaging unit, the moving path of the robot 100, the object to be grasped by the hand 124, and the grasping direction. The handwritten input information can be input by touching a corresponding part of the captured image on a touch panel superimposed on the display panel 341 with a user's finger or a touch pen, but is not limited to this. The handwritten input information may be trajectory information input using a pointing device such as a mouse. The handwritten input information may also be trajectory information of a part of the user's body (e.g., a finger) input into a three-dimensional input space using an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0022] 2 and 3 are diagrams showing examples of handwritten input information input to the captured image 310. The example of FIG. 2 shows handwritten input information 901 simulating grasping the grasp target 401 from the side. The example of FIG. 3 shows handwritten input information 902 simulating the movement path of the robot 100. Although not shown, handwritten input information simulating the rotation of the imaging unit may be input. For example, when a touch pen or the like is moved up, down, left, or right on the captured image 310, the viewpoint of the imaging unit follows that direction. The robot 100 may change the viewpoint by rotating its head, or may change the viewpoint by turning on the spot. For example, when the imaging unit is rotated in the pan direction, the robot 100 turns on the spot, and when the imaging unit is rotated in the tilt direction, the head of the robot 100 may be tilted. The handwritten input information may include handwritten character information. The handwritten input information input by the user to the captured image is transmitted to the robot 100 via the Internet 600.

[0023] The control system 10 switches between a plurality of input modes that cause the robot 100 to perform different types of operations. The plurality of input modes may include, for example, a first mode for changing the imaging direction of the imaging unit, a second mode for moving the robot 100, and a third mode for causing the robot 100 to perform a grasping operation.

[0024] FIG. 4 is an external perspective view showing an example of the external configuration of the robot 100. The robot 100 is broadly composed of a cart unit 110 and a main body unit 120. The cart unit 110 supports two drive wheels 111 and one caster 112, each installed on a running surface, in a cylindrical housing. The two drive wheels 111 are arranged so that their rotational axes coincide with each other. Each drive wheel 111 is independently rotated by a motor (not shown). The caster 112 is a driven wheel, and is provided such that a rotating shaft extending vertically from the cart unit 110 is separated from the rotational axis of the wheel to support the wheel, and follows the moving direction of the cart unit 110.

[0025] The cart unit 110 is provided with a laser scanner 133 on the periphery of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane for each step angle, and outputs whether or not an obstacle exists in each direction. Furthermore, if an obstacle exists, the laser scanner 133 outputs the distance to the obstacle.

[0026] The main body 120 mainly includes a trunk 121 mounted on the upper surface of the cart 110, a head 122 placed on the upper surface of the trunk 121, an arm 123 supported on the side of the trunk 121, and a hand 124 installed at the tip of the arm 123. The arm 123 and the hand 124 are driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 is driven via a motor (not shown) to grasp an object to be grasped. The trunk 121 can rotate around a vertical axis relative to the cart 110 by the driving force of the motor (not shown).

[0027] Head 122 mainly includes stereo camera 131 and display panel 141. Stereo camera 131 has a configuration in which two camera units having the same angle of view are arranged apart from each other, and outputs image signals captured by each camera unit.

[0028] The display panel 141 is, for example, a liquid crystal panel, and displays the face of a set character as an animation, and displays information about the robot 100 as text or icons. If the face of a character is displayed on the display panel 141, it is possible to give people around the display panel 141 the impression that the display panel 141 is a pseudo face.

[0029] Head 122 can rotate about a vertical axis relative to torso 121 by the driving force of a motor (not shown). Therefore, stereo camera 131 can capture images in any direction, and display panel 141 can present display content in any direction.

[0030] 5 is a block diagram showing an example of the block configuration of the robot 100. Here, the main elements related to the control of the operation based on the handwritten input information will be described, but the configuration of the robot 100 also includes other elements, and other elements that contribute to the control of the operation based on the handwritten input information may also be added.

[0031] The control unit 150 is, for example, a CPU, and is stored in a control unit provided in, for example, the body 121. The cart drive unit 145 includes the drive wheels 111, and a drive circuit and a motor for driving the drive wheels 111. The control unit 150 executes rotation control of the drive wheels by sending a drive signal to the cart drive unit 145. The control unit 150 also receives a feedback signal from an encoder or the like from the cart drive unit 145 to grasp the moving direction and moving speed of the cart unit 110.

[0032] Upper body drive unit 146 includes arm 123, hand 124, torso 121, head 122, and drive circuits and motors for driving these. Control unit 150 realizes gripping motions and gestures by sending drive signals to upper body drive unit 146. Control unit 150 also receives feedback signals from an encoder or the like from upper body drive unit 146 to grasp the positions and movement speeds of arm 123 and hand 124, and the orientations and rotation speeds of torso 121 and head 122.

[0033] The display panel 141 receives and displays the image signal generated by the control unit 150. Furthermore, the control unit 150 may generate an image signal of a character or the like and cause the display panel 141 to display the image, as described above.

[0034] The stereo camera 131 captures an image of the first environment in which the robot 100 exists in response to a request from the control unit 150, and transfers the captured image signal to the control unit 150. The control unit 150 performs image processing using the captured image signal, and converts the captured image signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether or not an obstacle exists in the moving direction in response to a request from the control unit 150, and transfers a detection signal that is the detection result to the control unit 150.

[0035] The hand camera 135 is, for example, a distance image sensor, and is used to recognize the distance, shape, direction, etc. of the object to be grasped. The hand camera 135 includes an image sensor in which pixels that photoelectrically convert an optical image incident from the target space are arranged two-dimensionally, and outputs the distance to the object for each pixel to the control unit 150. Specifically, the hand camera 135 includes an irradiation unit that irradiates the target space with pattern light, receives the reflected light with the image sensor, and outputs the distance to the object captured by each pixel from the distortion and size of the pattern in the image. The control unit 150 grasps the state of a wider surrounding environment with the stereo camera 131, and grasps the state near the object to be grasped with the hand camera 135.

[0036] The memory 180 is a non-volatile storage medium, and may be, for example, a solid state drive. The memory 180 stores various parameter values, functions, lookup tables, and the like used for control and calculation, in addition to a control program for controlling the robot 100. The memory 180 may store a learned model that uses an image of handwritten input information as an input image and outputs the meaning of a gripping motion simulated by the handwritten input information.

[0037] The communication unit 190 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 190 receives handwritten input information sent from the remote terminal 300 and passes it to the control unit 150. The communication unit 190 also transmits images captured by the stereo camera 131 to the remote terminal 300 under the control of the control unit 150.

[0038] The control unit 150 executes a control unit program read from the memory 180 to control the entire robot 100 and perform various arithmetic processing. The control unit 150 also serves as a function execution unit that executes various calculations and controls related to control. As such function execution units, the control unit 150 includes an imaging control unit 151, a movement control unit 152, and a grip control unit 153. At least a part of the functions of a switching unit 352, which will be described later, may be executed by the control unit 150.

[0039] When the input mode is the first mode, the imaging control unit 151 changes the imaging direction of the stereo camera 131 based on the handwritten input information. For example, the imaging control unit 151 may rotate the cart unit 110 or the head 122 in the direction in which the touch pen or a finger is moved.

[0040] When the input mode is the second mode, the movement control unit 152 moves the cart unit 110 based on the handwritten input information. For example, when the handwritten input information includes a linear figure, the movement control unit 152 generates a trajectory passing through points on the line. Then, the movement control unit 152 may send a drive signal to the cart drive unit 145 so that the cart unit 110 moves along the trajectory.

[0041] When the input mode is the third mode, the grip control unit 153 grips the grip target based on the handwritten input information. The grip control unit 153 recognizes the grip target appearing in the captured image using, for example, a trained model for recognition, and generates a trajectory of the hand 124 so as to grip the grip target from a direction according to the handwritten input information. The grip control unit 153 may then transmit a drive signal according to the generated trajectory to the upper body drive unit 146.

[0042] Furthermore, when the input mode is the fourth mode, the grip control unit 153 may place the gripping object on a table, a floor, or the like based on the handwritten input information. For example, the grip control unit 153 generates a trajectory of the hand 124 so as to place the gripping object at a position specified by the handwritten input information. Then, the grip control unit 153 may transmit a drive signal according to the generated trajectory to the upper body drive unit 146.

[0043] 6 is a block diagram showing an example of the block configuration of the remote terminal 300. Here, the main elements related to the process of inputting handwritten input information to the captured image received from the robot 100 will be described, but the remote terminal 300 may also include other elements, and other elements that contribute to the process of inputting handwritten input information may also be added.

[0044] Control unit 350 is, for example, a CPU, and executes a control program read from memory 380 to control the entire remote terminal 300 and perform various arithmetic processing. Control unit 350 also serves as a function execution unit that executes various arithmetic operations and controls related to control. As such a function execution unit, control unit 350 includes handwritten input information acceptance unit 351, switching unit 352, and notification unit 353.

[0045] The handwritten input information accepting unit 351 displays the captured image on the display panel 341 and accepts input of handwritten input information from the input unit 342. Note that the handwritten input information accepting unit 351 may accept input into a three-dimensional space. The captured image may be a three-dimensional image.

[0046] The switching unit 352 switches between a plurality of input modes for causing the robot 100 to execute different types of operations. The switching unit 352 may switch between a plurality of input modes in response to a selection operation. The selection operation may be performed, for example, by a user operating the remote terminal 300, or by a person around the user who supports the user. For example, the switching unit 352 displays buttons corresponding to each mode on the display panel 141. When an area corresponding to each button on the touch panel is pressed, the switching unit 352 switches to the mode corresponding to the button. The switching unit 352 may also select a mode corresponding to a voice input by the user.

[0047] The switching unit 352 may also input the captured image to the trained model and switch to a mode represented by information output from the trained model. The trained model is constructed by supervised learning using teacher data including, for example, a captured image and a label indicating an appropriate input mode corresponding to the captured image. For example, when a captured image showing an area in which the robot 100 can move (e.g., an area without obstacles) is input, the trained model outputs information indicating the second mode. For example, when a captured image showing a grasp target is input, the trained model outputs information indicating the third mode. For example, when a captured image showing only a part of the grasp target is input, the trained model may output information indicating the first mode. The switching unit 352 may also select one mode using a rule-based method.

[0048] The switching unit 352 may determine whether an area in which the robot 100 can move is shown in the captured image, and may switch to a mode other than the second mode if the area in which the robot 100 can move is not shown in the captured image. Analysis of the captured image may be performed by the robot 100 or may be performed by the remote terminal 300. The switching unit 352 may, for example, deactivate a button corresponding to the second mode.

[0049] The switching unit 352 may determine whether the robot 100 is gripping an object to be grasped, and may switch to a mode other than the third mode if the hand 124 of the robot 100 is not gripping an object to be grasped. The switching unit 352 may, for example, receive information indicating whether the object to be grasped is being grasped from the robot 100, and make a determination based on the received information. The robot 100 can determine whether the object to be grasped is being grasped based on the detection result of a sensor (e.g., a hand camera) provided on the hand 124. The switching unit 352 may deactivate a button corresponding to the third mode.

[0050] As already described, the switching unit 352 may be provided on the robot 100 side, not on the remote terminal 300. For example, when switching the input mode using a learning model, the switching unit 352 may be provided on the control unit 150 of the robot 100.

[0051] Notification unit 353 notifies the user operating remote terminal 300 of the current input mode. Notification unit 353 may display and output information (e.g., an icon) indicating the current input mode on display panel 341, or may output the information as sound from a speaker (not shown). This allows the user to input handwritten input information after checking the current input mode.

[0052] The display panel 341 is, for example, a liquid crystal panel, and displays, for example, a captured image sent from the robot 100 .

[0053] The input unit 342 includes a touch panel superimposed on the display panel 141, a push button provided on the periphery of the display panel 141, etc. The input unit 342 passes handwritten input information to the control unit 350. Examples of the handwritten input information are as shown in, for example, FIGS. 2 and 3.

[0054] Memory 380 is a non-volatile storage medium, such as a solid state drive, and stores a control program for controlling remote terminal 300, as well as various parameter values, functions, lookup tables, and the like used for control and calculation.

[0055] The communication unit 390 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 390 receives captured images sent from the robot 100 and passes them to the control unit 350. The communication unit 390 also cooperates with the control unit 350 to transmit handwritten input information to the robot 100.

[0056] 7 is an explanatory diagram showing an example of a display screen displayed on the display panel 341. The illustrated display screen 20 includes a mode selection button 22, a viewer 23, and a command button 24.

[0057] The mode selection buttons 22 include a button 221 corresponding to the first mode, a button 222 corresponding to the second mode, and a button 223 corresponding to the third mode. The mode selection buttons may further include a button (not shown) corresponding to a fourth mode. For example, when the button 222 is selected, the button 222 may be displayed in a manner (e.g., color) different from the buttons 221 and 223. The viewer 23 displays an image captured by the stereo camera 131. For example, the captured image may include a grasp target 401, an obstacle 402, and the like. The command button 24 displays information indicating the currently executed mode. For example, when the second mode is selected, the text "moving" or an animation indicating the movement of the robot 100 is displayed. The notification unit 353 can notify the current mode by the command button 24 or the like.

[0058] Next, an example of the operation of the control system 10 according to the first embodiment will be described. First, the switching unit 352 of the control unit 350 of the remote terminal 300 judges whether or not an area in which the robot 100 can move is shown in the captured image, and judges whether or not the hand 124 of the robot 100 is gripping a gripping object, and determines a mode candidate according to the judgment result (step S101). If an area in which the robot 100 can move is not shown in the captured image, the switching unit 352 sets a mode other than the second mode as the mode candidate. If the hand 124 is gripping a gripping object, the switching unit 352 sets a mode other than the third mode as the mode candidate. If an area in which the robot 100 can move is not shown in the captured image, and the hand 124 is gripping a gripping object, the switching unit 352 sets a mode other than the second mode and the third mode as the mode candidate. The switching unit 352 activates a button corresponding to the mode candidate, and deactivates a button corresponding to other than the mode candidate.

[0059] Next, the control unit 350 determines whether or not a selection operation has been performed (step S102). If no selection operation has been performed (NO in step S102), the process returns to step S102.

[0060] If a selection operation has been performed (YES in step S102), the switching unit 352 of the control unit 350 switches to the first mode (step S103), the second mode (step S104), the third mode (step S105), or the fourth mode (step S106) depending on the selection operation.

[0061] When switched to the first mode, the control unit 350 receives an instruction to change the shooting direction of the stereo camera 131 according to the handwritten input information, and the imaging control unit 151 of the robot 100 changes the shooting direction of the stereo camera 131 according to the change instruction (step S107).

[0062] When switched to the second mode, the control unit 350 receives an instruction to move the cart unit 110 in accordance with the handwritten input information, and the movement control unit 152 of the robot 100 moves the cart unit 110 in accordance with the movement instruction (step S108).

[0063] When the third mode is switched to, the control unit 350 receives a gripping operation instruction corresponding to the handwritten input information, and the grip control unit 153 of the robot 100 grips the gripping target object in accordance with the gripping operation instruction (step S109).

[0064] When the fourth mode is selected, the control unit 350 receives an instruction to place the object to be grasped according to the handwritten input information, and the grip control unit 153 of the robot 100 places the object to be grasped at a position according to the instruction (step S110).

[0065] Then, the control system 10 judges whether or not to end the remote operation (step S111). If the remote operation is not to be ended (NO in step S111), the process returns to step S101, and if the remote operation is to be ended (YES in step S111), the process ends.

[0066] Instead of steps S101 to S106, a step of switching the input mode using a learning model or the like may be provided.

[0067] Next, the effects of the control system 10 according to the first embodiment will be described. The actions performed by a robot include changing the viewpoint and moving a cart in addition to manipulating an object, and these actions are combined to achieve a meaningful task. However, in the conventional technology, there is a problem in that it is not possible to distinguish whether a certain piece of handwritten input information is an instruction for manipulating an object or an instruction for moving a cart. In a system that accepts free input as handwritten input characters (e.g., lines), when the tasks performed by a robot become complex and diverse, how to interpret the meaning of the input handwritten input information becomes particularly important.

[0068] The control system 10 according to the first embodiment is configured to be able to switch the input mode of handwritten input information, so that handwritten input characters can be easily interpreted, and the convenience of the system can be improved.

[0069] The present invention is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present invention. In the above embodiment, the robot 100 and the remote terminal 300 exchange captured images and handwritten input information via the Internet 600 and the system server 500, but the present invention is not limited to this. The robot 100 and the remote terminal 300 may exchange captured images and handwritten input information through direct communication.

[0070] In the above embodiment, the multiple input modes are the first to third modes or the first to fourth modes, but the present invention is not limited to this. The multiple modes may include any mode other than the first to fourth modes. The multiple input modes may not include some or all of the first to fourth modes.

[0071] In the above embodiment, the stereo camera 131 is used, but the present invention is not limited to this. The robot 100 may include any image capturing unit provided at any location in the first environment. The image capturing unit is not limited to a stereo camera, and may be a monocular camera or the like.

[0072] In the above embodiment, the robot 100 is described as having the hand 124 at the tip of the arm 123 as an end effector, but the present invention is not limited to this. The robot 100 may be any robot that is equipped with an end effector and performs a gripping operation using the end effector. The end effector may be a gripping unit other than a hand (for example, a suction unit, etc.).

[0073] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals. [Explanation of symbols]

[0074] 10. Control System 100 Robots 110 Bogie section 111 Drive wheel 112 Caster 120 Main body 121 Torso 122 Head 123 Arm 124 Hands 131 Stereo Camera 133 Laser Scanner 135 Hand Camera 141 Display Panel 145 Truck drive unit 146 Upper body drive unit 150 Control section 180 Memory 190 Communication Unit 300 Remote Terminal 341 Display Panel 342 Input Unit 350 Control section 380 Memory 390 Communication Unit 401 Grasped object 402 Obstacles 500 System Server 600 Internet 700 Wireless Router 901, 902 Input information 20 display screen 221, 222, 223 buttons

Claims

1. A control system for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: a handwritten input information receiving unit that receives the handwritten input information; a switching unit for switching between a plurality of input modes of the handwritten input information for causing the robot to execute different types of operations; A control system comprising:

2. a notification section for notifying a current input mode to a user who inputs the handwritten input information; The control system of claim 1 further comprising:

3. The handwriting input information includes trajectory information input using a finger, a touch pen, a pointing device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device. A control system according to claim 1 or 2.

4. The handwritten entry information is input to a captured image of an environment in which the robot exists; The plurality of input modes include a first mode for changing an imaging direction of the captured image, a second mode for moving the robot, and a third mode for performing a grasping operation by an end effector. A control system according to claim 1 or 2.

5. The switching unit switches to a mode other than the second mode when an area in which the robot can move is not captured in the captured image, and switches to a mode other than the third mode when the end effector is gripping a gripping object.

5. The control system of claim 4.

6. The switching unit switches between the plurality of input modes in response to a selection operation. A control system according to claim 1 or 2.

7. the handwritten entry information is input to a captured image of an environment in which the robot exists; The switching unit inputs the captured image into a learning model and switches to a mode represented by information output from the learning model. The control system of claim 1 .

8. A control method for determining an action of a robot based on handwritten input information input to an interface and controlling the action of the robot, comprising: receiving handwritten input information for the captured image; switching between a plurality of input modes of the handwritten input information for causing the robot to perform different types of actions; A control method comprising:

9. A program for causing a computer to execute a control method for determining an action of a robot based on handwritten input information inputted to an interface and controlling the action of the robot, comprising: The control method includes: receiving handwritten input information for the captured image; switching between a plurality of input modes of the handwritten input information for causing the robot to perform different types of actions; Including, the program.

Images