Control system, control method, and computer program product

Abstract

The present invention provides a control system, a control method, and a computer program product for causing a robot to perform a plurality of types of operations using handwriting input information. The control system causes the robot to perform an action based on the handwriting input information input to the interface. The control system includes: a handwritten input information reception unit that displays a captured image of an environment in which the robot is present, and receives input of handwritten input information to the displayed captured image; and a switching unit that switches a plurality of input modes of handwriting input information for causing the robot to execute different types of operations.

Description

Technical Field

[0001] The present disclosure relates to a control system, a control method, and a program. Background Art

[0002] Japanese Patent Application Laid-Open No. 2021-094605 discloses a remote operation system.

[0003] The remote operation system controls the motion of the robot based on the handwriting input information input to the interface. Summary of the invention

[0004] The robot's actions include moving the robot's viewpoint, moving the robot toward a destination, and holding an object. It is difficult to distinguish the types of actions using handwritten input information, which may impair the convenience of the system.

[0005] The present disclosure is an invention made to solve such a problem, and an object of the present disclosure is to provide a control system, a control method, and a program for switching an input mode of handwriting input information.

[0006] The control system in this embodiment determines the action of the robot based on the handwriting input information input to the interface and controls the action of the robot, wherein:

[0007] The control system comprises:

[0008] a handwriting input information accepting unit, which accepts input of the handwriting input information;

[0009] A switching unit switches between a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

[0010] The invention may further include a notification unit configured to notify a user who inputs the handwriting input information of a current input mode.

[0011] It may also be configured that the handwriting input information includes trajectory information inputted using a finger, a touch pen, a pointing device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0012] It can also be set that the handwriting input information is input with respect to a captured image of the environment in which the robot exists, and the multiple input modes include a first mode for changing the shooting direction of the captured image, a second mode for moving the robot, and a third mode for implementing a gripping action achieved by an end effector.

[0013] The switching unit may switch to a mode other than the second mode when the captured image does not show an area in which the robot can move, and may switch to a mode other than the third mode when the end effector is holding an object.

[0014] The switching unit may switch the plurality of input modes in response to a selection operation.

[0015] The handwriting input information may be input with respect to a captured image of an environment in which the robot exists, and the switching unit may input the captured image into a learning model and switch to a mode indicated by information output from the learning model.

[0016] The control method in this embodiment determines the action of the robot based on the handwriting input information input to the interface and controls the action of the robot, wherein:

[0017] The control method comprises:

[0018] a step of accepting input of the handwriting input information for the captured image;

[0019] A step of switching a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

[0020] The computer program product in this embodiment causes a computer to execute a control method for determining the motion of a robot based on handwriting input information input to an interface and controlling the motion of the robot, wherein:

[0021] The control method comprises:

[0022] a step of accepting input of the handwriting input information for the captured image;

[0023] A step of switching a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

[0024] The present disclosure can provide a control system, a control method, and a program for switching an input mode of handwriting input information.

[0025] The above and other objects, features and advantages of the present disclosure will be more fully understood from the detailed description given below and the accompanying drawings which are given as illustrations only and thus should not be regarded as limiting the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a conceptual diagram showing an example of an overall environment using the control system according to the first embodiment.

[0027] Figure 2 A diagram showing an example of handwriting input information.

[0028] Figure 3 A diagram showing an example of handwriting input information.

[0029] Figure 4 This is a perspective view showing an example of the appearance structure of a robot.

[0030] Figure 5 A block diagram showing an example of a block structure of a robot.

[0031] Figure 6 A block diagram showing an example of a block structure of a remote terminal.

[0032] Figure 7 A diagram showing an example of a display screen of a remote terminal.

[0033] Figure 8 This is a flowchart showing an example of the operation of the control system according to the first embodiment. DETAILED DESCRIPTION

[0034] Implementation Method 1

[0035] Hereinafter, the present invention will be described by way of the embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. In addition, not all of the configurations described in the embodiments are necessarily required as means for solving the problems.

[0036] Figure 1 1 is a conceptual diagram showing an example of an overall environment using the control system 10 according to Embodiment 1. The robot 100 that performs various actions in a first environment is remotely operated via a system server 500 connected to the Internet 600 by operating a remote terminal 300 (operation terminal) by a user who is a remote operator in a second environment separate from the first environment.

[0037] The robot 100 is connected to the Internet 600 via the wireless router 700 in the first environment. In addition, the remote terminal 300 is connected to the Internet 600 via the wireless router 700 in the second environment. The robot 100 performs a gripping motion by the hand 124 according to an operation of the remote terminal 300.

[0038] The robot 100 captures the first environment in which the robot 100 exists by means of the stereo camera 131 (capturing unit), and transmits the captured images to the remote terminal 300 via the Internet 600. In addition, the robot 100 identifies the graspable object that the hand 124 can grasp based on the captured images. In the first environment, there is a graspable object 401 such as a can. In addition, the shape of the graspable object 401 is not limited to a cylindrical shape.

[0039] The remote terminal 300 is, for example, a tablet terminal, and has a display panel 341 on which a touch panel is overlapped. The captured image received from the robot 100 is displayed on the display panel 341, and the user can indirectly visually confirm the first environment in which the robot 100 exists. In addition, the user can input handwriting input information in a handwritten manner for the captured image displayed on the display panel 341. The handwriting input information, for example, indicates the rotation of the imaging unit, the movement path of the robot 100, the gripping object or gripping direction of the hand 124, etc. Although there is a method of touching the portion of the captured image with the user's finger or a touch pen on the touch panel overlapped with the display panel 341 as an input method of the handwriting input information, it is not limited to this. The handwriting input information may also be trajectory information input using a pointing device such as a mouse. In addition, the handwriting input information may be trajectory information of a part of the user's body (eg, a finger) input into a three-dimensional input space using an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0040] Figure 2 as well as Figure 3 3 is a diagram showing an example of handwriting input information input with respect to the captured image 310 . Figure 2 The example of 901 represents handwriting input information 901 that simulates holding the holding object 401 from the side. Figure 3The example shows handwritten input information 902 that simulates the moving path of the robot 100. In addition, although the illustration is omitted, handwritten input information that simulates the rotation of the shooting unit can also be input. For example, when the touch pen or the like is moved up, down, left, and right on the captured image 310, the viewpoint of the shooting unit will follow this direction. The robot 100 can change the viewpoint by rotating the head or by turning on the spot. For example, it can be set that when the shooting unit is rotated in the direction of lens pan, the robot 100 turns on the spot, and when the shooting unit is rotated in the direction of lens pitch, the head of the robot 100 is tilted. In addition, handwritten text information can also be included in the handwritten input information. The handwritten input information input by the user for the captured image is sent to the robot 100 via the Internet 600.

[0041] The control system 10 switches between multiple input modes for making the robot 100 perform different types of actions. The multiple input modes may include, for example, a first mode for changing the imaging direction of the imaging unit, a second mode for moving the robot 100, and a third mode for making the robot 100 perform a gripping action.

[0042] Figure 4 The figure is a perspective view showing an example of the appearance structure of the robot 100. The robot 100 is roughly composed of a trolley part 110 and a main body part 120. The trolley part 110 supports two driving wheels 111 and a caster 112 which are respectively grounded to the running surface in a cylindrical frame. The two driving wheels 111 are arranged in a manner that the rotation axes are consistent with each other. Each driving wheel 111 is independently driven to rotate by a motor not shown. The caster 112 is a driven wheel, and is configured to axially support the wheel in a manner that the rotation axis extending in the vertical direction from the trolley part 110 is separated from the rotation axis of the wheel, and follows the moving direction of the trolley part 110 in a manner that imitates the moving direction of the trolley part 110.

[0043] The carriage 110 is provided with a laser scanner 133 on the peripheral portion of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane at each step angle and outputs whether there are obstacles in each direction. Furthermore, if there are obstacles, the laser scanner 133 outputs the distance to the obstacle.

[0044] The main body 120 mainly includes a body 121 mounted on the upper surface of the carriage 110, a head 122 placed on the upper surface of the body 121, an arm 123 supported by the side of the body 121, and a hand 124 provided at the top end of the arm 123. The arm 123 and the hand 124 are driven by a motor not shown in the figure, and hold the object to be held. The body 121 is driven by a motor not shown in the figure, and holds the object to be held. The body 121 can rotate around the vertical axis relative to the carriage 110 by the driving force of the motor not shown in the figure.

[0045] The head 122 mainly includes a stereo camera 131 and a display panel 141. The stereo camera 131 has a structure in which two camera units having the same viewing angle are arranged separately from each other, and outputs a photographing signal obtained by each camera unit.

[0046] The display panel 141 is, for example, a liquid crystal display, and displays the face of a set character using animation, or displays information related to the robot 100 using text or icons. If the face of a character is displayed on the display panel 141, it is possible to give the impression that the display panel 141 is a suspicious face to people around.

[0047] The head 122 can rotate around the vertical axis relative to the body 121 by the driving force of a motor (not shown). Therefore, the stereo camera 131 can shoot in any direction, and the display panel 141 can display content in any direction.

[0048] Figure 5 1 is a block diagram showing an example of a block structure of the robot 100. Although the main elements related to the control of the motion based on the handwriting input information are described here, the robot 100 also has other elements as a structure, and other elements that contribute to the control of the motion based on the handwriting input information may be added.

[0049] The control unit 150 is, for example, a CPU, and is stored in, for example, a controller unit provided in the body 121. The carriage drive unit 145 includes a drive wheel 111 and a drive circuit or a motor for driving the drive wheel 111. The control unit 150 controls the rotation of the drive wheel by sending a drive signal to the carriage drive unit 145. In addition, the control unit 150 receives a feedback signal from an encoder or the like from the carriage drive unit 145, thereby controlling the moving direction and moving speed of the carriage unit 110.

[0050] The upper body drive unit 146 includes an arm 123 and a hand 124, a body 121 and a head 122, and a drive circuit or a motor for driving these components. The control unit 150 sends a drive signal to the upper body drive unit 146 to implement the holding action and gesture. In addition, the control unit 150 receives a feedback signal from an encoder or the like from the upper body drive unit 146, thereby grasping the position and movement speed of the arm 123 and the hand 124, and the direction and rotation speed of the body 121 and the head 122.

[0051] The display panel 141 receives and displays the image signal generated by the control unit 150. In addition, the control unit 150 may generate an image signal of a character or the like as described above and display it on the display panel 141.

[0052] The stereo camera 131 captures the first environment where the robot 100 exists in response to a request from the control unit 150, and transmits a capture signal to the control unit 150. The control unit 150 performs image processing using the capture signal, or converts the capture signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether there is an obstacle in the moving direction in response to a request from the control unit 150, and transmits a detection signal as a detection result to the control unit 150.

[0053] The hand camera 135 is, for example, a distance image sensor, which is used to identify the distance, shape, direction, etc. of the grasped object. The hand camera 135 includes a shooting element in which pixels are arranged in a two-dimensional shape for photoelectric conversion of an optical image incident from the object space, and outputs the distance to the object to the control unit 150 for each pixel. Specifically, the hand camera 135 includes an irradiation unit that irradiates patterned light to the object space, and uses a shooting element to receive the reflected light, and outputs the distance of each pixel from the captured object according to the deformation or size of the pattern in the image. In addition, the control unit 150 uses the stereo camera 131 to grasp the conditions of the larger surrounding environment, and uses the hand camera 135 to grasp the conditions near the grasped object.

[0054] The memory 180 is a non-volatile storage medium, for example, a solid state drive can be used. In addition to the control program for controlling the robot 100, the memory 180 also stores various parameter values, functions, lookup tables, etc. used in control and calculation. The memory 180 can also store a learned model that uses an image of handwritten input information as an input image and outputs the meaning of the gripping action simulated by the handwritten input information.

[0055] The communication unit 190 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 190 receives handwriting input information transmitted from the remote terminal 300, and transfers it to the control unit 150. In addition, the communication unit 190 transmits the captured image captured by the stereo camera 131 to the remote terminal 300 under the control of the control unit 150.

[0056] The control unit 150 executes the control unit program read from the memory 180 to perform overall control of the robot 100 and various calculations. In addition, the control unit 150 also plays the role of a function execution unit that performs various calculations and controls related to the control. As such a function execution unit, the control unit 150 includes a shooting control unit 151, a movement control unit 152, and a gripping control unit 153. In addition, at least a part of the function of the switching unit 352 described later can also be executed by using the control unit 150.

[0057] When the input mode is the first mode, the imaging control unit 151 changes the imaging direction of the stereo camera 131 based on the handwriting input information. For example, the imaging control unit 151 may rotate the carriage unit 110 and the head 122 in the direction in which the touch pen or finger is moved.

[0058] When the input mode is the second mode, the movement control unit 152 moves the carriage unit 110 based on the handwriting input information. For example, when the handwriting input information includes a linear figure, the movement control unit 152 generates a trajectory passing through points on the line. Then, the movement control unit 152 may send a drive signal to the carriage drive unit 145 to move the carriage unit 110 along the trajectory.

[0059] When the input mode is the third mode, the grip control unit 153 grips the grip object based on the handwriting input information. The grip control unit 153 recognizes the grip object reflected in the captured image using, for example, a learned model for recognition, and generates a trajectory of the hand 124 by gripping the grip object in a direction corresponding to the handwriting input information. Then, the grip control unit 153 may also send a drive signal corresponding to the generated trajectory to the upper body drive unit 146.

[0060] In addition, when the input mode is the fourth mode, the grip control unit 153 may also place the grip object on a workbench or a floor, etc. based on the handwriting input information. The grip control unit 153 generates a trajectory of the hand 124 in a manner such that the grip object is placed at a position specified by the handwriting input information. Then, the grip control unit 153 may also send a drive signal corresponding to the generated trajectory to the upper body drive unit 146.

[0061] Figure 6 1 is a block diagram showing an example of a block structure of the remote terminal 300. Although the main elements related to the process of inputting handwriting input information with respect to the captured image received from the robot 100 are described here, the remote terminal 300 also has other elements as a structure, and other elements that contribute to the process of inputting handwriting input information may be added.

[0062] The control unit 350 is, for example, a CPU, and executes the control program read from the memory 380 to perform the overall control of the remote terminal 300 and various calculations. In addition, the control unit 350 also plays the role of a function execution unit that performs various calculations and controls related to the control. As such a function execution unit, the control unit 350 includes a handwriting input information receiving unit 351, a switching unit 352, and a notification unit 353.

[0063] Handwriting input information accepting unit 351 causes display panel 341 to display the captured image, and accepts input of handwriting input information from input unit 342. Handwriting input information accepting unit 351 may also accept input into a three-dimensional space. The captured image may also be a three-dimensional image.

[0064] The switching unit 352 switches between a plurality of input modes for causing the robot 100 to perform different types of actions. The switching unit 352 may also switch between a plurality of input modes according to a selection operation. The selection operation may be performed, for example, by a user who operates the remote terminal 300 or by a person who supports the user around the user. The switching unit 352, for example, causes the display panel 141 to display buttons corresponding to each mode. Furthermore, when an area corresponding to each button on the touch panel is pressed, the switching unit 352 switches to a mode corresponding to the button. In addition, the switching unit 352 may also select a mode corresponding to a voice input from the user.

[0065] In addition, the switching unit 352 may also input the captured image into the learned model and switch to the mode represented by the information output from the learned model. The learned model is constructed, for example, by supervised learning using teaching data including a captured image and a label representing an appropriate input mode corresponding to the captured image. For example, when a captured image reflecting an area where the robot 100 can move (for example, an area where there are no obstacles) is input, the learned model outputs information representing the second mode. For example, when a captured image reflecting an object to be grasped is input, the learned model outputs information representing the third mode. For example, when a captured image reflecting only a portion of an object to be grasped is input, the learned model may also output information representing the first mode. In addition, the switching unit 352 may also select a mode using a rule base method.

[0066] The switching unit 352 may also determine whether an area in which the robot 100 can move is shown in the captured image, and switch to a mode other than the second mode if an area in which the robot 100 can move is not shown. The analysis of the captured image may be performed by the robot 100 or by the remote terminal 300. For example, the switching unit 352 may also set a button corresponding to the second mode to an inactive state.

[0067] The switching unit 352 may also determine whether the robot 100 is holding the gripping object, and switch to a mode other than the third mode when the hand 124 of the robot 100 is not holding the gripping object. The switching unit 352 may also receive information indicating whether the gripping object is being held from the robot 100, and make a determination based on the received information. The robot 100 can determine whether the gripping object is being held based on the detection result of a sensor (e.g., a hand camera) provided on the hand 124. The switching unit 352 may also set a button corresponding to the third mode to an inactive state.

[0068] As described above, the switch unit 352 may be provided on the robot 100 side instead of the remote terminal 300 side. For example, when the input mode is switched using a learning model, the switch unit 352 may be provided in the control unit 150 of the robot 100 .

[0069] The notification unit 353 notifies the user who operates the remote terminal 300 of the current input mode. The notification unit 353 may display information (e.g., an icon) indicating the current input mode on the display panel 341, or may output voice from a speaker (not shown). Thus, the user can input handwriting input information after confirming the current input mode.

[0070] The display panel 341 is, for example, a liquid crystal panel, and displays, for example, a captured image transmitted from the robot 100 .

[0071] The input unit 342 includes a touch panel overlapped with the display panel 141, and a push button provided at the edge of the display panel 141. The input unit 342 transfers the handwriting input information to the control unit 350. Examples of the handwriting input information are as follows: Figures 2 to 3 As shown.

[0072] The memory 380 is a non-volatile storage medium, for example, a solid state drive, and stores various parameter values, functions, lookup tables, etc. used in control and calculation, in addition to a control program for controlling the remote terminal 300 .

[0073] The communication unit 390 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 390 receives the captured image sent from the robot 100 and transfers it to the control unit 350. In addition, the communication unit 390 cooperates with the control unit 350 to send handwriting input information to the robot 100.

[0074] Figure 7 34 is an explanatory diagram showing an example of a display screen displayed on the display panel 341. The illustrated display screen 20 includes a mode selection button 22, a viewer 23, and a command button 24.

[0075] The mode selection button 22 includes a button 221 corresponding to the first mode, a button 222 corresponding to the second mode, and a button 223 corresponding to the third mode. The mode selection button may also include a button corresponding to the fourth mode (not shown). For example, when the button 222 is selected, the button 222 may be displayed in a different manner (e.g., color) from the buttons 221 and 223. In the viewer 23, a captured image obtained by the stereo camera 131 is displayed. In the captured image, for example, a grip object 401 and an obstacle 402 may also be reflected. The command button 24 displays information indicating the mode currently being executed. For example, when the second mode is selected, a text such as "Moving" or an animation indicating the movement of the robot 100 is displayed. The notification unit 353 can notify the current mode through the command button 24, etc.

[0076] Next, an example of the operation of the control system 10 according to the first embodiment is described. First, the switching unit 352 of the control unit 350 of the remote terminal 300 determines whether the region where the robot 100 can move is shown in the captured image and whether the hand 124 of the robot 100 is holding a grasping object, and determines a mode candidate according to the determination result (step S101). When the region where the robot 100 can move is not shown in the captured image, the switching unit 352 sets a mode other than the second mode as a mode candidate. When the hand 124 is holding a grasping object, the switching unit 352 sets a mode other than the third mode as a mode candidate. When the region where the robot 100 can move is not shown in the captured image and the hand 124 is holding a grasping object, the switching unit 352 sets a mode other than the second mode and the third mode as a mode candidate. The switching unit 352 sets a button corresponding to the mode candidate to an active state, and sets a button corresponding to other than the mode candidate to an inactive state.

[0077] Next, the control unit 350 determines whether a selection operation has been performed (step S102 ). If a selection operation has not been performed (“No” in step S102 ), the process returns to step S102 .

[0078] When a selection operation is performed ("Yes" in step S102), the switching unit 352 of the control unit 350 switches to the first mode (step S103), switches to the second mode (step S104), switches to the third mode (step S105), or switches to the fourth mode (step S106) according to the selection operation.

[0079] When switched to the first mode, the control unit 350 receives a change instruction of the imaging direction of the stereo camera 131 according to the handwriting input information, and the imaging control unit 151 of the robot 100 changes the imaging direction of the stereo camera 131 according to the change instruction (step S107 ).

[0080] When the mode is switched to the second mode, the control unit 350 receives a movement instruction of the carriage unit 110 according to the handwriting input information, and the movement control unit 152 of the robot 100 moves the carriage unit 110 according to the movement instruction (step S108 ).

[0081] When the mode is switched to the third mode, the control unit 350 receives the instruction of the gripping operation corresponding to the handwriting input information, and the gripping control unit 153 of the robot 100 grips the gripping object according to the instruction of the gripping operation (step S109 ).

[0082] When the mode is switched to the fourth mode, the control unit 350 receives a placement instruction of the grip object corresponding to the handwriting input information, and the grip control unit 153 of the robot 100 places the grip object at a position corresponding to the placement instruction (step S110 ).

[0083] Then, the control system 10 determines whether the remote operation is terminated (step S111). If the remote operation is not terminated ("No" in step S111), the process returns to step S101, and if the remote operation is terminated ("Yes" in step S111), the process ends.

[0084] In addition, instead of steps S101 to S106 , a step of switching the input mode using a learning model or the like may be provided.

[0085] Next, the effects achieved by the control system 10 involved in embodiment 1 are described. In addition to the operation of objects, the actions performed by the robot also include changes in viewpoint and movement of the trolley, and these actions are combined to achieve meaningful tasks. However, in the prior art, there is a problem that it is impossible to distinguish whether a certain handwritten input information is an instruction related to the operation of an object or an instruction related to the movement of the trolley. In a system that accepts free input as handwritten input text (such as a line), when the tasks performed by the robot become complex and diverse, how to interpret the meaning of the input handwritten input information becomes particularly important.

[0086] Since the control system 10 according to the first embodiment is configured to switch the input mode of handwriting input information, the handwriting input characters can be easily interpreted, thereby improving the convenience of the system.

[0087] In addition, the present invention is not limited to the above-mentioned embodiment, and can be appropriately changed within the scope of the main purpose. Although in the above-mentioned embodiment, the robot 100 and the remote terminal 300 exchange the captured image and handwriting input information via the Internet 600 and the system server 500, it is not limited to this. The robot 100 and the remote terminal 300 may also exchange the captured image and handwriting input information through direct communication.

[0088] In addition, although in the above-mentioned embodiment, the case where the multiple input modes are the first mode to the third mode or the first mode to the fourth mode is described, it is not limited to these modes. The multiple modes may also include any mode other than the first mode to the fourth mode. In addition, the multiple input modes may also not include part or all of the first mode to the fourth mode.

[0089] In addition, although the stereo camera 131 is used in the above embodiment, the present invention is not limited thereto. The robot 100 may also include any imaging unit installed in any place in the first environment. The imaging unit is not limited to a stereo camera, and may be a monocular camera or the like.

[0090] In addition, although in the above embodiment, the robot 100 is described as an example of a hand 124 at the top of the arm 123 as an end effector, it is not limited to this. The robot 100 only needs to have an end effector and use the end effector to perform a gripping action. In addition, the end effector may also be a gripping part other than a hand (for example, a suction part, etc.).

[0091] The program includes a command group (or software code) for causing the computer to perform one or more functions described in the embodiment when read by the computer. The program may also be stored in a non-temporary computer-readable medium or a storage medium having an entity. As a non-limiting example, the computer-readable medium or the storage medium having an entity includes a random-access memory (RAM), a read-only memory (ROM), a cache memory, a solid-state drive (SSD) or other memory technology, a read-only compact disk (CD-ROM), a digital versatile disc (DVD), a Blu-ray (registered trademark) disc or other optical storage, a cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device. The program may also be sent on a temporary computer-readable medium or a communication medium. As a non-limiting example, a temporary computer-readable medium or a communication medium includes a propagation signal in an electrical, optical, acoustic or other form.

[0092] It is clear from the disclosure described in this way that the disclosed embodiments can be changed in many ways. Obviously such changes should not be regarded as departing from the spirit and scope of the present invention, and for those skilled in the art, all such changes should be included in the appended claims.

Claims

1. A control system that determines the motion of a robot based on handwriting input information input to an interface and controls the motion of the robot, wherein: The control system comprises: a handwriting input information accepting unit, which accepts input of the handwriting input information; A switching unit switches between a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

2. The control system according to claim 1, wherein: The device further includes a notification unit configured to notify a user who inputs the handwriting input information of a current input mode.

3. The control system according to claim 1 or 2, wherein: The handwriting input information includes trajectory information input using a finger, a touch pen, a pointing device, an augmented reality device, a virtual reality device, or a mixed reality device.

4. The control system according to claim 1 or 2, wherein: The handwriting input information is input with respect to a photographed image of an environment in which the robot exists, The plurality of input modes include a first mode for changing a photographing direction of the photographed image, a second mode for moving the robot, and a third mode for performing a gripping action by an end effector.

5. The control system according to claim 4, wherein: The switching unit switches to a mode other than the second mode when the region in which the robot can move is not shown in the captured image, and switches to a mode other than the third mode when the end effector is gripping an object to be gripped.

6. The control system according to claim 1 or 2, wherein: The switching unit switches the plurality of input modes according to a selection operation.

7. The control system of claim 1, wherein: The handwriting input information is input with respect to a photographed image of an environment in which the robot exists, The switching unit inputs the captured image into a learning model, and switches to a mode indicated by information output from the learning model.

8. A control method, which determines the action of a robot based on handwriting input information input to an interface and controls the action of the robot, wherein: The control method comprises: a step of accepting input of the handwriting input information for the captured image; A step of switching a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

9. A computer program product that causes a computer to execute a control method for determining a robot's motion based on handwriting input information input to an interface and controlling the robot's motion, wherein: The control method comprises: a step of accepting input of the handwriting input information for the captured image; A step of switching a plurality of input modes of the handwriting input information for causing the robot to perform different types of actions.

Images