Control system, control method, and computer program product

Abstract

The invention provides a control system, a control method and a computer program product for interacting with a user with regard to correction of handwritten input information. A control system (10) determines the operation of a robot on the basis of handwriting input information input to an interface, and controls the operation of the robot. A control system (10) is provided with: a handwritten input information reception unit (351) that receives input of handwritten input information; a determination unit (352) that determines whether or not it is necessary to correct the handwriting input information on the basis of the performability or efficiency of an operation based on the handwriting input information; and a presentation unit (353) that presents to the user information pertaining to the correction of the handwriting input information when it is necessary to correct the handwriting input information.

Description

Technical Field

[0001] The present disclosure relates to a control system, a control method, and a program. Background Art

[0002] Japanese Patent Application Laid-Open No. 2021-094605 discloses a remote operation system.

[0003] The remote operation system controls the actions of the robot based on the handwriting input information input to the interface. Summary of the invention

[0004] Due to various factors such as limitations on the robot's movements or the presence of obstacles, the robot may not be controlled based on handwriting input information. In addition, there may be more efficient and appropriate movements than those based on handwriting input information. In such cases, the robot's movements intended by the user may differ from the actual movements of the robot, and the operability may be reduced for the user.

[0005] The present disclosure is an invention made to solve such a problem, and an object of the present disclosure is to provide a control system, a control method, and a program for interacting with a user regarding correction of handwriting input information.

[0006] The control system in this embodiment determines the action of the robot based on the handwriting input information input to the interface and controls the action of the robot, wherein:

[0007] The control system comprises:

[0008] a handwriting input information accepting unit, which accepts input of the handwriting input information;

[0009] a determination unit configured to determine whether the handwriting input information needs to be corrected based on the feasibility or efficiency of the action based on the handwriting input information;

[0010] A display unit displays information related to the correction of the handwriting input information to the user when the handwriting input information needs to be corrected.

[0011] It may also be configured that the handwriting input information includes trajectory information inputted using a finger, a touch pen, a pointing device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0012] It may also be configured that, when the handwriting input information needs to be corrected, the display unit corrects the handwriting input information and displays the corrected handwriting input information to the user.

[0013] When the handwriting input information needs to be corrected, the presentation unit may present information on a portion of the robot operating based on the handwriting input information that interferes with an obstacle to the user.

[0014] It can also be set that the judgment unit judges that the handwriting input information needs to be corrected when part or all of the handwriting input information is included in a predetermined area of ​​the input screen, and the handwriting input information acceptance unit displays the predetermined area and the area outside the predetermined area in a recognizable manner when displaying the input screen.

[0015] The handwriting input information accepting unit may set the thickness of a line included in the handwriting input information based on the size of the robot when displaying the input handwriting input information.

[0016] The display unit may correct the position of the handwriting input information using segmentation information obtained based on machine learning.

[0017] The control method in this embodiment determines the action of the robot based on the handwriting input information input to the interface, and controls the action of the robot, wherein:

[0018] The control method comprises:

[0019] The step of accepting the input of the handwriting input information;

[0020] A step of determining whether the handwriting input information needs to be corrected according to the executability or efficiency of an action based on the handwriting input information;

[0021] In the case where the handwriting input information needs to be corrected, the step of presenting information related to the correction of the handwriting input information to the user.

[0022] The computer program product in this embodiment causes a computer to execute a control method for determining the action of a robot based on handwriting input information input to an interface and controlling the action of the robot, wherein:

[0023] The control method comprises:

[0024] The step of accepting the input of the handwriting input information;

[0025] A step of determining whether the handwriting input information needs to be corrected according to the executability or efficiency of an action based on the handwriting input information;

[0026] In the case where the handwriting input information needs to be corrected, the step of presenting information related to the correction of the handwriting input information to the user.

[0027] Through the present disclosure, it is possible to provide a control system, a control method, and a program for interacting with a user regarding correction of handwriting input information.

[0028] The above and other objects, features and advantages of the present disclosure will be more fully understood from the detailed description given below and the accompanying drawings which are given as illustrations only and thus should not be regarded as limiting the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a conceptual diagram showing an example of an overall environment using the control system according to the first embodiment.

[0030] Figure 2 A diagram showing an example of handwriting input information.

[0031] Figure 3 A diagram showing an example of handwriting input information.

[0032] Figure 4 This is a perspective view showing an example of the appearance structure of a robot.

[0033] Figure 5 A block diagram showing an example of a block structure of a robot.

[0034] Figure 6 A block diagram showing an example of a block structure of a remote terminal.

[0035] Figure 7 A diagram for explaining an example of a captured image to be displayed.

[0036] Figure 8 A diagram for explaining an example of a captured image to be displayed.

[0037] Fig. 9 A diagram for explaining an example of display information.

[0038] Fig.10 A diagram for explaining an example of display information.

[0039] Fig.11 A diagram for explaining an example of a method for correcting handwriting input information.

[0040] Fig.12 A diagram for explaining an example of a method for correcting handwriting input information.

[0041] Fig.13 A diagram for explaining an example of a method for correcting handwriting input information.

[0042] Fig.14 This is a flowchart showing an example of the operation of the control system according to the first embodiment. DETAILED DESCRIPTION

[0043] Implementation Method 1

[0044] Hereinafter, the present invention will be described by way of the embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. In addition, not all of the configurations described in the embodiments are necessarily required as means for solving the problems.

[0045] Figure 1 1 is a conceptual diagram showing an example of an overall environment using the control system 10 according to Embodiment 1. The robot 100 that performs various actions in a first environment is remotely operated via a system server 500 connected to the Internet 600 by operating a remote terminal 300 (operation terminal) by a user who is a remote operator in a second environment separate from the first environment.

[0046] The robot 100 is connected to the Internet 600 via the wireless router 700 in the first environment. In addition, the remote terminal 300 is connected to the Internet 600 via the wireless router 700 in the second environment. The robot 100 performs a gripping motion by the hand 124 according to an operation of the remote terminal 300.

[0047] The robot 100 captures the first environment in which the robot 100 exists by means of the stereo camera 131 (capturing unit), and transmits the captured images to the remote terminal 300 via the Internet 600. In addition, the robot 100 identifies the graspable object that the hand 124 can grasp based on the captured images. In the first environment, there is a graspable object 401 such as a can. In addition, the shape of the graspable object 401 is not limited to a cylindrical shape.

[0048] The remote terminal 300 is, for example, a tablet terminal, and has a display panel 341 on which a touch panel is overlapped. The captured image received from the robot 100 is displayed on the display panel 341, and the user can indirectly visually confirm the first environment in which the robot 100 exists. In addition, the user can input handwriting input information in a handwritten manner for the captured image displayed on the display panel 341. The handwriting input information, for example, indicates the rotation of the imaging unit, the movement path of the robot 100, the gripping object or gripping direction of the hand 124, etc. Although there is a method of touching the portion of the captured image using the user's finger or a touch pen on the touch panel overlapped with the display panel 341 as an input method for the handwriting input information, it is not limited to this. The handwriting input information may also include trajectory information input using a finger, a touch pen, or a pointing device (such as a mouse). In addition, the handwriting input information may also include trajectory information of a part of the user's body (eg, a finger) input into a three-dimensional input space using an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device.

[0049] Figure 2 as well as Figure 3 3 is a diagram showing an example of handwriting input information input with respect to the captured image 310 . Figure 2 The example of 901 represents handwriting input information 901 that simulates holding the holding object 401 from the side. Figure 3 The example shows handwritten input information 902 that simulates the moving path of the robot 100. In addition, although the illustration is omitted, handwritten input information that simulates the rotation of the shooting unit can also be input. For example, when the touch pen or the like is moved up, down, left, and right on the captured image 310, the viewpoint of the shooting unit will follow this direction. The robot 100 can change the viewpoint by rotating the head or by turning on the spot. For example, it can be set that when the shooting unit is rotated in the direction of lens pan, the robot 100 turns on the spot, and when the shooting unit is rotated in the direction of lens pitch, the head of the robot 100 is tilted. In addition, handwritten text information can also be included in the handwritten input information. The handwritten input information input by the user for the captured image is sent to the robot 100 via the Internet 600.

[0050] The control system 10 determines the action of the robot based on the handwriting input information input to the interface and controls the action of the robot. The control system 10 determines whether the handwriting input information needs to be corrected based on the feasibility or efficiency of the action based on the handwriting input information. In the case where the handwriting input information needs to be corrected, the control system 10 displays information related to the correction of the handwriting input information to the user. For example, the control system 10 can display the corrected handwriting input information or the information indicating the part that needs to be corrected. The control system 10 can cause the robot 100 to act based on the user's confirmation operation after the information is displayed, or it can display the information after the robot 100 acts based on the corrected handwriting input information.

[0051] Figure 4 The figure is a perspective view showing an example of the appearance structure of the robot 100. The robot 100 is roughly composed of a trolley part 110 and a main body part 120. The trolley part 110 supports two driving wheels 111 and a caster 112 which are respectively grounded to the running surface in a cylindrical frame. The two driving wheels 111 are arranged in a manner that the rotation axes are consistent with each other. Each driving wheel 111 is independently driven to rotate by a motor not shown. The caster 112 is a driven wheel, and is configured to axially support the wheel in a manner that the rotation axis extending in the vertical direction from the trolley part 110 is separated from the rotation axis of the wheel, and follows the moving direction of the trolley part 110 in a manner that imitates the moving direction of the trolley part 110.

[0052] The carriage 110 is provided with a laser scanner 133 on the peripheral portion of the upper surface. The laser scanner 133 scans a certain range in the horizontal plane at each step angle and outputs whether there are obstacles in each direction. Furthermore, if there are obstacles, the laser scanner 133 outputs the distance to the obstacle.

[0053] The main body 120 mainly includes a body 121 mounted on the upper surface of the carriage 110, a head 122 placed on the upper surface of the body 121, an arm 123 supported by the side of the body 121, and a hand 124 provided at the top end of the arm 123. The arm 123 and the hand 124 are driven by a motor not shown in the figure, and hold the object to be held. The body 121 is driven by a motor not shown in the figure, and holds the object to be held. The body 121 can rotate around the vertical axis relative to the carriage 110 by the driving force of the motor not shown in the figure.

[0054] The head 122 mainly includes a stereo camera 131 and a display panel 141. The stereo camera 131 has a structure in which two camera units having the same viewing angle are arranged separately from each other, and outputs a photographing signal obtained by each camera unit.

[0055] The display panel 141 is, for example, a liquid crystal display, and displays the face of a set character using animation, or displays information related to the robot 100 using text or icons. If the face of a character is displayed on the display panel 141, it is possible to give the impression that the display panel 141 is a suspicious face to people around.

[0056] The head 122 can rotate around the vertical axis relative to the body 121 by the driving force of a motor (not shown). Therefore, the stereo camera 131 can shoot in any direction, and the display panel 141 can display content in any direction.

[0057] Figure 5 1 is a block diagram showing an example of a block structure of the robot 100. Although the main elements related to the control of the motion based on the handwriting input information are described here, the robot 100 also has other elements as a structure, and other elements that contribute to the control of the motion based on the handwriting input information may be added.

[0058] The control unit 150 is, for example, a CPU, and is stored in, for example, a controller unit provided in the body 121. The carriage drive unit 145 includes a drive wheel 111 and a drive circuit or a motor for driving the drive wheel 111. The control unit 150 controls the rotation of the drive wheel by sending a drive signal to the carriage drive unit 145. In addition, the control unit 150 receives a feedback signal from an encoder or the like from the carriage drive unit 145, thereby controlling the moving direction and moving speed of the carriage unit 110.

[0059] The upper body drive unit 146 includes an arm 123 and a hand 124, a body 121 and a head 122, and a drive circuit or a motor for driving these components. The control unit 150 sends a drive signal to the upper body drive unit 146 to implement the holding action and gesture. In addition, the control unit 150 receives a feedback signal from an encoder or the like from the upper body drive unit 146, thereby grasping the position and movement speed of the arm 123 and the hand 124, and the direction and rotation speed of the body 121 and the head 122.

[0060] The display panel 141 receives and displays the image signal generated by the control unit 150. In addition, the control unit 150 may generate an image signal of a character or the like as described above and display it on the display panel 141.

[0061] The stereo camera 131 captures the first environment where the robot 100 exists in response to a request from the control unit 150, and transmits a capture signal to the control unit 150. The control unit 150 performs image processing using the capture signal, or converts the capture signal into a captured image in accordance with a predetermined format. The laser scanner 133 detects whether there is an obstacle in the moving direction in response to a request from the control unit 150, and transmits a detection signal as a detection result to the control unit 150.

[0062] The hand camera 135 is, for example, a distance image sensor, which is used to identify the distance, shape, direction, etc. of the grasped object. The hand camera 135 includes a shooting element in which pixels are arranged in a two-dimensional shape for photoelectric conversion of an optical image incident from the object space, and outputs the distance to the object to the control unit 150 for each pixel. Specifically, the hand camera 135 includes an irradiation unit that irradiates patterned light to the object space, and uses a shooting element to receive the reflected light, and outputs the distance of each pixel from the captured object according to the deformation or size of the pattern in the image. In addition, the control unit 150 uses the stereo camera 131 to grasp the conditions of the larger surrounding environment, and uses the hand camera 135 to grasp the conditions near the grasped object.

[0063] The memory 180 is a non-volatile storage medium, for example, a solid state drive can be used. In addition to the control program for controlling the robot 100, the memory 180 also stores various parameter values, functions, lookup tables, etc. used in control and calculation. The memory 180 can also store a learned model that uses an image of handwritten input information as an input image and outputs the meaning of the gripping action simulated by the handwritten input information.

[0064] The communication unit 190 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 190 receives handwriting input information transmitted from the remote terminal 300, and transfers it to the control unit 150. In addition, the communication unit 190 transmits the captured image captured by the stereo camera 131 to the remote terminal 300 under the control of the control unit 150.

[0065] The control unit 150 executes the control unit program read from the memory 180 to perform overall control of the robot 100 and various calculations. In addition, the control unit 150 also serves as a function execution unit that executes various calculations and controls related to the control. As such a function execution unit, the control unit 150 includes a shooting control unit 151, a movement control unit 152, and a gripping control unit 153. In addition, at least part of the functions of the judgment unit 352 and the display unit 353 described later may also be included in the control unit 150.

[0066] Based on the handwriting input information, the imaging control unit 151 changes the imaging direction of the stereo camera 131. For example, the imaging control unit 151 may rotate the carriage unit 110 and the head 122 in the direction in which the touch pen or the finger is moved.

[0067] The movement control unit 152 moves the carriage unit 110 based on the handwriting input information. For example, when the handwriting input information includes a line, the movement control unit 152 generates a trajectory passing through points on the line. Then, the movement control unit 152 may send a drive signal to the carriage drive unit 145 to move the carriage unit 110 along the trajectory.

[0068] The grip control unit 153 grips the grip object based on the handwriting input information. The grip control unit 153 recognizes the grip object reflected in the captured image using, for example, a learned model for recognition, and generates a trajectory of the hand 124 by gripping the grip object in a direction corresponding to the handwriting input information. Then, the grip control unit 153 may also send a drive signal corresponding to the generated trajectory to the upper body drive unit 146.

[0069] In addition, the grip control unit 153 may also place the grip object on a workbench or a floor, etc. based on the handwriting input information. The grip control unit 153 generates a trajectory of the hand 124 in a manner such that the grip object is placed at a position specified by the handwriting input information. Then, the grip control unit 153 may also send a drive signal corresponding to the generated trajectory to the upper body drive unit 146.

[0070] Figure 6 1 is a block diagram showing an example of a block structure of the remote terminal 300. Although the main elements related to the process of inputting handwriting input information with respect to the captured image received from the robot 100 are described here, the remote terminal 300 also has other elements as a structure, and other elements that contribute to the process of inputting handwriting input information may be added.

[0071] The control unit 350 is, for example, a CPU, which executes the control program read from the memory 380 to perform the overall control of the remote terminal 300 and various calculations. In addition, the control unit 350 also plays the role of a function execution unit that performs various calculations and controls related to the control. As such a function execution unit, the control unit 350 includes a handwriting input information receiving unit 351, a judgment unit 352, and a display unit 353.

[0072] The handwriting input information accepting unit 351 causes the display panel 341 to display the captured image, and accepts the input of handwriting input information from the input unit 342. In addition, the handwriting input information accepting unit 351 may also accept input into a three-dimensional space. The captured image may also be a three-dimensional image. In addition, the handwriting input information accepting unit 351 superimposes the image corresponding to the input handwriting input information on the captured image.

[0073] The handwriting input information receiving unit 351 may also display the predetermined area included in the captured image and the area outside the predetermined area in a recognizable manner. The predetermined area may be, for example, an obstacle (e.g., a table or a floor) that becomes an obstacle to the movement of the robot 100 or an area where the robot 100 is reflected. If part or all of the handwriting input information is included in the predetermined area, it is determined that the handwriting input information needs to be corrected.

[0074] Figure 7This is a diagram for explaining an example of a captured image 310 displayed by the handwriting input information receiving unit 351. In the captured image 310, an obstacle 402 that hinders the movement of the robot 100 is shown. In addition, the captured image 310 includes an area 311 where a part of the robot 100 is shown. The predetermined area described above includes an area where the obstacle 402 is shown, and the area 311. In the case where part or all of the handwriting input information representing the movement path of the robot 100 is included in the predetermined area, the robot 100 cannot move based on the handwriting input information. The area 312 outside the predetermined area is an area where handwriting input information can be input. Therefore, the handwriting input information receiving unit 351 displays the area 312 and the area outside the area 312 in different colors. Thereby, the user can be prompted to input the handwriting input information in an appropriate area. The handwriting input information receiving unit 351 can also add a specific light color to the area 312. The handwriting input information receiving unit 351 can also improve the recognition accuracy of the predetermined area and the area 312 based on the segmentation information. The segmentation information is information indicating the result of segmenting the captured image 310 using a learning model, etc. In addition, the handwriting input information accepting unit 351 may change the capturing direction when the capturing unit is directed upward or downward from the predetermined range when the user starts inputting handwriting input information.

[0075] Furthermore, when displaying the handwriting input information on the captured image, the handwriting input information accepting unit 351 may set the thickness of the line included in the handwriting input information based on the size of the robot 100. Thus, the user can input appropriate handwriting input information in consideration of the size of the robot 100. Figure 8 This is a diagram for explaining an example of a captured image 310 displayed by the handwriting input information receiving unit 351. In the captured image 310, handwriting input information 902a indicating the movement path of the robot 100 is displayed. The thickness of the line in the handwriting input information 902a is set according to the size of the robot 100. Since an object farther from the robot 100 appears smaller in the captured image 310, the thickness of the line corresponding to a position farther from the current position of the robot 100 is set to be thinner. Thus, even for a position farther from the current position of the robot 100, appropriate handwriting input information can be input.

[0076] Back to Figure 6, the judgment unit 352 judges whether the handwriting input information needs to be corrected based on the feasibility or efficiency of the action based on the handwriting input information. For example, when part or all of the handwriting input information representing the movement path of the robot 100 is included in the predetermined area, the action based on the handwriting input information cannot be performed. In addition, when the stability of the holding action of the robot 100 based on the handwriting input information is low, the action based on the handwriting input information cannot be performed. In such a case, the judgment unit 352 judges that the handwriting input information needs to be corrected. In addition, it can also be set that the judgment unit 352 judges that the handwriting input information needs to be corrected not only when the action cannot be performed, but also when the efficiency of the action is low.

[0077] When the handwriting input information needs to be corrected, the display unit 353 displays information related to the correction of the handwriting input information to the user. Thus, the remote terminal 300 can perform interaction related to the correction of the handwriting input information with the user. The display unit 353 can also prompt the user to correct the handwriting input information by displaying information to the user. The correction of the handwriting input information can also be a way of inputting appropriate handwriting input information. In addition, the display unit 353 can also correct the handwriting input information and display the corrected handwriting input information to the user.

[0078] Fig. 9 3 is a diagram for explaining an example of information displayed by the display unit 353. The handwritten input information 9021 indicated by the dotted line indicates a moving path where the robot 100 and the obstacle 402 may interfere. In this case, the judgment unit 352 judges that the handwritten input information 902 needs to be corrected. The display unit 353 determines the contact point P1 where the robot 100 and the obstacle 402 come into contact based on the segmentation information of the captured image 310 and the contact point information between the virtual bumper of the robot 100 and the obstacle 402. Then, the display unit 353 displays the handwritten input information 9022 indicating the moving path passing through the adjacent point P2 adjacent to the contact point P1. When a confirmation operation such as pressing a confirmation button is input, the display unit 353 may cause the robot 100 to perform an action based on the handwritten input information 9022.

[0079] Fig.10A diagram for explaining an example of information displayed by the display unit 353. The captured image 310 includes handwriting input information 901 indicating a gripping action performed by the robot 100. Area 314 indicates a portion of the hand 124 performing the action indicated by the handwriting input information 901 that interferes with an obstacle (floor). The display unit 353 displays the area 314 and areas other than the area 314 in a recognizable manner. The user can confirm the captured image 310 and input appropriate handwriting input information. The display unit 353 may also display the hand 124 performing an action based on the handwriting input information.

[0080] The display unit 353 may also correct the handwriting input information and cause the robot 100 to perform an action based on the corrected handwriting input information. The display unit 353 displays the corrected handwriting information to the user.

[0081] Reference Fig.11 , since the moving path represented by the handwriting input information 9021 indicated by the dotted line passes through the space where the gripping object 401 exists, the handwriting input information 9021 needs to be corrected. In this case, the display unit 353 corrects the handwriting input information 9021 to handwriting input information 9022 including a detour path that detours the gripping object 401. The detour path includes, for example, a point on the trajectory represented by the handwriting input information 9021 and an intersection point of a straight line passing through the viewpoint of the stereo camera 131 and the floor.

[0082] Reference Fig.12 , the handwritten input information 9021 indicated by the dotted line indicates a moving path passing near the obstacle 402 such as a workbench or a panel, and thus the handwritten input information 9021 needs to be corrected. In this case, the display unit 353 corrects the handwritten input information 9021 to the handwritten input information 9022 indicating a detour path passing the obstacle 402.

[0083] In addition, when the handwriting input information representing the movement path of the robot 100 passes through a predetermined area where the robot 100 is shown, the handwriting input information needs to be corrected. For example, when the trajectory represented by the handwriting input information includes a point that passes through an area within a predetermined radius from the robot 100, the display unit 353 may generate a new trajectory by removing the point from the trajectory. Then, the display unit 353 displays the handwriting input information corresponding to the generated trajectory to the user.

[0084] Fig.13A diagram for explaining an example of a method for correcting handwriting input information. When correcting the handwriting input information 901, the display unit 353 first corrects the position of the handwriting input information 901 until the holding point 9011 included in the U-shaped handwriting input information 901 is located in the area corresponding to the holding object 401. Then, the display unit 353 moves the handwriting input information 901 in stages in a direction away from the holding object 401 until it becomes possible to perform an action based on the handwriting input information 901. The display unit 353 may also use any other method to determine the holding position and posture that is closest to the holding position and posture inferred from the handwriting input information 901 and that can stably hold the holding object 401.

[0085] The display unit 353 may use segmentation information based on machine learning to correct the position of the handwritten input information 901. For example, the segmentation information may be used when positioning the grip point 9011 on the grip object 401. The segmentation information includes information indicating an area corresponding to the grip object 401.

[0086] Back to Figure 6 The display panel 341 is, for example, a liquid crystal panel, and displays, for example, a captured image transmitted from the robot 100 .

[0087] The input unit 342 includes a touch panel overlapped with the display panel 141, and a push button provided at the edge of the display panel 141. The input unit 342 transfers the handwriting input information to the control unit 350. Examples of the handwriting input information are as follows: Figures 2 to 3 As shown.

[0088] The memory 380 is a non-volatile storage medium, for example, a solid state drive, and stores various parameter values, functions, lookup tables, etc. used in control and calculation, in addition to a control program for controlling the remote terminal 300 .

[0089] The communication unit 390 is, for example, a wireless LAN unit, and performs wireless communication with the wireless router 700. The communication unit 390 receives the captured image sent from the robot 100 and transfers it to the control unit 350. In addition, the communication unit 390 cooperates with the control unit 350 to send handwriting input information to the robot 100.

[0090] Reference Fig.14, an example of the operation of the control system 10 according to Embodiment 1 is described. First, the handwriting input information accepting unit 351 of the remote terminal 300 displays the captured image and accepts the input of handwriting input information (step S101). At this time, the handwriting input information accepting unit 351 displays the captured image or the handwriting input information in a manner such that appropriate handwriting input information is input.

[0091] Next, the input of the handwriting input information performed by the user is completed (step S102). Next, the judgment unit 352 of the remote terminal 300 infers the action of the robot 100 based on the handwriting input information (step S103). Next, the judgment unit 352 determines whether the handwriting input information needs to be corrected based on the feasibility or efficiency of the inferred action (step S104). If no correction is required ("No" in step S104), the remote terminal 300 sends the handwriting input information to the robot 100 and executes step S108.

[0092] If the handwriting input information needs to be corrected ("Yes" in step S104), the display unit 353 of the remote terminal 300 displays information related to the correction of the handwriting input information to the user (step S105). The display unit 353 can display information related to the part of the handwriting input information that should be corrected, or can display the corrected handwriting input information.

[0093] Next, the handwriting input information receiving unit 351 determines whether the user has input a correction instruction for the handwriting input information (step S106). The correction instruction may be, for example, input of appropriate handwriting input information. In addition, the user may also give an instruction to correct the displayed handwriting input information. If the user has input a correction instruction ("Yes" in step S106), step S101 is executed.

[0094] If the user does not input a correction instruction (No in step S106), the display unit 353 automatically corrects the handwriting input information (step S107), and the remote terminal 300 sends the corrected handwriting input information to the robot 100. Then, the display unit 353 displays the corrected handwriting input information to the user. Then, the robot 100 performs an action based on the handwriting input information input by the user or the corrected handwriting input information (step S108).

[0095] Next, the effects achieved by the control system 10 according to the first embodiment are described. Although the user can freely write instructions to the robot by handwriting input text, there is a problem that the robot may not be able to act according to the instructions. For example, if the user's instructions are incorrect or the robot cannot correctly understand the user's intention, the handwriting input text needs to be corrected.

[0096] The control system 10 according to the first embodiment can improve operability by interacting with the user regarding correction of handwriting input information.

[0097] In addition, the present invention is not limited to the above-mentioned embodiment, and can be appropriately changed within the scope of the main purpose. Although in the above-mentioned embodiment, the robot 100 and the remote terminal 300 exchange the captured image and handwriting input information via the Internet 600 and the system server 500, it is not limited to this. The robot 100 and the remote terminal 300 may also exchange the captured image and handwriting input information through direct communication.

[0098] In addition, although the stereo camera 131 is used in the above embodiment, the present invention is not limited thereto. The robot 100 may also include any imaging unit installed in any place in the first environment. The imaging unit is not limited to a stereo camera, and may be a monocular camera or the like.

[0099] In addition, although in the above embodiment, the robot 100 is described as an example of a hand 124 at the top of the arm 123 as an end effector, it is not limited to this. The robot 100 only needs to have an end effector and use the end effector to perform a gripping action. In addition, the end effector may also be a gripping part other than a hand (for example, a suction part, etc.).

[0100] The program includes a command group (or software code) for causing the computer to perform one or more functions described in the embodiment when read by the computer. The program may also be stored in a non-temporary computer-readable medium or a storage medium having an entity. As a non-limiting example, the computer-readable medium or the storage medium having an entity includes a random-access memory (RAM), a read-only memory (ROM), a cache memory, a solid-state drive (SSD) or other memory technology, a read-only compact disk (CD-ROM), a digital versatile disc (DVD), a Blu-ray (registered trademark) disc or other optical storage, a cassette, a magnetic tape, a magnetic disk storage or other magnetic storage device. The program may also be sent on a temporary computer-readable medium or a communication medium. As a non-limiting example, a temporary computer-readable medium or a communication medium includes a propagation signal in an electrical, optical, acoustic or other form.

[0101] It is clear from the disclosure described in this way that the disclosed embodiments can be changed in many ways. Obviously such changes should not be regarded as departing from the spirit and scope of the present invention, and for those skilled in the art, all such changes should be included in the appended claims.

Claims

1. A control system that determines the motion of a robot based on handwriting input information input to an interface and controls the motion of the robot, wherein: The control system comprises: a handwriting input information accepting unit, which accepts input of the handwriting input information; a determination unit configured to determine whether the handwriting input information needs to be corrected based on the feasibility or efficiency of the action based on the handwriting input information; A display unit displays information related to the correction of the handwriting input information to the user when the handwriting input information needs to be corrected.

2. The control system according to claim 1, wherein: The handwriting input information includes trajectory information input using a finger, a touch pen, a pointing device, an augmented reality device, a virtual reality device, or a mixed reality device.

3. The control system according to claim 1 or 2, wherein: When the handwriting input information needs to be corrected, the display unit corrects the handwriting input information and displays the corrected handwriting input information to the user.

4. The control system according to claim 1 or 2, wherein: When the handwriting input information needs to be corrected, the presentation unit presents information on a portion of the robot operating based on the handwriting input information that interferes with an obstacle to the user.

5. The control system according to claim 1 or 2, wherein: The determination unit determines that the handwriting input information needs to be corrected when a part or all of the handwriting input information is included in a predetermined area of ​​the input screen, The handwriting input information accepting unit displays the predetermined area and areas other than the predetermined area in a identifiable manner when displaying the input screen.

6. The control system according to claim 1 or 2, wherein: The handwriting input information accepting unit sets the thickness of a line included in the handwriting input information based on the size of the robot when displaying the input handwriting input information.

7. The control system according to claim 1 or 2, wherein: The display unit corrects the position of the handwriting input information using segmentation information obtained based on machine learning.

8. A control method for determining the action of a robot based on handwriting input information input to an interface and controlling the action of the robot, wherein: The control method comprises: The step of accepting the input of the handwriting input information; A step of determining whether the handwriting input information needs to be corrected according to the executability or efficiency of an action based on the handwriting input information; In the case where the handwriting input information needs to be corrected, the step of presenting information related to the correction of the handwriting input information to the user.

9. A computer program product that causes a computer to execute a control method for determining a robot's motion based on handwriting input information input to an interface and controlling the robot's motion, wherein: The control method comprises: The step of accepting the input of the handwriting input information; A step of determining whether the handwriting input information needs to be corrected according to the executability or efficiency of an action based on the handwriting input information; In the case where the handwriting input information needs to be corrected, the step of presenting information related to the correction of the handwriting input information to the user.

Images