Operation assistance device and operation assistance method

Abstract

This operation assistance device comprises an estimation unit that estimates an operation constraint to be applied to a robot device on the basis of sensing information acquired when a user is remotely operating the robot device, a correction unit that corrects the remote operation on the basis of the estimation result of the estimation unit, and a generation unit that generates FB information for feeding back at least one of the estimation result of the estimation unit and the correction result of the correction unit to the user.

Description

Operation support device and operation support method

[0001] The present disclosure relates to an operation assistance device and an operation assistance method.

[0002] For example, Patent Document 1 discloses a technique for estimating a user's intention when remotely operating a robot device.

[0003] Japanese Patent Application Laid-Open No. 2022-157127

[0004] It is conceivable to estimate the user's intention and automatically correct the remote control. However, in this case, the user feels as if the robot device is moving on its own, which may reduce the user's operability in remote control.

[0005] One aspect of the present disclosure is to suppress a decrease in operability for a user in remotely controlling a robot device.

[0006] An operation assistance device according to one aspect of the present disclosure includes an estimation unit that estimates behavioral constraints to be applied to a robot device based on sensing information acquired when a user remotely operates the robot device, a correction unit that corrects the remote operation based on the estimation result of the estimation unit, and a generation unit that generates feedback information to feed back to the user at least one of the estimation result of the estimation unit and the correction result of the correction unit.

[0007] An operation assistance method according to one aspect of the present disclosure includes acquiring sensing information when a user remotely operates a robotic device, estimating behavioral constraints to be applied to the robotic device based on the acquired sensing information, correcting the remote operation based on the estimation result, and generating feedback information for feeding back at least one of the estimation result and the correction result to the user.

[0008] 1 is a diagram illustrating an example of a schematic configuration of an operation assistance system 100 according to an embodiment. FIG. 1 is a diagram illustrating an example of a schematic configuration of the operation assistance system 100. FIG. 2 is a diagram illustrating an example of estimation by an estimation unit 61. FIG. 3 is a diagram illustrating an example of FB information. FIG. 4 is a diagram illustrating an example of feedback based on FB information. FIG. 5 is a diagram illustrating an example of response information. FIG. 6 is a flowchart illustrating an example of processing (operation assistance method) executed in the operation assistance system 100. FIG. 7 is a flowchart illustrating an example of processing (operation assistance method) executed in the operation assistance system 100. FIG. 8 is a diagram illustrating a first embodiment. FIG. 9 is a diagram illustrating a first embodiment. FIG. 10 is a diagram illustrating a second embodiment. FIG. 11 is a diagram illustrating a second embodiment. FIG. 12 is a diagram illustrating a third embodiment. FIG. 13 is a diagram illustrating a third embodiment. FIG. 14 is a diagram illustrating a fourth embodiment. FIG. 15 is a diagram illustrating a fourth embodiment. FIG. 16 is a diagram illustrating a fourth embodiment. FIG. 17 is a diagram illustrating a modification of the fourth embodiment. FIG. 18 is a diagram illustrating a modification of the fourth embodiment. FIG. 19 is a diagram illustrating a modification of the fourth embodiment. FIG. 20 is a diagram illustrating a modification of the fourth embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that unless otherwise specified, the same elements will be designated by the same reference numerals and redundant description will be omitted.

[0010] The present disclosure will be described in the following order: 0. Introduction 1. Embodiment 2. Examples 2.1 First Example 2.2 Second Example 2.3 Third Example 2.4 Fourth Example 2.5 Fifth Example 3. Modification 4. Example of Hardware Configuration 5. Conclusion

[0011] 0. Introduction Remote control of robotic devices (e.g., robotic arms) is difficult because it is difficult to accurately reflect the user's intentions and there are limitations to the reproducibility of the environment in which the robotic device is placed. This can result in a low success rate for remote control and long operation times.

[0012] For example, if the user's intention is estimated as in Patent Document 1, it may be possible to automatically (autonomously) correct the remote control based on the estimation results. However, in this case, the user may feel as if the robot device is moving on its own. Furthermore, if the user's intention cannot be accurately estimated, the remote control may not be performed as intended by the user. It is possible to accumulate information about the remote control and use it as data for machine learning, but it is difficult to obtain effective data.

[0013] The disclosed technology addresses at least some of the above-described problems. As will be described in detail later, for example, information regarding automatic correction of the remote operation of the robot device is fed back to the user, and the content of the feedback can be further modified by the user. This prevents a decrease in the ease of remote operation of the robot device for the user and allows the remote operation of the robot device to be closer to the operation intended by the user. This also increases the possibility of obtaining effective learning data that can be used for machine learning, etc.

[0014] 1. Embodiment Fig. 1 is a diagram showing an example of a schematic configuration of an operation assistance system 100 according to an embodiment. The operation assistance system 100 includes an operation assistance device 3, a robot device 4, a sensor device 5, and an operation assistance device 6. A user of the operation assistance system 100 is referred to as user 1 and illustrated. The user 1 remotely controls the robot device 4. The robot device 4 touches and moves an object 2.

[0015] Each device in the operation assistance system 100 is communicatively connected as appropriate so that necessary information can be transmitted and received. Information may be interpreted as data, and may be interpreted as appropriate within a consistent range. In the example shown in Figure 1, communication is performed via a network 9. Various known communication network technologies may be used.

[0016] The operation assistance device 3 is a device used by the user 1 when remotely operating the robot device 4. As a component of the operation assistance device 3, a UI unit 31 is indicated by a reference numeral in FIG.

[0017] The UI unit 31 is a user interface unit that accepts operations (user operations) of the operation assistance device 3 by the user 1 and presents information to the user 1. In the example shown in FIG. 1, the operation assistance device 3 includes two UI units 31. The first UI unit is referred to as UI unit 31-1 and is illustrated. The second UI unit is referred to as UI unit 31-2 and is illustrated. When there is no particular need to distinguish between these, they are simply referred to as UI units 31.

[0018] In this example, the UI unit 31-1 is an HMD (Head Mounted Display), which presents information to the user 1 and accepts user operations. Examples of presentation modes include displaying an image and outputting sound. The image may be interpreted as a video, and these may be interpreted as appropriate within a consistent range. The image may be an AR (Augmented Reality) image. Examples of user operation modes include voice operation, gaze operation, etc.

[0019] When the robot device 4 is remotely operated, an image near the robot device 4, for example, an image including the target object 2, is displayed to the user 1. This image is also called a robot image. The robot image may be an image seen from the viewpoint of the robot device 4, or may be an image seen from another viewpoint, for example, a viewpoint above the robot device 4. The robot image is captured by, for example, the robot device 4 (or a sensor device 5-1 described later), transmitted from there to the operation assistance device 3, and displayed by the UI unit 31.

[0020] In this example, the UI unit 31-2 is a haptic device (tactile device) that accepts user operations. The user 1 remotely controls the robot device 4 by manually operating the UI unit 31-2. The remote control can also be called a leader-follower operation, in which the UI unit 31-2 is the leader device (master device) and the robot device 4 is the follower device (slave device).

[0021] It should be noted that various known user interface technologies that can be used to remotely control the robot device 4 may be used together with or instead of the above-described HMD and haptic device. For example, the UI unit 31-1 may be a stationary or portable display device. The UI unit 31-2 may be a game controller or the like.

[0022] The robot device 4 is a remotely controlled robot device that operates in accordance with external commands (corresponding to operation command information described later). The operation may be interpreted as movement, motion, etc., and may be appropriately interpreted as long as there is no contradiction.

[0023] The robot device 4 has a movable part. As an example of a movable part, an arm 42 is indicated by a reference numeral in FIG. 1. In this example, the robot device 4 includes two arms 42, an arm 42-R (right arm) and an arm 42-L (left arm). When simply referring to an arm 42, this may be understood to refer to at least one of the arm 42-R and the arm 42-L. Note that the arm 42 itself may also be referred to as a robot device, and the terms robot device 4 and arm 42 may be interpreted as appropriate within the scope of no contradiction.

[0024] The sensor device 5 senses (detects, etc.) the user 1, the object 2, the robot device 4, and the environment in which they are placed. Information indicating the results of sensing by the sensor device 5 is referred to as sensing information. For example, the sensing information may include information related to images (stereo RGB images, depth images, etc.), the trajectory of the user 1's line of sight (eye tracking), the trajectory of the object's movement, proximity, point clouds, force, pressure, vibration, acceleration, amount of slippage, contact position and contact area, etc. Various known sensing technologies may be used.

[0025] The sensor device 5 includes a sensor device 5-1 (first sensor device) and a sensor device 5-2 (second sensor device). The sensor device 5-1 is placed near the user 1 and senses the user 1 and the environment in which the user 1 is located. The sensor device 5-1 may be a plurality of sensor devices. The sensor device 5-2 is placed near the object 2 and the robot device 4 and senses the object 2, the robot device 4, and the environment in which they are located. The sensor device 5-2 may be a plurality of sensor devices.

[0026] Further details of the operation support system 100 will be described with reference to FIG.

[0027] 2 is a diagram showing an example of a schematic configuration of the operation assistance system 100. Among the components of the operation assistance system 100, the block configuration of the operation assistance device 3, the robot device 4, and the operation assistance device 6 is shown as an example.

[0028] The operation assistance device 3 includes a storage unit 30 and a UI unit 31. The storage unit 30 stores information used by the operation assistance device 3. An example of the information stored in the storage unit 30 is a program 301. The program 301 is a program (software) for causing a computer to function as the operation assistance device 3.

[0029] As described above, the UI unit 31 presents information and accepts user operations. When remotely operating the robot device 4, the UI unit 31 generates operation command information according to the content of the remote operation. The operation command information provides the robot device 4 with target values ​​for the remote operation. Examples of the target values ​​include values ​​such as the position, posture, joint angle, speed, acceleration, and force of each part of the robot device 4. The generated operation command information is transmitted to the operation assistance device 6.

[0030] The operation command information transmitted from the operation assistance device 3 to the operation assistance device 6 is modified as necessary and transmitted from the operation assistance device 6 to the robot device 4. As will be described later, the operation assistance device 6 stores information related to remote operation (corresponding to history information 602). Information indicating whether or not to perform this storage (save on / off information) may also be transmitted from the operation assistance device 3 to the operation assistance device 6. This information may be transmitted from the operation assistance device 3 to the operation assistance device 6 together with the operation command information (for example, incorporated into the operation command information), or may be transmitted from the operation assistance device 3 to the operation assistance device 6 separately from the operation command information.

[0031] The robot device 4 includes a movement control unit 41 and an arm 42 (an example of a movable unit). The movement control unit 41 controls the movement of the arm 42 based on the movement command information described above. Examples of controlled objects include position, posture, joint angle, speed, acceleration, and force. The movement control unit 41 controls the arm 42, more specifically, controls actuators (motors, etc.) included in the arm 42, so that these values ​​approach target values ​​given by the movement command information.

[0032] The operation assistance device 6 includes a storage unit 60, an estimation unit 61, a correction unit 62, and a generation unit 63. The storage unit 60 stores information used by the operation assistance device 6. Examples of the information stored in the storage unit 60 include a program 601 and history information 602. The program 601 is a program (software) for causing a computer to function as the operation assistance device 6. The history information 602 will be described later.

[0033] The estimation unit 61 acquires sensing information from the sensor device 5, more specifically, sensing information acquired when the user 1 remotely controls the robot device 4, for example. The correction unit 62 performs estimation based on the acquired sensing information. Estimation may be interpreted to mean identification, recognition, determination, etc. Various estimations based on the sensing information are possible. This will be described with reference to FIG. 3 as well.

[0034] 3 is a diagram showing examples of estimation by the estimation unit 61. Examples of estimation include situation estimation, intention estimation, and motion constraint estimation. Each of these will be described below in order.

[0035] <Situation Estimation> The estimation unit 61 estimates a situation related to remote operation based on sensing information from the sensor device 5. The situation may include the situations of the user 1, the target object 2, and the robot device 4.

[0036] For example, the estimation unit 61 recognizes each object, such as the user 1, the target object 2, the robot device 4, etc., based on information such as images and proximity sense included in the sensing information, and also estimates their position, posture, contour, plane, etc.

[0037] Various known algorithms for estimation may be used. For object recognition, a YOLO (You Only Look Once) technique may be used. For pose estimation, a 6DoF-Pose estimation technique based on an image and a model of the object 2 measured in advance may be used. The pose may include the pose of the robot device 4 for grasping the object 2. A ContactGraspNet technique may be used.

[0038] 3 shows examples of situations that can be estimated by the estimation unit 61, including the shape of the robot device 4, the shape of the target object 2, the posture of the robot device 4, the posture of the target object 2, the trajectory (of the movement) of the robot device 4, the trajectory of the target object 2, and the trajectory of the line of sight of the user 1. For example, such various shapes, postures, trajectories, etc. are estimated as situations related to remote operation.

[0039] <Intention Estimation> The estimation unit 61 estimates the intention of the user 1 regarding remote operation based on the estimated situation. For example, the intention of the user 1 is estimated based on the line of sight of the user 1 and the movement trajectory of the robot device 4. An intention of reaching the target object 2 or an intention to move to the position of the line of sight may be estimated. Various estimation techniques may be used, such as a rule-based algorithm or a trained (deep learning, etc.) classifier.

[0040] Figure 3 shows examples of the intentions of user 1 that can be estimated by the estimation unit 61, including an intention to approach the object 2 (reaching), an intention to grasp, an intention to grasp with both hands, an intention to go straight, an intention to rotate, an intention to rotate relative to the object (twist), and an intention to avoid a collision.

[0041] The intention to approach the target 2 (reaching) is the intention to move the arm 42 closer to the target 2. For example, when the arm 42 moves straight toward the target 2, reaching is estimated.

[0042] The intention to grasp is the intention to grasp the object 2 with the arm 42 (for example, one of the arm 42-R and the arm 42-L). For example, the intention to grasp is estimated when the arm 42 is in contact with the object 2 or has approached the object 2 to a position just before that.

[0043] The intention of grasping with both hands is the intention to grasp the object 2 with both the arm 42-R and the arm 42-L (both hands). For example, when both the arm 42-R and the arm 42-L are in contact with the object 2, or when one of the arms is in contact with the object 2 and the other is close to the object 2, the intention of grasping with both hands is estimated.

[0044] The intention to move straight is to move the arm 42 in a straight line while the arm 42 is gripping the object 2, thereby moving the object 2 in a straight line. For example, an intention to move straight is estimated when the arm 42 is gripping the object 2 or has started to move in a straight line.

[0045] The intention to rotate is an intention to rotate the arm 42 while the arm 42 is gripping the object 2, thereby rotating the object 2. For example, an intention to rotate is estimated when the arm 42 is gripping the object 2 or has started to rotate.

[0046] The intention of relative rotation is the intention to rotate the arm 42-R and the arm 42-L relative to each other while each of them is gripping a different part of the object 2, thereby twisting (partially rotating) the object 2. For example, the intention of relative rotation is presumed when the arm 42-R and the arm 42-L are in contact with different parts of the object 2 or when they have begun to rotate relative to each other.

[0047] The intention to avoid collision is the intention to avoid collision of the arm 42 with the object 2. A collision may mean contact with the arm 42 outside the intended range of a desired action (such as grasping). It can also be said that the intention to avoid collision may always be present during remote operation. For example, it may be assumed that the intention to avoid collision is always present during remote operation.

[0048] <Movement Constraint Estimation> The estimation unit 61 estimates a movement constraint to be applied to the robot device 4 based on the above-mentioned estimated intention. The movement constraint may be estimated based on the above-mentioned estimated situation. The movement constraint includes, for example, constraints on the position, posture, movement trajectory, etc. of the robot device 4. When a movement constraint is applied, the movement of the robot device 4 is limited within the range of the movement constraint.

[0049] Figure 3 shows examples of movement constraints that the estimation unit 61 can estimate, including a constraint on straight-line movement, a constraint on translational movement, a constraint on grasping movement, a constraint on two-handed grasping movement, a constraint on rotational movement, a constraint on relative rotational movement (twisting movement), and a constraint on collision avoidance movement.

[0050] The restriction on linear movement is, for example, the restriction on the linear movement of the arm 42, and in this case, it indicates that the movement of the arm 42 is restricted so that it moves linearly.

[0051] The constraint on translational movement is, for example, the constraint on translational movement of the two arms 42. In this case, the constraint on translational movement means that the movement of the arms 42-R and 42-L is constrained so that they move linearly (in the same direction) while maintaining their relative positions. For example, when the intention to move straight is estimated as described above, the constraint on the straight movement is estimated.

[0052] The constraint on the grasping operation is, for example, a constraint on the operation of grasping the object 2 by the arm 42. In this case, the constraint on the grasping operation indicates that the operation of the arm 42 is constrained so that the arm 42 grasps the object 2. For example, when the intention to grasp described above is estimated, the constraint on the grasping operation is estimated.

[0053] The constraint of the two-handed grasping motion is, for example, the constraint of the two-handed grasping motion of the object 2 by the arm 42-R and the arm 42-L. In this case, the constraint of the two-handed grasping motion means that the motions of the arm 42-R and the arm 42-L are constrained so that both of them grasp the same object 2. For example, when the intention of grasping with both hands described above is estimated, the constraint of the grasping motion is estimated.

[0054] The constraint on the rotational movement is, for example, a constraint on the rotational movement of the arm 42. In this case, the constraint on the rotational movement indicates that the movement of the arm 42 is constrained so that the arm 42 rotates, for example, around a specific rotation axis or in a specific rotation direction. For example, when the intention to rotate as described above is estimated, the constraint on the rotational movement is estimated.

[0055] The rotation axis may be identified from the shape (circular, cylindrical, etc.) of the object 2. Alternatively, the rotation axis and even the rotation direction may be identified based on the actual initial movement of the arm 42 gripping the object 2. For example, in a situation where the trajectory of the arm 42 is restricted so that it rotates, a Hough transform or the like may be used to select an appropriate plane from candidates perpendicular to the rotation axis to identify the orientation of the axis, and the position of the rotation axis may be identified from that.

[0056] The constraint on the relative rotational movement is, for example, the constraint on the twisting movement due to the relative rotational movement of the arm 42-R and the arm 42-L. In this case, the constraint on the relative rotational movement indicates that the movement of the arm 42-R and the arm 42-L is constrained so that they rotate relative to each other while each grips a different part of the object 2. For example, when the intention of relative rotation described above is estimated, the constraint on the relative rotational movement is estimated.

[0057] The constraint of the collision avoidance operation is, for example, a constraint on the collision avoidance operation of the arm 42 with the object 2. In this case, the constraint of the collision avoidance operation indicates that the operation of the arm 42 is constrained so that the arm 42 does not collide with the object 2. For example, if the intention to avoid collision as described above is estimated, the constraint of the collision avoidance operation is estimated. For example, the constraint of the collision avoidance operation can be estimated at all times during remote operation.

[0058] Returning to FIG. 2 , various estimations including, for example, the situation estimation, intention estimation, and behavioral constraint estimation as described above are performed by the estimation unit 61. The estimation unit 61 may calculate the reliability of the estimation result. The reliability may be an index that indicates the likelihood of the estimation result numerically (percentage value, etc.). For example, when multiple intentions are estimated as the intentions of the user 1, the reliability of each of them may be calculated. When multiple behavioral constraints are estimated as behavioral constraints of the robot device 4, the reliability of each of them may be calculated. The priority of each of them may also be calculated.

[0059] The correction unit 62 corrects the remote operation of the robot device 4 by the user 1 based on the estimation result of the estimation unit 61. For example, the correction unit 62 corrects the motion command information so that the motion constraint estimated by the estimation unit 61 is applied to the robot device 4. The corrected motion command information is transmitted to the robot device 4. The motion control unit 41 of the robot device 4 controls the motion of the arm 42 in accordance with the motion command information. The remote operation is automatically corrected (autonomous correction).

[0060] Correction by the correcting unit 62 is not essential, and there may be cases where correction is not performed. For example, the correcting unit 62 determines whether or not to actually correct the action command information (whether correction is necessary) based on various parameters (which may also be called indexes). Examples of parameters include the reliability and priority of the above-mentioned action constraints. A threshold determination for the parameters may be used. For example, the correcting unit 62 corrects the action command information when the reliability is equal to or greater than a threshold, and does not correct the action command information when the reliability is less than the threshold. The threshold may be variable and may be changed, for example, by a user operation on the operation assistance device 3 (based on response information, which will be described later).

[0061] The generation unit 63 generates FB (feedback) information. The FB information is information for feeding back at least one of the estimation result of the estimation unit 61 and the correction result of the correction unit 62 to the user 1. The FB information will be described with reference to FIG. 4 as well.

[0062] 4 is a diagram illustrating an example of the FB information, which includes an estimation result and a correction result.

[0063] The estimation result indicates the content of the estimation result of the estimation unit 61. Examples of the estimation result include the estimated intention of the user 1, the estimated behavioral constraint of the robot device 4, and the reliability. The intention of the user 1 and the behavioral constraint of the robot device 4 are as described above with reference to FIG. 3. The estimation result may include multiple intentions, and may also include multiple behavioral constraints. The reliability indicates the reliability of the estimation result, for example, the reliability of each of the multiple behavioral constraints.

[0064] The correction result indicates the content of the correction result of the correction unit 62. An example of the correction result indicates the movement constraint of the robot apparatus 4 reflected in the movement command information. For example, when the correction unit 62 corrects the movement command information to reflect the movement constraint indicated in the estimation result described above in the movement of the robot apparatus 4, the movement constraint can be the correction result.

[0065] In addition to the above, the FB information may include various information related to the estimation by the estimation unit 61 and the correction by the correction unit 62. An example of other FB information is a threshold value used by the correction unit 62 to determine whether or not correction is necessary.

[0066] Returning to Fig. 2, for example, the above-described FB information is generated by the generation unit 63. The generated FB information is transmitted from the operation assistance device 6 to the operation assistance device 3. The UI unit 31 (for example, the UI unit 31-1 in Fig. 1) of the operation assistance device 3 provides feedback based on the FB information to the user 1. Various modes of feedback are possible. This will be described with reference to Fig. 5 as well.

[0067] 5 is a diagram showing an example of feedback based on the FB information. Examples of feedback modes include verbal notification, superimposed image display, reaction force output, and vibration generation.

[0068] In the linguistic notification, the FB information is notified by text, voice, etc. For example, the motion constraints related to the estimation or correction, the above-mentioned parameters (reliability, priority, etc.), thresholds therefor, etc. are displayed in text or output as voice.

[0069] In the superimposed image display, an image showing the FB information is superimposed on the robot image described above. The image showing the FB information may be text, graphics, or a combination thereof.

[0070] For example, motion constraints related to estimation or correction and related information (such as rotation axes) may be displayed. Multiple motion constraints (candidates) may be displayed together with their reliability. The reliability may be expressed in various ways, such as color shading, shape size, transparency, etc.

[0071] The above-mentioned linguistic notification, superimposed image display, etc. can also be used for interactive operations, such as presenting the most reliable estimated motion constraint to user 1 in response to an inquiry from user 1, before the correction unit 62 corrects the motion command information.

[0072] In the reaction force output, a virtual reaction force is output so as to restrict the operation of the UI unit 31-2 (such as a haptic device) to an operation corresponding to the motion constraint related to the estimation or correction. For example, if the motion constraint is a collision avoidance constraint, a reaction force is output so as to prevent an operation that would cause the arm 42 of the robot device 4 to collide with the object 2.

[0073] When vibration is generated, the UI unit 31-2 (such as a haptic device) vibrates. The vibration is output when the correction unit 62 corrects the operation command information, thereby informing the user 1 of the fact that the correction has been made.

[0074] For example, in various modes as described above, feedback based on the FB information is provided to the user 1. By receiving the feedback, the user 1 can confirm whether the operation assistance device 6 has correctly estimated his / her intention and whether appropriate motion constraints have been applied for that purpose. In other words, the user 1 can know whether the remote operation of the robot device 4 by the user 1 is being appropriately assisted.

[0075] The user 1 can also respond to the feedback. Specifically, the user 1 operates the UI unit 31 (for example, the UI unit 31-1 in FIG. 1) of the operation assistance device 3 to input the response content. The UI unit 31 generates response information indicating the input response content. The response information will be described with reference to FIG. 6 as well.

[0076] 6 is a diagram showing an example of response information, which may include consent and correction instructions.

[0077] Consent means agreement with the content presented in the feedback. For example, if the feedback presents a behavioral constraint related to an estimate or correction, consent means agreement to the application of the behavioral constraint.

[0078] The correction instruction is a correction of the content presented in the feedback. Examples of the correction instruction include a correction of the estimated intention of the user 1 and a correction of the estimated movement constraint of the robot device 4. For example, if the feedback presents an estimated intention of the user 1 and an estimated or corrected movement constraint of the robot device 4, the correction instruction means a correction of the estimated intention of the user 1 and the movement constraint of the robot device 4.

[0079] Examples of correction instructions for correcting the motion constraints include changing the motion constraint (such as its type), changing related information (such as the rotation axis), changing thresholds for parameters (such as reliability and priority), etc. An instruction may be given as to whether or not to perform correction by the corrector 62 (turning on or off the automatic correction function).

[0080] The user 1 can modify the motion constraints, thereby bringing the motion of the robot device 4 closer to the motion intended by the user 1. For example, in an environment where erroneous estimation is likely to occur, the threshold can be set high to prevent a decrease in operability due to the application of inappropriate motion constraints. Customization is also possible to obtain the operability desired by the user 1 for various environments and tasks.

[0081] The response content that forms the basis of the response information may be input by operating, for example, a button, a foot key, a keyboard, or the like that may be included in the UI unit 31. The operation may be a voice input operation, a character input operation, or the like. Simple operations are provided that do not require the remote control of the robot device 4 to be stopped. Interactive operations such as those described above are also possible.

[0082] To allow the user 1 to respond to the feedback calmly, the robot device 4 may be configured to stop the remote control loop and switch to a response information input mode by pressing a button or the like. This can improve safety. In addition, since unnecessary data is no longer included, the possibility of obtaining clear learning data increases.

[0083] When the FB information is displayed superimposed on an image, a target may be directly selected from the superimposed figures, characters, etc. using a mouse, a VR controller, hand gestures, etc., and a correction instruction may be given. When selecting an arbitrary object (such as the target 2) or indicating its movement (for example, a movement axis), an object such as the target 2 may be directly clicked, a rectangular selection operation may be performed, or a direction may be specified by a drag operation.

[0084] 2 , for example, in the various modes described above, response content that will be the basis for a response is input, and response information is generated. The generated response information is transmitted from the operation assistance device 3 to the operation assistance device 6.

[0085] The estimation unit 61 of the operation assistance device 6 updates the estimation result based on the response information from the operation assistance device 3. Updating includes changing, correcting, etc. the estimation result, and may also include maintaining the estimation result (not changing, correcting, etc.). For example, if the response information is information of agreement, the estimation unit 61 maintains the estimation result. On the other hand, if the response information is information of a correction instruction, the estimation unit 61 updates the estimation result to reflect the content of the correction instruction. For example, if an instruction is given to correct the intention of the user 1, the estimation unit 61 re-estimates the movement constraint of the robot device 4 based on the corrected intention of the user 1. If an instruction is given to correct the movement constraint of the robot device 4, the estimation unit 61 sets the corrected movement constraint as the estimation result.

[0086] The correction unit 62 corrects the remote operation, i.e., corrects the operation command information, based on the updated estimation result. The operation control unit 41 of the robot device 4 controls the operation of the arm 42 in accordance with the corrected operation command information. This brings the remote operation of the robot device 4 closer to the operation intended by the user 1.

[0087] For example, in the above manner, remote operation of the robot device 4 proceeds. Various types of information used in remote operation of the robot device 4, such as sensing information, operation command information, feedback information, and response information, are stored as history information 602 in the storage unit 60 of the operation assistance device 6. The history information 602 can be used, for example, for machine learning of the control of the robot device 4.

[0088] 7 and 8 are flowcharts showing an example of processing (operation assistance method) executed in the operation assistance system 100. Description of content that overlaps with the above will be omitted where appropriate.

[0089] 7, feedback is provided to the user 1, but no response is made thereto. In step S1, the estimation unit 61 of the operation assistance device 6 acquires sensing information from the sensor device 5. In step S2, the estimation unit 61 of the operation assistance device 6 performs various estimations, specifically, the situation estimation, intention estimation, and action constraint estimation described above with reference to FIG.

[0090] In step S3, the correction unit 62 of the operation assistance device 6 corrects the operation command information based on the estimation result. Furthermore, the generation unit 63 generates feedback information based on the estimation result and the correction result. An example of the feedback information is as described above with reference to FIG. 4. The corrected operation command information is transmitted to the robot device 4, and the feedback information is transmitted to the operation assistance device 3.

[0091] In step S4, the UI unit 31 of the operation assistance device 3 provides feedback to the user 1. For example, feedback based on the FB information is provided to the user 1 in various modes as described above with reference to Fig. 5. The user 1 proceeds with remote operation of the robot device 4 while checking the assistance status of the remote operation.

[0092] In step S5, the UI unit 31 of the operation assistance device 3 generates operation command information in accordance with the content of the remote operation of the robot device 4 by the user 1. The generated operation command information is transmitted to the operation assistance device 6. This operation command information can be corrected by the correction unit 62 of the operation assistance device 6, as described above in step S3.

[0093] In step S6, the operation of the robot device 4 is controlled. The operation control unit 41 of the robot device 4 controls the operation of the arm 42 (an example of a movable part) in accordance with the operation command information from the operation assistance device 6.

[0094] The processes of steps S1 to S7 are repeatedly executed, whereby the user 1 remotely controls the robot device 4.

[0095] In the flow of Fig. 8, a response to the feedback is also performed. The processes of steps S11 and S12 are the same as the processes of steps S1 and S2 in Fig. 7 described above. The estimation unit 61 of the operation assistance device 6 acquires sensing information and performs estimation.

[0096] In step S13, the generating unit 63 of the operation assistance device 6 generates the FB information. The FB is transmitted to the operation assistance device 3.

[0097] The process of step S14 is the same as the process of step S4 in Fig. 7 described above. The UI unit 31 of the operation assistance device 3 provides feedback to the user 1. In the example shown in Fig. 8, the user 1 responds to the feedback.

[0098] In step S15, the UI unit 31 of the operation assistance device 3 generates response information. The user 1 operates the UI unit 31 to input a response to the feedback provided in the previous step S14. The UI unit 31 generates response information indicating the input response. The response information is transmitted from the operation assistance device 3 to the operation assistance device 6. The user 1 continues remotely operating the robot device 4.

[0099] In step S16, the estimation unit 61 of the operation assistance device 6 updates the estimation result based on the response information. For example, if the response information is a correction instruction, the estimation result is updated to reflect the content of the correction instruction.

[0100] In step S17, information used for the remote operation is stored in the storage unit 60 of the operation assistance device 6. For example, sensing information, operation command information, feedback information, response information, etc. are added to the history information 602 in the storage unit 60.

[0101] The process of step S18 is the same as the process of step S5 in Fig. 7 described above. The UI unit 31 of the operation assistance device 3 generates operation command information. The generated operation command information is transmitted to the operation assistance device 6.

[0102] In step S19, the correction unit 62 of the operation assistance device 6 corrects the operation command information based on the estimation result. The estimation result used here is the estimation result after being updated in the previous step S16. The corrected operation command information is transmitted to the robot device 4. Note that if the response information indicates consent to the feedback, there is no need to correct the operation command information. For example, the operation command information from the operation assistance device 3 is transmitted to the robot device 4 without being corrected.

[0103] The process of step S20 is the same as the process of step S6 in Fig. 7. The movement control unit 41 of the robot device 4 controls the movement of the arm 42 (an example of a movable unit) in accordance with the movement command information from the operation assistance device 6.

[0104] The processes of steps S10 to S20 are repeatedly executed, whereby the user 1 remotely controls the robot device 4.

[0105] According to the operation assistance system 100 described above, the estimation results of the estimation unit 61 and the correction results of the correction unit 62 of the operation assistance device 6 are fed back to the user 1. Without feedback, for example, the user 1 may get the sense that the robot device 4 is moving on its own, which may result in a decrease in the operability of remote operation of the robot device 4 for the user 1. By providing feedback as described above, the user 1 can know what estimations, corrections, etc. have been made in the operation assistance device 6. This makes it possible to prevent a decrease in the operability of remote operation.

[0106] Furthermore, when the user 1 responds to the feedback, the estimation result of the estimation unit 61 is updated, and the correction by the correction unit 62 is changed. This allows the movement of the robot device 4 to be closer to the movement intended by the user 1. Various parameters, such as thresholds for the reliability of movement constraints, can also be adjusted, ensuring customization according to the environment, task, and preferences of the user 1. This improves operability, further increasing the possibility of shortening operation time and improving the success rate of operation. It also contributes to an improved operation experience. It can be applied to various environments and tasks, and can also be applied to cluttered environments where environmental recognition is difficult.

[0107] Information related to remote operation is stored (saved) as history information 602 in the storage unit 60 of the operation assistance device 6. For example, by storing information such as the trajectory of the movement of the robot device 4, clean data with less unnecessary movements and variations can be obtained. For example, this data can be used as data for machine learning. By using such data for machine learning and applying it to the robot device 4, the possibility of improving the success rate of the task increases.

[0108] 2. Examples Several specific examples of remote control of the robot device 4 based on the technologies described so far will be described as Examples 1 to 5. It goes without saying that the various technologies described so far or technologies based on them may be used as appropriate, even if no particular explanation is given.

[0109] 9 to 12 show a first embodiment. The object 2 is a bottle, and is illustrated as a bottle 2a. A user 1 remotely controls the robot device 4 so that the robot device 4 opens the bottle lid.

[0110] 9, the arm 42-L is gripping the body of the bottle 2a. Meanwhile, the arm 42-R is positioned near the bottle 2a but is not gripping it. From this situation, it can be inferred that the user 1 intends to, for example, rotate the lid, or to grip and move (carry) the bottle 2a with both arms.

[0111] As shown in Figure 10, the FB information presents a question asking whether the user 1 intends to rotate the lid of the bottle 2a or to carry the bottle 2a. In response, response information is generated indicating that the user intends to hold the body of the bottle 2a with one hand and the lid of the bottle 2a with the other hand in order to open the lid. The estimation result is updated based on this response information. The updated estimation result indicates a rotational motion constraint (corresponding to the relative rotational motion constraint in Figure 3) for rotating the lid of the bottle 2a. The motion command information is modified based on the estimation result.

[0112] As shown in FIG. 11, the arm 42-L grips the body of the bottle 2a (the lower part of the bottle 2a), and the arm 42-R grips the lid of the bottle 2a (the upper part of the bottle 2a). Then, as shown by the arrow AR1, the arm 42-R rotates relative to the arm 42-L. This rotational movement is restricted. Specifically, as shown in FIG. 12, a rotation axis is identified as indicated by the dashed line. The arm 42-R rotates around the rotation axis.

[0113] 13 to 16 show a second embodiment. The object 2 is a door, and is shown as a door 2b. The user 1 remotely controls the robot device 4 so that the robot device 4 opens the door.

[0114] 13 , the arm 42 is gripping the knob of the door 2b. From this situation, it is unclear even to the user 1 how the door 2b will move (for example, by rotating or sliding) to open, or how the knob of the door 2b will move. For the time being, the following intentions of the user 1 are estimated: to rotate the knob of the door 2b as indicated by the arrow AR2, to rotate the door 2b as indicated by the arrow AR3, and to slide the door 2b as indicated by the arrow AR4. The reliability of each intention is shown as 33%.

[0115] The user 1 remotely controls the robot device 4 to check the movement (initial movement) of the door 2b and the knob. Here, as shown in Fig. 14, when the knob of the door 2b is slightly moved toward the back, the door 2b rotates in the direction of the arrow AR2 and opens. This shows that to open the door 2b, the door 2b should be rotated in the direction of the arrow AR3.

[0116] As shown in Figure 15, response information is generated to restrict rotation as indicated by arrow AR3. The estimation result is updated based on this response information. The updated estimation result indicates a rotational motion constraint (corresponding to the rotational motion constraint in Figure 3) for rotating door 2b in the direction of arrow AR3. The operation command information is corrected based on the estimation result. The robot device 4 operates based on the corrected operation command information, and door 2b opens.

[0117] 16 to 19 show a third embodiment. The target object 2 is a bottle 2a, and the user 1 remotely controls the robot device 4 so that the robot device 4 holds and moves the bottle 2a with both arms 42-L and 42-R (both hands).

[0118] 16, each of the arms 42-R and 42-L is positioned near the bottle 2a. From this situation, it is estimated that the user 1 intends to, for example, rotate the lid of the bottle 2a and to grasp the bottle 2a with both hands.

[0119] As shown in Figure 17, the FB information is presented in the form of a superimposed image, asking whether user 1 intends to rotate the lid of bottle 2a as indicated by arrow AR5, or to align arms 42-R and 42-L so that they both grasp bottle 2a (fix their relative positions) as indicated by the lock mark.

[0120] The actual intention of the user 1 is to grasp the bottle 2a with both hands, and response information indicating this is generated. In the example shown in Figure 18, the user 1 directs his / her gaze toward the bottle 2a, thereby indicating his / her intention to grasp the same bottle 2a with both the arm 42-R and the arm 42-L (with both hands).

[0121] As shown in FIG. 19 , the coincidence constraint is applied, and the reliability is also indicated as a high value of 99%. Furthermore, the user 1's gaze is directed toward the destination of the bottle 2a, thereby indicating an intention to grasp the bottle 2a with both hands and move it in the direction of arrow AR6. This response information is reflected in the estimation result. The estimation result after reflecting the response information indicates motion constraints (corresponding to the two-handed grasping motion constraint and the translational motion constraint in FIG. 3 ) for grasping the bottle 2a with arms 42-R and 42-L and moving it translationally in the direction of arrow AR6. The motion command information is corrected based on the estimation result. The robot device 4 operates based on the corrected motion command information, and the bottle 2a is transported.

[0122] 20 to 25 show a fourth embodiment, in which the robot device 4 is prevented from colliding with the target object 2.

[0123] 20 , the arm 42 is moving in the direction of the arrow AR7. The user 1 is moving the arm 42 in the direction of the arrow AR7 in order to grasp the object 2 with the arm 42. From this situation, it is estimated that the user 1 intends, for example, to avoid the arm 42 colliding with the object 2 in a manner that would prevent the arm 42 from grasping the object 2. However, if the arm 42 continues to move in this manner, there is a possibility that such a collision will occur.

[0124] 21, a collision avoidance constraint is presented as FB information. At the same time, the UI unit 31-2 (such as a haptic device) outputs a reaction force as indicated by the arrow AR8. This restricts the operation of the UI unit 31-2 so that an operation that would cause the arm 42 to collide with the object 2 cannot be performed.

[0125] After the arm 42 has avoided collision with the object 2, the positional relationship between the arm 42 and the object 2 is assumed to be as shown in Fig. 22. If this positional relationship (for example, the angle of the arm 42 relative to the object 2) remains unchanged, it will be difficult for the arm 42 to grasp the object 2.

[0126] Therefore, as shown in Figure 23, response information indicating an intention to grasp the object 2 is generated. This response information is reflected in the estimation result. The estimation result after reflecting the response information indicates an operation constraint (corresponding to the constraint of the grasping operation in Figure 3) for grasping the object 2 with the arm 42. The operation command information is corrected based on the estimation result. The robot device 4 operates based on the corrected operation command information, and the object 2 is grasped.

[0127] Specifically, as shown in Fig. 24, the arm 42 moves as indicated by arrow AR9 so as to obtain a positional relationship for grasping the object 2 with the arm 42. Then, as shown in Fig. 25, the arm 42 grasps the object 2.

[0128] 26 to 29 are diagrams illustrating a modification of the fourth embodiment. When there is a delay between the actual operation of the robot device 4 and the presentation of a robot image to the user 1, positioning, collision avoidance, and the like are performed based on the estimation result of the intention of the user 1. Note that the delay may be caused by, for example, the communication over the network 9 described above, or the processing in each of the operation assistance devices 3, 6, and the robot device 4.

[0129] 26 , it is assumed that user 1 is moving arm 42 in the direction of arrow AR10 in order to grasp object 2 with arm 42. It is presumed that user 1 intends to avoid colliding with object 2 in a manner that would prevent arm 42 from grasping object 2. If arm 42 continues to move in this manner, there is a possibility that such a collision will occur. However, user 1 is not yet aware of this because there is a delay in the presentation of the robot image.

[0130] As shown in Fig. 27, the collision avoidance constraint is presented as FB information. At the same time, the UI unit 31-2 (such as a haptic device) vibrates. This notifies the user 1 that the operation command information has been modified to reflect the collision avoidance constraint. Response information for modifying the information is generated as necessary. The user's operation for generating the response information (the input operation of the response content) may be performed with the remote control loop of the robot device 4 stopped.

[0131] After the arm 42 has avoided collision with the object 2, the arm 42 operates as shown in FIGS. 24 and 25 described above to grasp the object 2.

[0132] 28 and 29 show a fifth embodiment of the present invention, in which the robot device 4 is prevented from colliding with itself.

[0133] 28 illustrates, as components of the robot device 4, the body 4a and elbow 4b, which are indicated by symbols, in addition to the arm 42 described above. It is assumed that the user 1 is remotely controlling the robot device 4 so that the arm 42 grasps the object 2. At this time, it is assumed that the elbow 4b of the robot device 4 is approaching the body 4a, although the user 1 is unaware of this. It is estimated that the user 1 intends to avoid a self-collision of the robot device 4 (in this example, a collision between the elbow 4b and the body 4a).

[0134] As shown in Fig. 29, the FB information indicates that the elbow 4b is approaching the torso 4a, and presents a collision avoidance constraint for avoiding self-collision between them. This notifies the user 1 that the motion command information has been modified to reflect the collision avoidance constraint. Response information for modifying the information is generated as necessary. Remote operation proceeds by performing operations similar to those described above while avoiding self-collision.

[0135] For example, various operational assistance such as those in the above-mentioned first to fifth embodiments is provided by the operational assistance system 100.

[0136] 3. Modifications The disclosed technology is not limited to the above-described embodiment. For example, in the above-described embodiment, an example has been described in which two operation assistance devices, an operation assistance device 3 and an operation assistance device 6, exist. However, the functions of the operation assistance device 3 and the operation assistance device 6 may be integrated into one device. For example, some or all of the functions of the operation assistance device 6 may be incorporated into the operation assistance device 3. Furthermore, the functions of the operation assistance device 3 and the operation assistance device 6 may be distributed across three or more devices.

[0137] In the above embodiment, an example has been described in which the robot device 4 has two arms 42, the arm 42-R and the arm 42-L. However, the robot device 4 may have one arm 42, or three or more arms 42. Furthermore, the robot device 4 may have various other movable parts in addition to or instead of the arm 42.

[0138] 30 is a diagram showing an example of the hardware configuration of the device. At least a part of each of the operation assistance device 3, the robot device 4, and the operation assistance device 6 described so far can be realized by, for example, the computer 1000 shown in the figure.

[0139] The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected to each other via a bus 1050.

[0140] The CPU 1100 operates based on programs stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs. Examples of the programs are the programs 301 and 601 shown in FIG. 2, which have been described above.

[0141] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0142] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records programs for the encoding method, decoding method, and information processing method according to the present disclosure, which are examples of program data 1450.

[0143] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0144] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU receives data from input devices such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined computer-readable recording medium. Examples of the medium include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disc), tape media, magnetic recording media, or semiconductor memory.

[0145] When computer 1000 functions as the various devices described above, CPU 1100 of computer 1000 realizes those functions by executing programs loaded onto RAM 1200. The programs may be stored in HDD 1400. CPU 1100 reads and executes program data 1450 from HDD 1400, but as another example, CPU 1100 may obtain the program from another device via external network 1550.

[0146] Each of the above components may be configured using general-purpose materials or may be configured using hardware specialized for the function of each component. Such configurations may be changed as appropriate depending on the technical level at the time of implementation.

[0147] 5. Summary The techniques described above can be specified, for example, as follows. One of the techniques disclosed is an operation assistance device 6. As described with reference to FIGS. 1 to 6 , the operation assistance device 6 includes an estimation unit 61 that estimates a movement constraint to be applied to the robot device 4 based on sensing information acquired when the user 1 remotely operates the robot device 4, a correction unit 62 that corrects the remote operation (e.g., movement command information) based on the estimation result of the estimation unit 61, and a generation unit 63 that generates feedback information to feed back at least one of the estimation result of the estimation unit 61 and the correction result of the correction unit 62 to the user 1.

[0148] According to the above-described operation assistance device 6, feedback information is generated to feed back the estimation result of the estimation unit 61 and the correction result of the correction unit 62 to the user 1. By providing feedback based on such feedback information, the user 1 can know what estimation, correction, etc. has been performed in the operation assistance device 6. This makes it possible to suppress a decrease in the operability of the user 1 in remotely operating the robot device 4.

[0149] 1 to 3 and the like, the robot device 4 includes one or more arms 42, and the motion constraints may include at least one of a constraint on the linear motion of the arm 42, a constraint on the translational motion of the two arms 42 (arm 42-R and arm 42-L), a constraint on the grasping motion of the object 2 by the arm 42, a constraint on the grasping motion of the object 2 with both hands by the two arms 42, a constraint on the rotational motion of the arm 42, a constraint on the twisting motion due to the relative rotational motion of the two arms 42, and a constraint on the collision avoidance motion of the arm 42 with the object 2. For example, based on the estimation results of such motion constraints, it is possible to correct the remote operation or generate feedback information.

[0150] 2 and 3 , the estimation unit 61 may estimate the intention of the user 1 based on the sensing information, and may estimate a motion constraint to be applied to the robot device 4 based on the estimated intention of the user 1. As described with reference to FIG. 3 , the intention of the user 1 may include at least one of an intention to approach the object 2, an intention to grasp the object 2, an intention to grasp the object 2 with both hands, an intention to move the object 2 straight, an intention to rotate the object 2, and an intention to twist the object 2. For example, by estimating such an intention of the user 1, it is possible to correct remote operation or generate feedback information based on the estimation result.

[0151] 2 and 3, the estimation unit 61 may estimate a situation related to remote operation and estimate the intention of the user 1 based on the estimated situation. As described with reference to FIG. 3, the situation related to remote operation may include at least one of the shape of the robot device 4, the shape of the target object 2, the posture of the robot device 4, the posture of the target object 2, the trajectory of the robot device 4, the trajectory of the target object 2, and the gaze trajectory of the user 1. For example, based on such a situation related to remote operation, the remote operation can be modified or FB information can be generated.

[0152] As described with reference to Fig. 4 etc., the FB information may include multiple motion constraints and their reliability estimated by the estimation unit 61. This allows multiple motion constraint candidates to be fed back to the user 1 together with their reliability.

[0153] As described with reference to Fig. 2 and Fig. 6, the estimation unit 61 may update the estimation result based on the response information to the FB information, and the correction unit 62 may correct the remote operation based on the updated estimation result. As described with reference to Fig. 6, the response information may include an instruction to correct the estimation result. This allows the movement of the robot device 4 to be closer to the movement intended by the user 1.

[0154] 5 and the like, the feedback to the user 1 based on the FB information may include at least one of a verbal notification, a superimposed image display, a reaction force output, and a vibration generation. For example, the feedback based on the FB information can be provided to the user 1 in various ways such as these.

[0155] 2 and the like, the operation assistance device 6 may include a storage unit 60 that stores and saves information used in remote operation (as history information 602). This makes it possible to obtain data for machine learning of the robot device 4, for example.

[0156] The operation assistance method described with reference to Figures 1 to 8 is also one of the disclosed techniques. The operation assistance method includes acquiring sensing information when the user 1 remotely operates the robot device 4 (steps S1 and S11), estimating motion constraints to be applied to the robot device 4 based on the acquired sensing information (steps S2 and S12), correcting the remote operation based on the estimation results (steps S3 and S19), and generating feedback information for feeding back at least one of the estimation results and the correction results to the user 1 (steps S3 and S13). As described above, this operation assistance method can also suppress deterioration in operability of remote operation.

[0157] The effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.

[0158] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0159] The present technology can also be configured as follows: (1) An operation assistance device comprising: an estimation unit that estimates operation constraints to be applied to a robot device based on sensing information acquired while a user is remotely operating the robot device; a correction unit that corrects the remote operation based on the estimation result of the estimation unit; and a generation unit that generates feedback information for feeding back at least one of the estimation result of the estimation unit and the correction result of the correction unit to the user. (2) The operation assistance device described in (1), wherein the robot device includes one or more arms, and the operation constraints include at least one of: a constraint on a straight movement of the arm; a constraint on a translational movement of two arms; a constraint on a grasping movement of an object by an arm; a constraint on a grasping movement of an object with both hands by the two arms; a constraint on a rotational movement of the arm; a constraint on a twisting movement due to a relative rotational movement of the two arms; and a constraint on a collision avoidance movement of the arm with the object. (3) The operation assistance device according to (1) or (2), wherein the estimation unit estimates the user's intention based on the sensing information, and estimates a behavioral constraint to be applied to the robotic device based on the estimated user's intention. (4) The operation assistance device according to (3), wherein the user's intention includes at least one of an intention to approach an object, an intention to grasp an object, an intention to grasp an object with both hands, an intention to move an object straight, an intention to rotate an object, and an intention to twist an object. (5) The operation assistance device according to (3) or (4), wherein the estimation unit estimates a situation related to the remote operation, and estimates the user's intention based on the estimated situation. (6) The operation assistance device according to (5), wherein the situation related to the remote operation includes at least one of a shape of the robotic device, a shape of an object, an attitude of the robotic device, an attitude of the object, a trajectory of the robotic device, a trajectory of the object, and a trajectory of the user's line of sight. (7) The operation assistance device according to any one of (1) to (6), wherein the feedback information includes a plurality of motion constraints estimated by the estimation unit and their reliability.(8) The operation assistance device according to any one of (1) to (7), wherein the estimation unit updates an estimation result based on response information to the feedback information, and the correction unit corrects the remote operation based on the updated estimation result. (9) The operation assistance device according to (8), wherein the response information includes an instruction to correct the estimation result. (10) The operation assistance device according to any one of (1) to (9), wherein the feedback to the user based on the feedback information includes at least one of verbal notification, superimposed image display, reaction force output, and vibration generation. (11) The operation assistance device according to any one of (1) to (10), further comprising a storage unit that stores and saves information used for the remote operation. (12) An operation assistance method including: acquiring sensing information when a user remotely operates a robot device; estimating a motion constraint to be applied to the robot device based on the acquired sensing information; correcting the remote operation based on the estimation result; and generating feedback information for feeding back at least one of the estimation result and the correction result to the user.

[0160] 100 Operation support system 1 User 2 Object 3 Operation support device 30 Memory unit 301 Program 31 UI unit 31-1 UI unit 31-2 UI unit 4 Robot device 41 Motion control unit 42 Arm 42-R Arm 42-L Arm 4a Torso 4b Elbow 5 Sensor device 5-1 Sensor device 5-2 Sensor device 6 Operation support device 60 Memory unit 601 Program 602 History information 61 Estimation unit 62 Correction unit 63 Generation unit 9 Network

Claims

1. An operation assistance device comprising: an estimation unit that estimates motion constraints to be applied to a robot device based on sensing information acquired when a user is remotely operating the robot device; a correction unit that corrects the remote operation based on the estimation result of the estimation unit; and a generation unit that generates feedback information for feeding back at least one of the estimation result of the estimation unit and the correction result of the correction unit to the user.

2. The operation assistance device according to claim 1, wherein the robot device includes one or more arms, and the movement constraints include at least one of: a constraint on the linear movement of the arms; a constraint on the translational movement of the two arms; a constraint on the grasping movement of an object by the arms; a constraint on the grasping movement of an object with both hands by the two arms; a constraint on the rotational movement of the arms; a constraint on the twisting movement caused by the relative rotational movement of the two arms; and a constraint on the collision avoidance movement of the arms with the object.

3. The operation assistance device according to claim 1, wherein the estimation unit estimates the user's intention based on the sensing information, and estimates a behavioral constraint to be applied to the robot device based on the estimated user's intention.

4. The operation assistance device of claim 3, wherein the user's intention includes at least one of an intention to approach an object, an intention to grasp an object, an intention to grasp an object with both hands, an intention to move an object in a straight line, an intention to rotate an object, and an intention to twist an object.

5. The operation assistance device according to claim 3, wherein the estimation unit estimates a situation related to the remote operation, and estimates the user's intention based on the estimated situation.

6. The operation assistance device according to claim 5, wherein the situation related to the remote operation includes at least one of the shape of the robot device, the shape of an object, the posture of the robot device, the posture of an object, the trajectory of the robot device, the trajectory of an object, and the trajectory of the user's line of sight.

7. The operation assistance device according to claim 1, wherein the feedback information includes a plurality of motion constraints estimated by the estimation unit and their reliability.

8. The operation assistance device according to claim 1, wherein the estimation unit updates the estimation result based on response information to the FB information, and the correction unit corrects the remote operation based on the updated estimation result.

9. The operation assistance device according to claim 8, wherein the response information includes an instruction to correct the estimation result.

10. The operation assistance device according to claim 1, wherein the feedback to the user based on the FB information includes at least one of: verbal notification, superimposed image display, reaction force output, and vibration generation.

11. The operation assistance device according to claim 1, further comprising a storage unit for storing and saving information used in the remote operation.

12. An operation assistance method comprising: acquiring sensing information when a user is remotely operating a robot device; estimating motion constraints to be applied to the robot device based on the acquired sensing information; correcting the remote operation based on the estimation result; and generating feedback information for feeding back at least one of the estimation result and the correction result to the user.