Remote operation control device, operation control system, remote operation control method, and program

Abstract

To provide a remote operation control device, an operation control system, a remote operation control method, and program which can widen objects to be handled and kinds of works.SOLUTION: A remote operation control device comprises: a distance holding instruction part which instructs an instruction to hold a distance between end effectors to be constant; and a control command generation part which modifies, according to a behavior instructed from the distance holding instruction part, target positions of the end effectors to hold the relative relationship between the end effectors based on movement instruction from an operator and which calculates the control command of an arm based on geometric restriction corresponding to the modified target positions of the end effectors and a constraint expression including a virtual spring which applies arbitrary power to an operation object operated by a robot.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a remote operation control device, an operation control system, a remote operation control method, and a program. [Background technology]

[0002] A technology has been developed to remotely control a robot that performs a task on an object (see, for example, Patent Document 1). The task of such a task is to have the robot grasp the object with its gripping part. In a typical remote-controlled robot, the operator directly commands the hand position of the multi-degree-of-freedom arm. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 21226 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technology was insufficient for remote-controlled robots equipped with multiple multi-degree-of-freedom arms. For example, in dual-arm tasks, it is important to maintain the relative relationship between both hands in addition to the object. With conventional technology, this inevitably increases the burden on the operator. Also, depending on the object, maintaining the relationship between both hands can affect the execution of the task and the outcome. For this reason, conventional technology had the problem of increasing dependency on the skill of the operator. This resulted in a problem of limiting the types of tasks and objects that can be performed by remote control.

[0005] The present invention has been made in consideration of the above problems, and aims to provide a remote operation control device, an operation control system, a remote operation control method, and a program that can expand the types of objects and tasks that can be handled. [Means for solving the problem]

[0006] (1) In order to achieve the above object, a remote operation control device according to one embodiment of the present invention is a remote operation control device that recognizes a movement of an operator, transmits the movement of the operator to a robot having a plurality of arms and end effectors connected to the arms for controlling an object to be operated by the robot, and includes: a distance maintenance instruction unit that issues an instruction to maintain a constant distance between the end effectors in a remote operation in which the robot operates an object to be operated; and a control command generation unit that calculates a control command for the arm from a constraint expression that, in response to an instruction from the distance maintenance instruction unit, modifies the target positions of the end effectors so as to maintain the relative relationship between the end effectors, and adds a virtual spring that applies an arbitrary force to an object to be operated by the robot to a geometric constraint corresponding to the modified target of the end effectors.

[0007] (2) In addition, in the remote control device according to one aspect described in (1) above, the control command for the arm may be a target joint angle of the arm, and when calculating the target joint angle, inverse kinematics may be solved under a constraint condition in which the virtual spring that applies an arbitrary force to an object operated by the robot is added to a geometric constraint corresponding to the modified target of the end effector.

[0008] (3) In addition, in the remote control device according to one embodiment described in (1) or (2) above, the force of the virtual spring may be a force f corresponding to f=k (spring constant) Δx by increasing or decreasing the target relative position by Δx.

[0009] (4) In addition, in a remote control device according to one embodiment of the present invention described in any one of (1) to (3) above, the force of the virtual spring may be independently controllable for each of six degrees of freedom of position and orientation in three-dimensional space.

[0010] (5) In addition, in a remote operation control device according to one embodiment of the present invention described in any one of (1) to (4) above, the control command generation unit may maintain a relative relationship between the end effectors by maintaining a coordinate transformation indicating a relative relationship between a second end effector connected to a second arm other than the first arm among the multiple arms, as viewed from a first end effector connected to a first arm among the multiple arms, constant.

[0011] (6) In addition, in a remote operation control device according to one embodiment of the present invention described in any one of (1) to (5) above, the control command generation unit may estimate an operation intention of the operator using at least one of the operation instructions and an image captured by an imaging device installed in the work environment, and may issue an instruction to maintain a constant relative relationship between the end effectors based on the estimated operation intention.

[0012] (7) In order to achieve the above object, a remote operation control system according to one embodiment of the present invention comprises a robot having a plurality of arms and end effectors connected to each of the arms, and an operation control device that controls the movement of the robot, the operation control device comprising: a distance maintenance instruction unit that issues instructions to maintain a constant distance between the end effectors, and a control command generation unit that calculates a control command for the arm from a constraint expression that modifies the target position of the end effector so as to maintain the relative relationship between the end effectors in response to an operation instructed by the distance maintenance instruction unit, based on an operator's movement instruction, and adds a virtual spring that applies an arbitrary force to an object operated by the robot to a geometric constraint corresponding to the modified target of the end effector.

[0013] (8) In order to achieve the above-mentioned object, a remote operation control method according to one embodiment of the present invention is an operation control method in an operation control device that controls the operation of a robot having a plurality of arms and end effectors connected to each of the arms, which operation control method issues an instruction to keep the distance between the end effectors constant, modifies the target positions of the end effectors so as to maintain the relative relationship between the end effectors in accordance with the operation instruction of an operator for the instructed operation, and calculates a control command for the arms from a constraint expression that adds a virtual spring that applies an arbitrary force to an object operated by the robot to a geometric constraint corresponding to the modified target of the end effector.

[0014] (9) In order to achieve the above object, a program according to one embodiment of the present invention causes a computer of an operation control device that controls the operation of a robot having a plurality of arms and end effectors connected to each of the arms to issue an instruction to maintain a constant distance between the end effectors, corrects the target positions of the end effectors in accordance with the operation instruction of an operator so as to maintain the relative relationship between the end effectors, and calculates a control command for the arms from a constraint expression that adds a virtual spring that applies an arbitrary force to an object operated by the robot to a geometric constraint corresponding to the corrected target of the end effector. Effect of the Invention

[0015] According to (1) to (9), the relative relationship between the hands can be maintained to the extent possible, and it is also possible to apply force to the work object, thereby expanding the types of objects and tasks that can be handled. According to (2), the relative position and orientation between the hands are introduced as constraints for the quadratic programming problem in inverse kinematics, and a virtual spring is also added to this, so any elasticity can be reproduced by changing the spring coefficient k according to the target object. Also, according to (2), it is possible to handle both attractive and repulsive forces depending on the sign. According to (4), the constraints are described independently for each of the six degrees of freedom (position and orientation) in three-dimensional space, so that elasticity can be exerted in any direction. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of a robot according to an embodiment and an example of a working environment. FIG. [Diagram 2] 1 is a diagram illustrating an example of the configuration of an operation control system according to a first embodiment. [Diagram 3] 5A to 5C are diagrams for explaining the content and procedure of a process for correcting the relative positions between both hands according to the first embodiment. [Figure 4] FIG. 13 is a diagram for explaining both hand restraint. [Diagram 5] 1A and 1B are diagrams for explaining an example of a repulsive force acting on a flexible operation target object; [Figure 6] FIG. 1 is a diagram illustrating an example of the configuration of an operation control system according to an embodiment. [Figure 7] 13A to 13C are diagrams for explaining the process contents and procedures for restricting the degree of freedom of operation in addition to restricting the relative relationship according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size. In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted. In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0018] (Robots, working environment) FIG. 1 is a diagram showing an example of the configuration of a robot according to this embodiment and an example of a working environment. The robot 2 includes, for example, an arm 23 (23R, 23L), an end effector 24 (24R, 24L), a body 26 (base), a head 27, and sensors 21 (21a, 21b, 21c, 21d) attached to the end effector 24 or the head 27. The end effector 24 includes at least two or more fingers. The sensor 21 is, for example, a camera that can also obtain depth information. The shape and configuration of the robot 2 shown in FIG. 1 are merely an example and are not limited thereto. For example, the robot 2 may include three or more arms.

[0019] The robot 2 is controlled by an operation control device 3 using operation information obtained from an operation input unit 4 operated by a worker and environmental information obtained from an environmental sensor 5. For example, as shown in FIG. 1, the robot 2 grasps an operation target object obj with both end effectors 24. The operation target object obj is, for example, a sheet-like or plate-like band-like member. Note that the operation target object obj may be any other object as long as it is an object that can be grasped by both end effectors 24.

[0020] First Embodiment (Configuration of Operation Control System) Next, a configuration example of the operation control system 1 will be described. 2 is a diagram showing an example of the configuration of the operation control system according to this embodiment. The operation control system 1 includes, for example, a robot 2, an operation control device 3, an operation input unit 4, an environmental sensor 5, an HMD 6, and a robot model storage unit 7. The robot 2 includes, for example, a sensor 21, a driving unit 22, an arm 23, an end effector 24, a driving unit 25, a body 26 (FIG. 1), and a head 27 (FIG. 1). The operation control device 3 includes, for example, an acquisition unit 31, a distance maintenance instruction unit 32, a control command generation unit 33, a control unit 34, a storage unit 35, and an image generation unit 36. The HMD 6 includes, for example, an image display unit 61 and a gaze detection unit 62 .

[0021] The operation input unit 4 is an input unit for the operator to input operation instructions to the robot 2, and is, for example, a data glove, an operation switch, etc. The operation input unit 4 and the operation control device 3 are connected wirelessly or by wire.

[0022] The environmental sensor 5 is, for example, an RGB (red, green, blue) D sensor that can also acquire depth information D. The environmental sensor 5 may be, for example, a distance sensor. The environmental sensor 5 and the operation control device 3 are connected wirelessly or by wire. The environmental sensor 5 may be provided in the robot 2, for example.

[0023] The HDM 6 is, for example, a head mounted display. The HDM 6 and the operation control device 3 are connected wirelessly or by wire. The HDM 6 includes, for example, an image display unit 61 and a gaze detection unit 62.

[0024] The image display unit 61 displays an image acquired from the operation control device 3. The operator inputs an operation instruction through the operation input unit 4 while viewing the image displayed on the image display unit 61. Note that the image display unit 61 may be an image display device such as an image display device of a personal computer or a monitor.

[0025] The line-of-sight detection unit 62 detects the line of sight of the operator, and outputs to the operation control device 3 an operation instruction including the detected line-of-sight information (operator sensor value).

[0026] The robot 2 will now be described. The robot 2 has at least two arms 23, each of which is connected to an end effector 24.

[0027] The sensor 21 is, for example, an RGBD sensor, a six-axis sensor, an acceleration sensor, a gyroscope sensor, a magnetic sensor, or the like. When a plurality of sensors are provided, for example, two sensors may be used to track the movement of each finger. When the sensor 21 is an RGBD sensor, the sensor 21 is attached, for example, to the head 27, the back or inside of the hand 24, or the wrist of the hand 24. When the sensor 21 is a six-axis sensor, the sensor 21 is attached, for example, to the fingertip of the finger portion 241. The detection value detected by the sensor 21 may include a timestamp.

[0028] The driving unit 22 includes a driving circuit and an actuator. The driving unit 22 drives the arm 23. The actuator is attached to, for example, a shoulder joint and an elbow joint. The actuator includes an encoder and a torque sensor.

[0029] The arm 23 is connected to the body 26. In the arm 23, for example, a first link connects the shoulder joint and elbow joint, and a second link connects the elbow joint and the wrist joint of the hand 24.

[0030] The end effector 24 has at least two fingers 241 and is a hand, a gripper, or the like.

[0031] The driving unit 25 includes a driving circuit and an actuator. The driving unit 25 drives the hand 24. The actuators are attached to, for example, a wrist joint and a finger joint. The actuators include an encoder and a torque sensor.

[0032] The operation control device 3 will be described. The acquisition unit 31 acquires information detected by each of the operation input unit 4, the environmental sensor 5, and the gaze detection unit 62 of the HMD 6. The acquisition unit 31 includes, for example, an image acquisition unit 311, a sensor information acquisition unit 312, and an operation information acquisition unit 313.

[0033] The image acquisition unit 311 acquires RGBD information from the environmental sensor 5 and RGBD information from the sensor 21 of the robot 2 .

[0034] The sensor information acquisition unit 312 acquires sensor detection values ​​from the environmental sensor 5 and sensor detection values ​​from the sensor 21 of the robot 2 .

[0035] The operation information acquisition unit 313 acquires an operation instruction from the operation input unit 4 .

[0036] The distance maintenance instruction unit 32 estimates the operator's intention to operate using information on the operation object obj, such as the shape and size of the operation object obj, and the acquired sensor detection value, and detects the timing or period for maintaining the relative relationship between the hands based on the estimation result. Alternatively, the distance maintenance instruction unit 32 may acquire an instruction from the operator regarding the timing or period for maintaining the relative relationship between the hands constant. The distance maintenance instruction unit 32 issues an instruction to maintain the relative relationship between the end effectors 24 constant at the timing or period for maintaining the estimated or acquired relative relationship between the hands constant. Note that maintaining the relative relationship between the hands means maintaining the relative relationship between the hands constant, but it is possible to change the inclination in the plane (xy plane) direction of the operation object, for example.

[0037] The control command generation unit 33 corrects the target positions of the end effectors 24 in response to the operation instructions received from the distance maintenance instruction unit 32 so as to maintain the relative relationship between the end effectors 24, and calculates the target joint angle of the arm 23 from the corrected target positions of the end effectors 24.

[0038] The control unit 34 controls the driving unit 22 to control the operation of the arm 23 and controls the driving unit 25 to control the operation of the end effector 24 based on the control command generated by the control command generating unit 33. The control unit 34 calculates the posture of the robot 2, for example, at predetermined time intervals, using the sensor information (including time stamps) and the three-dimensional shape model. The control unit 34 may calculate the posture of the arm 23 or the end effector 24 according to the work, but may not calculate the posture of the body 26 or the head 27, etc. The control unit 34 may also estimate taxonomy information (for example, see Reference 1) related to the work based on the information acquired by the operation information acquiring unit 313, and control the robot 2 using the estimated taxonomy information as well. The control unit 34 may acquire information related to the work object based on information acquired from the environmental sensor 5, for example. Alternatively, the control unit 34 may acquire information related to the work object by acquiring the result of setting or input by the operator, for example, operating the operation input unit 4. The information related to the work object is, for example, the name of the work object or the name of a material.

[0039] Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77

[0040] The storage unit 35 stores, for example, a program, a threshold value, etc. used by the control unit 34 for control. The storage unit 35 stores operation object information. Note that the operation object information may be stored in advance, may be input by the operator, or may be measured and obtained by the environmental sensor 5 or the like and stored. The storage unit 35 stores a spring coefficient for each material or each work object.

[0041] The image generation unit 36 ​​extracts an image of the work environment including the end effector 24 from an image captured by the environmental sensor 5, or generates the captured image as an image to be provided to the HMD 6, for example.

[0042] The robot model storage unit 7 stores a three-dimensional shape model of the robot 2, the geometric structures and dimensions of the arm 23 and the end effector 24, and the like.

[0043] (Relative position correction between hands) Correction of the relative positions between both hands will now be described. Fig. 3 is a diagram for explaining the content and procedure of the process of correcting the relative positions between both hands according to this embodiment.

[0044] (Step S1) The operation control device 3 acquires a hand instruction x for both arms input by the operator.

[0045] (Step S2) The operation control device 3 corrects the hand target using the constraint conditions for both hands for the acquired hand instruction x so as to maintain the relative relationship (both hands constraint; first-stage correction). By this process, the shapes of the left and right hands of the robot 2 are aligned to an arbitrary shape, as shown by symbols g1 and g2.

[0046] (Step S3) The operation control device 3 calculates the hand target x new Taking into account the geometric constraints of the two arms, a constraint equation with spring terms in the geometric constraints, described later, is solved to generate a joint angle target q (inverse kinematics; second-stage correction). The constraint conditions are constraints on the relative relationship between the two hands. The joint angle targets obtained by solving the constraints guarantee the corrected relationship between the two hands of the robot. The inverse kinematics, constraint conditions, etc. used in the second-stage correction will be described later.

[0047] (Step S4) The operation control device 3 outputs the joint angle target q corrected in the second stage to the robot 2 to control the movement of the robot 2.

[0048] In the example of this embodiment, a plurality of end effectors 24 can be used simultaneously or individually in response to an operation command from an operator. In this embodiment, by performing the processes of steps S2 and S3, a plurality of end effectors 24 can be used simultaneously in response to an operation command from an operator. Alternatively, in this embodiment, by performing the process of step S2 and not performing the process of step S3, the multiple end effectors 24 can be used individually in response to an operation command from the operator. Alternatively, in this embodiment, by not performing the process of step S2 and performing the process of step S3, the multiple end effectors 24 can be used individually in response to an operation command from the operator.

[0049] Even if the process of step S2 is performed and the process of step S3 is not performed, the burden on the operator and the dependency on the skill can be reduced. Also, if the process of step S2 is not performed and the process of step S3 is performed, a target value that aligns the relationship between the two hands is created. In this process, a feasible target value that satisfies the relationship between the two hands and hardware constraints is then generated in step S3. Therefore, even if the target value is not corrected in step S2 and is inappropriate, the effect of maintaining the relationship between the two hands can be obtained as long as the constraints are imposed in step S3. In other words, the processing in step S2 is a Want, and the processing in step S3 is a Must.

[0050] FIG. 4 is a diagram for explaining the restraint of both hands. Reference symbol g100 is an example of the position and posture of the left hand, with six degrees of freedom. Reference symbol g120 is an example of the position and posture of the right hand, with six degrees of freedom. The six degrees of freedom are the degree of freedom in the X-axis direction, the degree of freedom in the Y-axis direction, the degree of freedom in the Z-axis direction, the degree of freedom of rotation in the X-axis direction, the degree of freedom of rotation in the Y-axis direction, and the degree of freedom of rotation in the Z-axis direction. For example, in this embodiment, holding the right hand relative to the left hand (coordinate transformation) so that it remains constant (A is constant), that is, restraining it (reference symbol g110 in FIG. 4), is referred to as "restraining both hands."

[0051] In this type of both-hand restraint, for example, when a worker remotely controls the robot 2 to perform a task using both hands, the operation control device 3 first evaluates the position and posture of the approaching hand at each predetermined timing. If the evaluation result indicates that the position and posture of the hand should be maintained, the operation control device 3 restrains the relative relationship of the hands so as to maintain the relationship between the hands at that arbitrary timing. In this case, the operation object is specified or known in advance. Also, the operator's operation intention may be estimated by the distance maintenance instruction unit 32 based on, for example, the operation input information and the operation object.

[0052] In the example described using Figure 4, an example was described in which the relative relationship of the right hand as viewed from the left hand is constrained to be constant, but it is also possible to constrain the relative relationship of the left hand as viewed from the right hand to be constant.

[0053] (Intention Presumption) The distance maintenance instruction unit 32 estimates the operator's operation intention using, for example, at least one of the operator's gaze information and the operation instruction. The distance maintenance instruction unit 32 may also estimate the intention using the detection value of the environmental sensor 5, etc. The distance maintenance instruction unit 32 estimates the operator's operation intention, for example, by the Grasp Taxonomy method (see, for example, Reference 1). The distance maintenance instruction unit 32 classifies the operator state by classifying the posture, i.e., the grip posture, of the operator or the robot 2, for example, by the Grasp Taxonomy method, and estimates the operator's operation intention. The distance maintenance instruction unit 32 inputs, for example, at least one of the operator's gaze information, the operation instruction, and the detection value of the environmental sensor 5 into a learned model stored in the memory unit 35, and estimates the operator's operation intention (see, for example, Patent Application No. 2022-006498). The position and posture of each tip of the end effector 24 are detected by the control unit 34 using detection values ​​of the encoders of the sensors 21 attached to the arm 23 and the end effector 24. The control unit 34 may also estimate the operation intention.

[0054] In this manner, in this embodiment, the operator can operate one of the end effectors 24, while the system constrains the control target (inverse kinematics input) of the hand position and orientation of the other end effector. In this embodiment, in order to deal with cases where the relative relationship cannot be maintained due to differences in geometric characteristics between the two arms, such as the reach range and self-interference of the arm 23, constraints are added in the inverse kinematics so that deviations in the relative relationship fall within an allowable range. Inverse kinematics in this case refers to constrained mathematical optimization using quadratic programming, and the relative relationship is described as one of the constraint conditions. As a result, according to this embodiment, the burden on the operator can be reduced.

[0055] (Control Principle) In order to realize multiple targets such as the hand position and center of gravity position of a robot by constraint conditions, inverse kinematics using quadratic programming is known. In this embodiment, the inverse kinematics is solved in the acceleration dimension to obtain the target joint angular acceleration or the target observation angle velocity, and the constraints on the relationship between the two hands in the following equation (2) are converted into the form of a quadratic programming problem in equation (1) using Jacobian or matrix calculations, for example, and solved to generate a torque target value that serves as a control command. The following equation (1) shows a quadratic programming problem generalized as an n-dimensional vector. Note that acceleration is used to generate a torque command value from inverse dynamics. On the other hand, if there is no need to be limited to torque or acceleration, velocity can also be used.

[0056]

number

[0057] In equation (1), x(∈R n ) is a variable vector and joint angle target, and Q(∈R n×n , n is a natural number, R is a real number) is a constant matrix, and c(∈R n ) is a constant vector, and T represents the transpose. The joint angle target x is the joint angular acceleration q ··If the constraints can be expressed as a coefficient matrix A and a vector b that is a boundary condition that determines the constraints, then any constraint can be set. In this embodiment, the boundary conditions are constraints between the two hands, and for example, the relative relationship of the right hand as seen from the left hand is allowed to be a1 in the x direction and a2 in rotation.

[0058] In a dual-arm robot arm, the relative position and orientation between the hands is expressed as d∈R. 6 The target relative position and orientation is d * , d * If the allowable error is Δd, the relationship between the relative position and orientation to be maintained is expressed by the following equation (2).

[0059]

number

[0060] In addition, in equation (2), the target relative position and orientation (6 dimensions) d * includes a spring term, as in the second term on the right-hand side of the following equation (3).

[0061]

number

[0062] In formula (2), the first term on the right side represents the relative position and posture to be maintained. The second term on the right side represents a virtual spring, where f is force and k is the spring coefficient. Here, (f / k)<0 represents an attractive force, and (f / k)>0 represents a repulsive force. In other words, the force of the virtual spring is a force f that corresponds to f=kΔx by increasing or decreasing the target relative position by Δx. Also, formulas (2) and (3) are simplified, but each element has six degrees of freedom of position and posture. That is, each of the six degrees of freedom also has a virtual spring. For this reason, the distance maintenance instruction unit 32 may switch the value of the virtual spring to an on state or an off state for each degree of freedom. Alternatively, the distance maintenance instruction unit 32 may set the value of the virtual spring for each degree of freedom. These controls may be switched during work depending on the work content, work state, intention, etc.

[0063] In this manner, in this embodiment, in remote control of a robot, the target posture is changed by the amount of the virtual spring with respect to the position and posture of both arms that is to be maintained. For example, when it is desired to grasp and pull a work object with both end effectors 24, d * If you want to press the button, set (f / k)>0 so that the button is larger. * The distance maintenance instruction unit 32 controls the maintenance operation by controlling or switching the positive / negative and magnitude of (f / k) in this way. Note that the distance maintenance instruction unit 32 determines whether the work is a pulling work or a pushing work based on, for example, instructions from the worker or an estimated intention.

[0064] Then, by expanding the formula (2), it becomes possible to maintain the relative relationship of the dual-arm robot arms as a constraint condition of the inverse kinematics.

[0065] FIG. 5 is a diagram for explaining an example of a repulsive force acting on a flexible operation target object. In the example of Fig. 5, only the force pulling the operation target in the horizontal direction is exerted, and the virtual spring characteristics are not given in the vertical direction or the torsional direction. Note that in Fig. 5, the arrows shown on each end effector are the hand targets in the xyz directions. In this way, according to this embodiment, since a force can be applied in any direction, it is possible to adapt it to the situation.

[0066] As described above, in this embodiment, a term of a virtual spring is added to the expression of the constraint condition of "the relative position and orientation between the hand tips as the constraint condition of the quadratic programming problem in the inverse kinematics" in order to generate a force. As a result, according to this embodiment, any elasticity can be reproduced by changing the spring coefficient k according to the object. As described above, this embodiment can handle both attractive and repulsive forces depending on the sign of (f / k). Furthermore, according to this embodiment, the constraints are described independently for each of the six degrees of freedom (position and orientation) in three-dimensional space, so that elasticity can be exerted in any direction.

[0067] The control of the virtual spring during holding may be performed, for example, with respect to the resultant force in the xyz directions, or may be performed in one direction such as the y direction (for example, the vertical direction or the horizontal direction). The same applies to the torsional composition of rotation. In this manner, in this embodiment, each of the six degrees of freedom (position and orientation) can be set and controlled independently, that is, a virtual spring is set for each of the six degrees of freedom. The control method of this embodiment can be used for both translational and rotational movements. In addition to the above-mentioned examples, the method can also be applied to, for example, wringing out a cloth such as a dusting cloth, twisting a string, etc.

[0068] According to this embodiment, the relative relationship between the hands is maintained to the extent possible, and it is also possible to apply force to an object, thereby expanding the types of objects and tasks that can be handled.

[0069] <Second embodiment> In this embodiment, in addition to restricting the relative relationship, the degree of freedom of operation is also restricted.

[0070] (Configuration of Operation Control System) An example of the configuration of the operation control system 1A will be described. 6 is a diagram showing an example of the configuration of the operation control system according to this embodiment. The operation control system 1A includes, for example, a robot 2, an operation control device 3A, an operation input unit 4, an environment sensor 5, an HMD 6, and a robot model storage unit 7. The robot 2 includes, for example, a sensor 21, a driving unit 22, an arm 23, an end effector 24, a driving unit 25, a body 26 (FIG. 1), and a head 27 (FIG. 1). The operation control device 3A includes, for example, an acquisition unit 31, a distance maintenance instruction unit 32, a control command generation unit 33A, a control unit 34, a storage unit 35, an image generation unit 36, and an operation freedom restriction unit 37. The HMD 6 includes, for example, an image display unit 61 and a gaze detection unit 62 .

[0071] The operation freedom restriction unit 37 restricts the degree of freedom that the operator can operate in response to an action instruction from the operator.

[0072] The control command generating unit 33A modifies the target position of the end effector in response to an operator's motion instruction, whose degree of freedom has been restricted by the operation freedom restricting unit 37, in response to an instruction given by the relative relationship maintenance instruction unit 32, so as to maintain the relative relationship between the end effectors.

[0073] Next, an example of the process contents and process steps will be described. FIG. 7 is a diagram for explaining the process contents and procedures for restricting the degree of freedom of operation in addition to restricting the relative relationship according to this embodiment.

[0074] (Step S101) The operation control device 3A acquires a hand instruction x input by the operator.

[0075] (Step S102) The operation control device 3A limits the degree of freedom that the operator can operate with respect to the acquired hand instruction x (limitation of the degree of freedom of operation).

[0076] (Step S103) The operation control device 3A controls the limited hand input x new In response to this, the hand target is corrected to maintain the relative relationship between the two hands, as shown by symbol g330 (both hands are constrained; first stage correction).

[0077] (Step S104) The operation control device 3A calculates the hand target x' corrected in the first stage. new For (symbol g340), a joint angle target q having a spring term in the geometric constraint is generated taking into account the geometric constraint of the dual arms (inverse kinematics; second stage correction).

[0078] (Step S105) The operation control device 3A outputs the joint angle target q corrected in the second stage to the robot 2 to control the movement of the robot 2.

[0079] In this way, in this embodiment, in addition to the first embodiment, a function for restricting the degree of freedom of the operable hand position and posture is combined. As a result, according to this embodiment, the burden on the operator can be further reduced.

[0080] The work object in the first and second embodiments described above is a flexible object that may break when a force larger than a predetermined force is applied, such as a sheet-like or plate-like belt-like member. The belt-like member is, for example, a member wound in a roll.

[0081] The method of this embodiment is not limited to the robot 2 having two arms, but can also be applied to an arm having three or more degrees of freedom. In this case, joint angle targets are generated for the multiple arms under corrections and constraints, or the corrections and constraints described in the second embodiment. The method of this embodiment can also be applied to combinations of various types of multi-degree-of-freedom arms with different geometric structures. In this case, if the shape or number of joints of the arm or end effector changes, the above-mentioned method can be used because the coefficient matrix A in equation (1) contains Jacobian information including joint information.

[0082] Furthermore, in the method of this embodiment, the allowable deviation can be set for each of the six degrees of freedom in inverse kinematics. As a result, in the method of this embodiment, the allowable amount of a specific axis can be intentionally increased, which is expected to improve operability in a specific task. In other words, since the relative relationship between the hands is included in vector b in formula (1), the error tolerance for each of the six degrees of freedom can be arbitrarily set as a six-dimensional vector. For example, if the allowable error in the x direction is strict and a certain degree of allowable error can be allowed in the y direction, the width of the constraint (tolerance) can be set for each degree of freedom, such as setting the allowable error in the x direction to, for example, 1 mm and the allowable error in the y direction to, for example, 10 mm. Note that such error tolerance is set by the operation control device 3, for example, based on the estimated operation intention of the operator.

[0083] In addition, a program for realizing all or part of the functions of the operation control device 3 in the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processing performed by the operation control device 3. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to storage devices that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0084] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0085] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0086] 1,1A...operation control system, 2...robot, 3,3A...operation control device, 4...operation input unit, 5...environment sensor, 6...HMD, 7...robot model memory unit, 21,21a,21b,21c,21d...sensor, 22...drive unit, 23,23R,23L...arm, 24,24R,24L...end effector, 25...drive unit, 26...body, 27...head, 31...acquisition unit, 32...distance maintenance instruction unit, 33,33A...control command generation unit, 34...control unit, 35...memory unit, 36...image generation unit, 37...operation freedom restriction unit, 311...image acquisition unit, 312...sensor information acquisition unit, 313...operation information acquisition unit, 61...image display unit, 62...gaze detection unit

Claims

1. A remote control system for recognizing an operator's movement, transmitting the operator's movement to a robot having a plurality of arms and end effectors connected to the arms for manipulating an object to be operated by the robot, a distance maintenance instruction unit that issues an instruction to maintain a constant distance between the end effectors; In response to an operation instructed by the distance maintenance instruction unit, The operation instruction of the operator is corrected to a target position of the end effector so as to maintain a relative relationship between the end effectors; a control command generation unit that calculates a control command for the arm from a constraint expression that adds a virtual spring that applies an arbitrary force to an object to be operated by the robot to a geometric constraint corresponding to the corrected target of the end effector; A remote control device comprising:

2. the control command for the arm is a joint angle target for the arm, When calculating the joint angle target, an inverse kinematics is solved under a constraint condition in which a geometric constraint corresponding to the modified target of the end effector is added to the virtual spring that applies an arbitrary force to an object operated by the robot. The remote control device according to claim 1 .

3. The virtual spring force is a force f corresponding to f=k (spring coefficient) Δx by increasing or decreasing the target relative position by Δx. The remote control device according to claim 1 .

4. The force of the virtual spring can be controlled independently for each of the six degrees of freedom of position and orientation in three-dimensional space. The remote control device according to claim 1 .

5. The control command generating unit is maintaining a constant coordinate transformation indicating a relative relationship between a first end effector connected to a first arm of the plurality of arms and a second end effector connected to a second arm other than the first arm, thereby maintaining a relative relationship between the end effectors; The remote control device according to claim 1 .

6. The control command generating unit is using at least one of the operation instruction and an image captured by an image capture device installed in the work environment, an operation intention of the operator is estimated, and an instruction to maintain a constant relative relationship between the end effectors is issued based on the estimated operation intention. The remote control device according to claim 1 .

7. A robot including a plurality of arms and an end effector connected to each of the arms; and an operation control device that controls an operation of the robot, The operation control device includes: a distance maintenance instruction unit that issues an instruction to maintain a constant distance between the end effectors; In response to an operation instructed by the distance maintenance instruction unit, Correcting the target positions of the end effectors based on the motion instructions of the operator so as to maintain the relative relationship between the end effectors; a control command generation unit that calculates a control command for the arm from a constraint expression that adds a virtual spring that applies an arbitrary force to an object to be operated by the robot to a geometric constraint corresponding to the corrected target of the end effector; An operation control system comprising:

8. A remote operation control method for an operation control device that controls the operation of a robot having a plurality of arms and end effectors connected to each of the arms, comprising: issuing an instruction to keep the distance between the end effectors constant; For the operation instructed above, Correcting the target positions of the end effectors based on the motion instructions of the operator so as to maintain the relative relationship between the end effectors; Calculating a control command for the arm from a constraint expression that adds a virtual spring that applies an arbitrary force to an object to be operated by the robot to a geometric constraint corresponding to the modified target of the end effector; Remote operation control method.

9. A computer of an operation control device that controls the operation of a robot having a plurality of arms and end effectors connected to each of the arms, issuing an instruction to keep the distance between the end effectors constant; For the operation instructed above, The operation instruction of the operator is corrected to a target position of the end effector so as to maintain a relative relationship between the end effectors; Calculating a control command for the arm from a constraint expression that adds a virtual spring that applies an arbitrary force to an object to be operated by the robot to a geometric constraint corresponding to the modified target of the end effector; program.

Images