Operation control method, operation control device and program

Abstract

To provide an operation control method capable of performing object operation without using object information, an operation control device and a program.SOLUTION: An operation control method includes a step for sending acting force information when an operator operates an object to an end effector, a calculation step for calculating target acting force for holding the object by the end effector from an acquired state of the operator and the acting force information, a detection step for detecting contact force at a contact point at which a contact part is in contact with the object, and a control step for controlling movement of a drive part of the end effector so as to make acting force acting on the object from each contact element match the target acting force. In the calculation step, it sets a central coordinate system with the central coordinates of a contact point command value of the end effector calculated from the acting force information as an original point, calculates a contact point position of the end effector in the central coordinate system, and uses a command value based on a contact point position obtained by converting the coordinates of the contact point information according to a change in the central coordinate system calculated from updated acting force information as a correction command value.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an operation control method, an operation control device, and a program. [Background technology]

[0002] When a human remotely controls a robot to manipulate an object, it has been proposed to apply information such as the joint angles of the operator's fingers obtained from sensors such as a data glove to the robot (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-14166 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, robots may not be able to manipulate objects properly due to factors such as sensor errors, differences in finger length between robots and humans, and the lack of force-based commands. Even if a robot were to recognize an object and perform manipulation somewhat automatically, accurate recognition itself was difficult with conventional technology due to factors such as occlusion. In other words, with conventional technology, object manipulation without using object information was difficult.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an operation control method, an operation control device, and a program that enable object manipulation without using object information. [Means for solving the problem]

[0006] (1) In order to achieve the above object, one aspect of the present invention provides an operation control method for an end effector having a plurality of contact portions capable of manipulating an object, the operation control method comprising: a step of sending to the end effector actuation force information when an operator manipulates the object; a calculation step of calculating a target actuation force for gripping the object by the end effector from the acquired state of the operator and actuation force information; a detection step of detecting contact forces at contact points where the contact portions are in contact with the object; and a control step of controlling operation of a drive unit of the end effector so that the actuation forces acting on the object from each contact element match the target actuation force. The calculation step sets a central coordinate system whose origin is the central coordinate of a contact point command value of the end effector calculated from the actuation force information, calculates a contact point position of the end effector in the central coordinate system, coordinate-transforms the contact point position in accordance with the change in the central coordinate system calculated from the updated actuation force information, and uses a command value based on the coordinate-transformed contact point position as a corrected command value.

[0007] (2) In the operation control method according to one aspect of (1) above, the calculation step may calculate a contact tangent component at the contact point that is a difference between the displacement of the acting force information and the displacement of the contact point position due to coordinate transformation, and add the contact tangent component to the coordinate-transformed contact point position to use the result as a correction command value.

[0008] (3) In the operation control method according to one aspect of (2) above, the contact point position may be updated using the contact tangent component, and the contact point position may be subjected to coordinate transformation in accordance with a change in the central coordinate system calculated from the next updated acting force information.

[0009] (4) In the operation control method according to any one of the above (1) to (3), the control step may determine whether the operator is instructing the end effector having a plurality of finger portions to move away from the object after the end effector comes into contact with the object, or whether the operator is instructing the object to move while holding it, and whether the operator's fingers are outside the surface of the object or inside the object, based on the relative vectors and normal directions of each of the plurality of finger portions in the central coordinate system.

[0010] (5) In the operation control method according to any one of the above (1) to (3), the control step may set a tentative contact point position within a predetermined range on the surface of the object, and determine whether the operator is instructing the end effector to move away from the object after the end effector has come into contact with the object, or whether the operator is trying to stop the end effector from moving away from the object before the end effector moves away, based on the tentative contact point position and a command value given by the operator.

[0011] (6) In the operation control method according to any one of the above (1) to (5), the end effector may have a plurality of finger portions, and the central coordinate system may be configured such that one of the plurality of finger portions is used as a reference, an axial direction passing through the central coordinate from the reference finger portion is set as a first axial direction, a second axial direction is set within a plane including the first axial direction, and a third axial direction is set relative to the plane.

[0012] (7) In the operation control method according to any one of the above (1) to (6), the acting force information may be a joint angle of a finger of the operator, and the central coordinate system may be a contact point command value of the end effector calculated by matching the joint angle of the end effector to the joint angle of the operator.

[0013] (8) In order to achieve the above object, one aspect of the present invention provides an operation control device for an end effector having a plurality of contact portions capable of manipulating an object, the operation control device comprising: an output unit that sends acting force information when an operator manipulates the object to the end effector; a calculation unit that calculates a target acting force for gripping the object by the end effector from the acquired state of the operator and acting force information; a detection step that detects contact forces at contact points where the contact portions are in contact with the object; and a control unit that controls the operation of a drive unit of the end effector so that the acting forces acting on the object from each contact element match the target acting force, wherein the calculation unit sets a central coordinate system whose origin is the central coordinate of a contact point command value of the end effector calculated from the acting force information, calculates the contact point position of the end effector in the central coordinate system, coordinate-transforms the contact point position in accordance with the change in the central coordinate system calculated from the updated acting force information, and uses a command value based on the coordinate-transformed contact point position as a corrected command value.

[0014] (9) In order to achieve the above object, a program according to one aspect of the present invention is a program that causes a computer of an operation control device for an end effector having a plurality of contact portions capable of manipulating an object to send to the end effector information on acting force exerted when an operator manipulates the object, calculate a target acting force for gripping the object by the end effector from the acquired state of the operator and acting force information, detect contact forces at contact points where the contact portions are in contact with the object, control the operation of a drive unit of the end effector so that the acting forces exerted on the object from each contact element coincide with the target acting force, set a central coordinate system whose origin is the central coordinate of a contact point command value of the end effector calculated from the acting force information, calculate a contact point position of the end effector in the central coordinate system, coordinate-transform the contact point position in accordance with the change in the central coordinate system calculated from the updated acting force information, and use a command value based on the coordinate-transformed contact point position as a corrected command value. [Effects of the Invention]

[0015] According to the above (1) to (9), it is possible to perform object manipulation without using object information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of an operation control system according to an embodiment. [Figure 2] FIG. 10 illustrates an example of an end effector with multiple fingers. [Figure 3] FIG. 10 is a diagram for explaining center coordinates according to the embodiment. [Figure 4] FIG. 10 is a diagram for explaining an example of a constraint. [Figure 5] 10 is a flowchart of a remote control process according to the embodiment. [Figure 6] FIG. 10 is a diagram for explaining a first example of a process for generating a correction command value. [Figure 7] FIG. 10 is a diagram for explaining a second example of the process for generating a correction command value. [Figure 8] 10A and 10B are diagrams for explaining an operation when the operator's finger is far away from the object. [Figure 9] 10A and 10B are diagrams for explaining how to distinguish between an operation of lifting a finger and an operation of moving an object. [Figure 10] FIG. 10 is a diagram for explaining a control method when an object is grasped with three fingers and then one finger is released from the object. [Figure 11] 10 is a flowchart of a process for determining finger release or the like according to the embodiment. [Figure 12] 10 is a flowchart of a process of determining whether or not a contact has occurred again according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of an operation control method, an operation control device, and a program according to the present invention will be described with reference to the drawings. Note that in the drawings used in the following description, the scale of each component has been appropriately changed so that each component is large enough to be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0018] [overview] In this embodiment, for example, an object is manipulated by a robot during remote control. In this embodiment, an operator (hereinafter sometimes referred to as a "pilot") controls the arm and hand based on images captured by a camera attached to the robot until the hand equipped with the end effector comes into contact with the object. In this embodiment, grasping support is then provided by calculating the balance of forces based on the specified grasping strength and contact sensor information on the hand, rather than object information, and controlling the finger forces to manipulate the object. Note that object manipulation includes, for example, manipulating the object's orientation and rotation direction, moving (sliding) a finger on the object, and releasing a finger after touching the object. Furthermore, support refers to the manipulation control device correcting the instruction value given by the operator during remote control and appropriately operating the end effector to manipulate the object.

[0019] [Example of operation control system configuration] Fig. 1 is a diagram showing an example of the configuration of an operation control system according to this embodiment. As shown in Fig. 1, the operation control system 1 includes, for example, an end effector 10 (robot), an operation control device 2 (remote control device), an operation input unit 3, an environmental sensor 4, and an image display unit 5. The end effector 10 includes, for example, finger units 11, a force sensor 12, a tactile sensor 13, a drive unit 14, a control unit 15, and a communication unit 16. The end effector 10 may also include an imaging unit on, for example, the back of the hand. The operation control device 2 includes, for example, an acquisition unit 21, a setting unit 22, a detection unit 23, a calculation unit 24, a communication unit 25, a storage unit 26, and a control unit 27. The calculation unit 24 includes, for example, a center coordinate calculation unit 241, a contact point position calculation unit 242, a coordinate conversion unit 243, and a tangential component calculation unit 244.

[0020] The operation input unit 3 is, for example, a data glove, and is worn on the hand of the operator. The operation input unit 3 detects the results of operations performed by the operator and outputs them to the operation control device 2. The data glove includes, for example, a number of sensors 31 and an output unit 32.

[0021] The sensor 31 is, for example, a bending sensor, an eversion sensor, a sensor that measures the curvature of the palm (palmar arch), and the flexion and abduction of the wrist. The sensor 31 detects the movement of the user's hand and fingers as joint angle data. The joint angles may be detected by the operation input unit 3 or by the operation control device.

[0022] The environmental sensor 4 is, for example, an RGB (red, green, blue) D imaging device that can also measure depth information D. The environmental sensor 4 is installed, for example, within a range where it can image an object or the end effector 10. There may be multiple environmental sensors 4.

[0023] The image display unit 5 is, for example, an HMD (head mounted display) or an image display device. The image display unit 5 is installed on the operator's side. The operation control device 2 and the image display unit 5 are connected by wire or wirelessly. If the image display unit 5 is an HMD, the HMD may be configured to detect the operator's line of sight. Then, the HMD may be configured to output the detected line of sight information to the operation control device 2.

[0024] The end effector 10 is a multi-fingered hand having, for example, three fingers 11. The end effector 10 is connected to an arm (not shown).

[0025] The force sensor 12 may be, for example, a pressure sensor or a six-axis force sensor (6AF sensor) capable of measuring the magnitude of force or torque acting in multiple directions in real time.

[0026] The tactile sensor 13 is, for example, a tactile sensor that detects information sensed by the sense of touch of a human hand and converts it into an electrical signal. Note that the contact position between the object and the finger 11 may be detected using a known contact position detection algorithm based on the detection value detected by a six-axis sensor.

[0027] The drive unit 14 includes, for example, a drive circuit and an actuator. The drive unit 14 drives each joint of the end effector 10 under the control of the control unit 15.

[0028] The control unit 15 transmits the detection values ​​detected by each sensor to the operation control device 2 via the communication unit 16. The control unit 15 acquires the control information transmitted by the operation control device 2 via the communication unit 16.

[0029] The communication unit 16 transmits the information output by the control unit 15 to the operation control device 2. The communication unit 16 receives the control information transmitted by the operation control device 2 and outputs the received control information to the control unit 15.

[0030] The operation control device 2 controls the end effector 10. The operation control device 2 and the operation input unit 3 are connected to each other by wire or wirelessly. The operation control device 2 and the environmental sensor 4 are connected to each other by wire or wirelessly. The operation control device 2 may be provided in a robot or the like equipped with the end effector 10.

[0031] The acquisition unit 21 acquires the results of the operation performed by the operator (the state of the operator) from the operation input unit 3. The acquisition unit 21 acquires the detection values ​​of the force sensor 12 as state quantities of the device (such as the joint angles of the hand). The acquisition unit 21 acquires images (including depth information) captured by the environment sensor 4, and performs image processing on the information contained in the acquired images to acquire state quantities of the object to be operated (such as the position and orientation of the object).

[0032] The setting unit 22 sets a target acting force for gripping an object by the end effector 10. The setting unit 22 sends acting force information when the operator manipulates an object to the end effector 10. The setting unit 22 calculates the target acting force from the state of the operator and the acting force information acquired by the acquisition unit 21. That is, the target acting force, which is the target for force balance, is calculated by combining, for example, the gripping force instructed by the operator (for example, information instructed by a data glove) and operator information (various information such as the size of the operator's hand and line of sight).

[0033] The detection unit 23 uses the information acquired by the acquisition unit 21 to detect the contact point that is in contact with the object, and the contact force at the contact point that is in contact with the object.

[0034] The calculation unit 24 calculates a target acting force for gripping an object by the end effector 10 from the state of the operator and acting force information from the sensor 31.

[0035] The center coordinate calculation unit 241 calculates center coordinates whose origin is the center coordinate of the contact point command value of the end effector 10 calculated from the acting force information. In the following description, a coordinate system centered on the center coordinates is referred to as a center coordinate system. In the center coordinate system, for example, the x-axis direction, y-axis direction, and z-axis direction are set with the first finger (for example, the thumb) as the reference. Note that the center coordinates are coordinates in the world coordinate system. Furthermore, the center coordinates change according to the fingertip position because the fingertip position (contact point position) changes according to the operator's instruction.

[0036] The contact point position calculation unit 242 calculates the contact point position pi (i is an integer equal to or greater than 1) of the end effector 10 with respect to the object in the central coordinate system. p1 is, for example, the position of the thumb tip, p2 is, for example, the position of the index finger tip, and p3 is, for example, the position of the ring finger tip. At the start of grasping, the contact point position calculation unit 242 stores the robot's fingertip position pi (at time t=1) on the coordinate system of the central coordinates in the storage unit 26. After the start of grasping, for example, at time t=2, the contact point position calculation unit 242 stores the newly contacted fingertip position as the robot's fingertip position pi on the central coordinates in the storage unit 26. Furthermore, for example, at time t=2, the contact point position calculation unit 242 updates the fingertip position that had been in contact up to that point by calculating it from the final command value up to that point, and stores it in the storage unit 26 as the updated fingertip position pi. The fingertip position is expressed as a position with the central coordinate in the world coordinate system as the origin.

[0037] The coordinate conversion unit 243 converts the coordinates of the contact point position pi stored in the storage unit 26 in accordance with the change in the calculated center coordinates from the acting force information at the updated time. In other words, the coordinate conversion unit 243 converts the fingertip position pi recorded in the storage unit 26 at the time of gripping in accordance with the change in the center coordinates, thereby converting it into the fingertip position after the change in the center coordinates.

[0038] The tangential component calculation unit 244 calculates the contact tangential component at the contact point of the difference between the displacement of the acting force information before and after the update and the displacement of pi due to the coordinate transformation.

[0039] The communication unit 25 transmits and receives information to and from the end effector 10 .

[0040] The storage unit 26 stores, for example, programs, mathematical formulas, threshold values, etc. used by the operation control device 2. The storage unit 26 stores the positions where the sensors (force sensor 12, tactile sensor 13) are attached. The storage unit 26 stores the relationship between the size and mass of an object in a table format or in the form of a mathematical formula.

[0041] The control unit 27 generates a command value based on the result of the operator's operation. The control unit 27 outputs the generated command value to the end effector 10 via the communication unit 25, and causes the contact portion of the end effector 10 to contact the object. The contact portion is, for example, the finger portion 11, the fingertip of the finger portion 11, or the pad of the finger portion 11. After contact, the control unit 27 generates a corrected command value by adding the contact tangential component to the coordinate-transformed contact point position pi. After the finger portion 11 of the end effector 10 contacts the object, the control unit determines whether the operation is to remove the finger portion 11 from the object or to move the object while holding it, and corrects the command value accordingly. The control unit 27 outputs the corrected command value to the end effector 10 via the communication unit 25. The control unit 27 provides the image captured by the environmental sensor 4 to the image display unit 5 via the communication unit 25.

[0042] End Effector An example of an end effector 10 with multiple fingers will now be described. Fig. 2 is a diagram showing an example of an end effector with multiple fingers. Note that Fig. 2 shows an example with four fingers 11, but the number of fingers 11 is not limited to this and may be, for example, two, three, or five or more. The fingertip of the finger portion 11 is provided with, for example, a force sensor 12 (12a, 12b, 12c, 12d) which is a six-axis sensor. Furthermore, tactile sensors 13 are attached to the pads of the fingers, palms, etc. Also, as indicated by the symbol g1, for example, rubber is attached to the fingertip of the finger portion 11.

[0043] The rubber may be an object that generates friction against an object to a certain extent, or a protrusion, etc. In the following description, the functional part of the end effector 10, including the fingers 11 and the palm, is also referred to as a "hand."

[0044] [Center coordinates, center coordinate system] FIG. 3 is a diagram for explaining the center coordinates according to this embodiment. Image g10 is an image of the operator's hand during remote control. Note that the data glove is omitted from image g10. The work involves, for example, grasping an object and moving the object while grasping it.

[0045] Image g20 shows the contact position between the fingertip and the object g27 when the robot's finger 11 grasps the object g27 by remote control, and the position of the fingertip based on the command value. The black circle g21 shows the position of the robot's fingertip when grasping. The chain-line circle g22 shows the position of the fingertip based on the robot's fingertip command value calculated by matching the joint angle of the robot to the joint angle of the operator measured with a data glove or the like.

[0046] During remote operation, the fingertip position based on the operator's instruction may enter the object g27, as shown in image g20. In this embodiment, the state of image g20 is permitted. In this embodiment, the center of the fingertip position based on the command value after the robot's fingertip makes contact is defined as the "center coordinate g24." The center coordinate also represents a virtual object posture command value. In the following example, the center coordinate is calculated from the operator's applied force information (such as a contact point position command value), but this is not limited to this. For example, the operator's object posture command value may be input in any form (such as by operating another device simultaneously with the hand wearing the data glove, or through some other interface), or may be input independently of the applied force information. The center coordinate g24 is, for example, the load center or center of gravity of a triangle g23 formed by the three fingertip positions. In this embodiment, the x-axis, y-axis, and z-axis directions are set relative to this center coordinate, for example, based on a specific finger (e.g., the thumb). The x-axis direction is, for example, from the thumb to the center coordinate. The y-axis direction is, for example, the depth direction. The z-axis direction is, for example, the thickness direction of the object. The reference finger is not limited to the thumb, but may be another finger. In this way, the central coordinate system uses one of the multiple finger portions 11 as a reference, and the axis direction from the reference finger portion passing through the central coordinate is set as the first axis direction (x-axis direction), the second axis direction (y-axis direction) is set within a plane including the first axis direction, and the third axis direction (z-axis direction) is set relative to the plane. In this embodiment, if there is an excess force in this central coordinate system after grasping, a correction command value that imposes a constraint on the excess force is generated, and the end effector 10 is controlled.

[0047] The force command changes from moment to moment in response to the remote control command. The generation of such command values ​​and force balance is achieved using, for example, the method described in Japanese Patent Application No. 2023-046654. In this method, the target acting force is a scalar value based on the operator's operation. The weight of the object and the estimated gripping force based thereon are also calculated using, for example, the method described in Japanese Patent Application No. 2023-046654. Therefore, the information used to manipulate the object is the target acting force and contact information between the object and the fingers of the end effector 10.

[0048] In such control, if the command value gets inside the object, it may result in, for example, excessive force being generated or the gripping force being too strong, making it impossible to properly control the object. In this case, the grip becomes unstable. For this reason, in this embodiment, the center coordinate is used by utilizing the fingertip position that has entered the object. Then, in this embodiment, the command value is corrected by subtracting the component of the operator's finger movement that has entered the object from the movement of the center coordinate and the finger movement based on the virtual command value based on the center coordinate.

[0049] The joint angles of the operator's fingers are detected by the sensor 31 of the operation input unit 3 worn by the operator. The grip center position is the center (or center of gravity) of the fingertip positions where the fingers 11 of the end effector 10 actually contact the object. In contrast, the center coordinates are the center (or center of gravity) of the virtual fingertip positions based on command values ​​for movement based on the operator's joint angles. The posture of the object can also be defined and controlled by setting the x, y, and z axes that pass through these center coordinates.

[0050] The control unit 27 determines whether the normal direction component enters the object g27 based on the command value of the relative vector. This determination is made after the finger 11 comes into contact with the object.

[0051] [Constraint example] Next, an example of the constraints will be described with reference to Fig. 4. Image g30 is an image of, for example, manipulating a sphere g31 with two fingers (first finger and second finger) of a robot. Reference symbols g32 and g33 represent the fingertips of each finger. Arrows g36 and g37 represent command values ​​for the fingertips before correction. The example of image g30 shows a case where, while holding the sphere g31, it is desired to slide the fingertip g33 of the second finger on the sphere g31 (for example, a task of changing hands). In such a task, it is desired to cancel the normal direction component g34 of the tangential direction component g35 and normal direction component g34 of the command value g36 and control it using the tangential direction component g35.

[0052] Image g40 is an image of an operation in which, for example, a rectangular parallelepiped g41 is grasped with three fingers (first finger, second finger, and third finger) and then the third finger is moved to change the posture of the object. In image g40, for example, the object is lifted in the direction of arrow g45. Symbols g42, g43, and g44 represent the fingertips of each finger. The chain circle g46 represents the position after the fingertips have been moved. In this type of operation, unlike the operation in image g30, the operation is performed using normal direction components.

[0053] As shown in Figure 4, we cannot simply constrain the modal direction component. For this reason, in this embodiment, the operation content is determined based on the movement of the operator's fingertip in the central coordinate system, and any excess force that is applied in accordance with the operation is cancelled.

[0054] [Operation Procedure] Next, an example of the operation procedure will be explained using Fig. 5 to Fig. 7. Fig. 5 is a flowchart of the remote control process according to this embodiment. The operator controls the end effector 10 based on images captured by the environment sensor 4 or a camera attached to the robot until the finger 11 of the robot comes into contact with an object.

[0055] (Step S1) The acquisition unit 21 acquires the state of the operator detected by the sensor 31 of the operation input unit 3.

[0056] (Step S2) The acquisition unit 21 acquires the detection data detected by the environment sensor 4.

[0057] (Step S3) The acquisition unit 21 acquires the detection value of the force sensor 12 as a state quantity of the device (such as the joint angle of the hand). The acquisition unit 21 acquires the image (including depth information) captured by the environment sensor 4, and performs image processing on the information contained in the acquired image to acquire the state quantity of the object to be operated (such as the position and orientation of the object).

[0058] (Step S4) The control unit 27 brings the contact portion (for example, the fingertip or finger pad) of the end effector 10 into contact with the object based on the state of the operator, the state quantity of the operation target object, and the state quantity of the device.

[0059] (Step S5) At the start of grasping (time t=1), the center coordinate calculation unit 241 calculates center coordinates with the center coordinates of the contact point command value of the end effector 10 calculated from the acting force information as the origin.

[0060] (Step S6) At the start of grasping, the contact point position calculation unit 242 calculates the contact point position pi of the end effector 10 with the object in the central coordinate system, and stores the calculated fingertip position pi of the robot on the coordinate system of the central coordinates in the memory unit 26.

[0061] (Step S7) The control unit 27 determines whether or not the instruction from the operator (fingertip position, joint angle) has changed. If the instruction from the operator has changed (Step S7; YES), the control unit 27 proceeds to the processing of Step S8. If the instruction from the operator has not changed (Step S7; NO), the control unit 27 repeats the processing of Step S7.

[0062] (Step S8) After the change (time t=2), the center coordinate calculation unit 241 recalculates the center coordinates with the center coordinates of the contact point command value of the end effector 10 calculated from the acting force information as the origin.

[0063] (Step S9) The coordinate transformation unit 243 transforms the coordinates of the contact point position pi stored in the storage unit 26 based on the applied force information at the updated time in accordance with the change in the calculated center coordinates.

[0064] (Step S10) The coordinate conversion unit 243 calculates the displacement of the fingertip position.

[0065] (Step S11) The tangential component calculation unit 244 calculates the contact tangential component at the contact point of the difference between the displacement of the acting force information before and after the update and the displacement of pi due to the coordinate transformation.

[0066] (Step S12) The control unit 27 determines whether or not the contact tangent component is intruding into the object. If the contact tangent component is intruding into the object (step S12; YES), the control unit 27 proceeds to the processing of step S13. If the contact tangent component is not intruding into the object (step S12; NO), the control unit 27 proceeds to the processing of step S14.

[0067] (Step S13) The control unit 27 deletes the contact tangent component that penetrates into the object.

[0068] (Step S14) After the process of step S13, the control unit 27 generates a corrected command value by adding the contact tangential component to the coordinate-transformed contact point position pi. Alternatively, after step S12, the control unit 27 generates the command value without correcting it.

[0069] FIG. 6 is a diagram for explaining a first example of a process for generating a correction command value. Image g50 is an image of the force acting on an object based on the movement of the operator's fingertip remotely. In the example of image g50, the object is grasped and the finger is moved (slid) on the object as shown by arrow g51.

[0070] Image g60 is an illustration of the change in the center coordinate position when fingertips g62 and g63 of the end effector 10 are in contact with a spherical object g61. Arrows g64 and g65 are the same vectors as the arrow g51 in image g50 and represent tangential forces. The fingertip positions are p1 and p2. A dashed line g66 is a line connecting fingertip g62 and fingertip g63. Point g67 is the center coordinate before the change, for example, the midpoint of the dashed line g66. Arrow i represents the displacement of the fingertip for object manipulation (a tentative command value based on the operator's instructions). Point g68 is the center coordinate after the change.

[0071] Image g70 is an image for explaining the relative vectors of the fingers of the right hand and the removal of components undesirable for control. Point g71 is the fingertip position. Arrow ii (g76) is the displacement. Arrow g74 is the relative vector calculated by subtracting the displacement for object manipulation of each finger (arrow i in image g60) from the operator's finger displacement. Arrow iii (g77) is the component remaining after the component of the relative vector that is in the normal direction has been removed (arrow g73). The circle g72 indicates that the arrow g73 has been deleted, while the arrow g77 remains. The arrow g75 is the arrow to the right of g51 (= g65 = operator finger displacement).

[0072] Thereafter, the calculation unit 24 calculates the amount of manipulation of the object's position and orientation, as well as the amount of finger sliding on the object, from the movement of the operator's finger. The control unit 27 controls the robot to appropriately execute these operations based on the information calculated by the calculation unit 24.

[0073] FIG. 7 is a diagram for explaining a second example of the process for generating a correction command value. Image g80 is an image of the movement of the operator's fingertip remotely controlling the object and the change in the center coordinate. In the example of image g80, as shown by arrow g81, the image shows the operation of lifting an object by moving the ring finger upward. Symbol g82 indicates the x, y, and z axes in the center coordinate. Arrow g83 indicates the direction of rotation of the center coordinate when the fingertip is moved as shown by arrow g81.

[0074] Image g90 shows the contact relationship between an object and a finger in the robot's operational space, as well as the center coordinates. p1 is, for example, the position of the thumb tip, p2 is, for example, the position of the index finger tip, and p3 is, for example, the position of the ring finger tip. The image of image g90 is an image of a rectangular parallelepiped object g91 being pinched by its side between the thumb and index finger, supported by the ring finger on the bottom, and lifted as indicated by arrow i (displacement). Symbol g92 represents the x, y, and z axes in the center coordinates. Arrow g93 indicates the direction of rotation of the center coordinate when the fingertip is moved as in g81, and the displacement of the robot's fingertip for object manipulation based on this is indicated by arrow i. In this case, displacement i and the movement of the operator's finger are close to each other, so the relative vector ii is almost zero.

[0075] Image g100 shows the normal direction component in image g90. In the case of images g80 and g90, the normal direction component is incorporated into the object, but the displacement i (g101) resulting from coordinate transformation due to the change in center coordinates caused by arrow g81 is added to the component (almost 0) of relative vector ii that cancels the normal direction, leaving a result (g102) ≒ i (direction of movement in the normal direction).

[0076] [When the pilot's finger is far away from the object] When the control unit 27 determines that the operator's finger has been significantly removed from the object, it can remove the robot's finger, and the operator can rejoin the object manipulation by touching the finger again.

[0077] FIG. 8 is a diagram for explaining an operation when the operator's finger is far away from the object. Image g110 is an example of a case where the operator wants to remove the finger from the object. In such a case, for example, the operator moves only the finger that he or she wants to remove outward as indicated by the arrow g111. Image g120 shows an example of a state in which an object is being held between the thumb and index finger and moved to the left. Arrow g122 shows the movement of the ring finger. For example, when moving an object from right to left on the paper, the operator applies more force to the ring finger than to the thumb. As seen in images g110 and g120, when looking at only the thumb, the force applied to the thumb when wanting to remove a finger from an object and when wanting to move an object appears to be the same. For this reason, when judging by the thumb alone, in the case of image g120, it can be judged as an operation of removing a finger from an object.

[0078] FIG. 9 is a diagram for explaining how to distinguish between an operation of lifting a finger and an operation of moving an object. Image g130 shows the state of releasing the finger from the object. Arrows g131 and g132 are relative vectors of the operator's finger position as viewed from the coordinate-transformed fingertip position pi. Arrow g133 represents the movement of the center coordinate. Image g140 shows the state in which an object is being gripped and moved. Arrows g141 and g142 are relative vectors of the operator's finger position as viewed from the coordinate-transformed fingertip position pi. Arrow g143 represents the movement of the center coordinate.

[0079] Thus, when determining whether the operator wants to remove their finger from an object, comparing the operator's fingertip position with the robot's current fingertip position is difficult to handle due to tracking delays. Therefore, in this embodiment, as shown in Figure 9, the operator's finger position as viewed from the coordinate-transformed fingertip position pi and the normal direction are used to determine whether the operation is to remove the finger or to move the object. For example, in images g130 and g140, the relative vector and normal direction for each finger are used to determine whether the operator's finger is outside the object surface or inside the object.

[0080] FIG. 10 is a diagram for explaining a control method in the case where an object is grasped with three fingers and then one finger is released from the object. Image g150 is an example of grasping an object g151 (e.g., a sphere) with three fingers. Arrows g152 to g154 represent the forces acting on the object from each of the three fingers. In this case, the control unit 27 calculates the balance of forces based on the specified grasping strength (scalar value) and the detection information of the tactile sensor 13 on the hand, and controls the force of each finger. In this case, the positions of the three fingertips in contact are used to calculate the balance of forces and the center coordinates.

[0081] Image g160 is an example of holding an object g152 with two fingers and then releasing one finger. Arrows g162-g162 represent the forces acting on the object from each of the two fingers. Arrow g164 represents the direction and amount of movement of the finger. In this case, the calculation of the center coordinates uses the positions of the two fingertips in contact, excluding the force balance calculation and the releasing finger.

[0082] In such a case, in order to prevent the central coordinate position from being pulled in the direction of separation, the calculation unit 24 excludes the finger determined to be released from the calculation of the balance of the gripping force and the calculation of the central coordinate.The calculation unit 24 then processes the coordinate transformation of the fingertip position in the same manner (no deletion of the normal direction is done because separation is desired, and movement is performed based on the change in the central coordinate of the other fingers to maintain the balance of the force) until the contact between the object and the finger is lost.

[0083] In addition, the control unit 27 applies force again if the command value indicates that the finger has entered the object before being released. In order to determine whether the finger has entered the object, the control unit 27 calculates the fingertip position pi when considering the deletion of the normal direction that moves away from the object, in addition to the fingertip position pi. s The fingertip position pi s is a reference point, a virtual position, and a threshold position. s If the fingertip position pi is closer to the inside, the finger to be released is not judged. The reason for this is that if the fingertip position pi of the finger being released is used as is, the reference point for the penetration judgment will be away from the object surface.

[0084] In addition, the fingertip position pi s is a position that is a predetermined distance away from the fingertip position, for example, and is set near the object surface position. As a result, in this embodiment, the position of the command value of the operator's fingertip is set to the fingertip position pi s When the fingertip position pi sis used to determine whether the operator is trying to stop releasing the end effector 10 from the object before the release, and the fingertip position pi is used to generate a command value.

[0085] If there are two or fewer fingers exerting force, the center coordinates are calculated from the fingers in contact, a similar position correction is performed, and no gripping force is output. In other words, for example, if an object is being grasped with three fingers 11 and then two fingers 11 separate from the object, the object cannot be grasped, and no gripping force is calculated or output to the end effector 10. After this, the control unit 27 continues control while determining whether the remaining fingers 11 also separate from the object or whether an instruction is received for the separated finger 11 to contact the object again. In this way, even if only one finger 11 is in contact with the object, the control described above is performed by setting the center coordinates to, for example, the load center.

[0086] [Finger release determination process] 11 is a flowchart of the process of determining whether or not a finger has been released according to this embodiment. Note that the following process is performed after the robot's fingertip has come into contact with an object by remote control by the operator.

[0087] (Step S101) At the start of grasping (time t=1), the center coordinate calculation unit 241 calculates center coordinates with the center coordinates of the contact point command value of the end effector 10 calculated from the acting force information as the origin.

[0088] (Step S102) At the start of grasping, the contact point position calculation unit 242 calculates the contact point position pi of the end effector 10 with the object in the central coordinate system, and stores the calculated fingertip position pi of the robot on the coordinate system of the central coordinates in the memory unit 26.

[0089] (Step S103) The control unit 27 determines whether or not the fingertip position has been displaced. If the fingertip position has been displaced (Step S103; YES), the control unit 27 proceeds to the processing of Step S104. If the fingertip position has not been displaced (Step S104; NO), the control unit 27 repeats the processing of Step S103.

[0090] (Step S104) After the displacement (time t=2), the center coordinate calculation unit 241 calculates center coordinates with the center coordinates of the contact point command value of the end effector 10 calculated from the acting force information as the origin. Subsequently, the center coordinate calculation unit 241 obtains the amount of change in the center coordinates.

[0091] (Step S105) After the displacement (time t=2), the coordinate transformation unit 243 transforms the coordinates of the contact point position pi according to the change in the calculated center coordinates from the acting force information at the updated time.

[0092] (Step S106) The control unit 27 determines whether the operation is a finger lifting operation or an object moving operation based on the normal direction and the finger position of the operator as viewed from the fingertip position pi that has been coordinate-transformed based on the change in the center coordinates. If the operation is a finger lifting operation (step S106; finger lifting operation), the control unit 27 proceeds to processing of step S107. If the operation is an object moving operation (step S106; object moving operation), the control unit 27 proceeds to processing of step S108.

[0093] (Step S107) The control unit 27 excludes the finger determined to be released from the calculation of the balance of the gripping forces and the calculation of the center coordinate. Subsequently, the calculation unit 24 calculates the balance of the gripping forces and recalculates the center coordinate. After the processing, the control unit 27 proceeds to the processing of step S109.

[0094] (Step S108) The calculation unit 24 calculates the balance of the gripping forces and recalculates the center coordinates. After the process, the control unit 27 proceeds to the process of step S109.

[0095] (Step S109) The control unit 27 generates a command value based on the calculation result of step S107 or step S108.

[0096] The above processing makes it possible to appropriately determine whether the operator wants to remove the finger from the object or move the object while holding it. As a result, according to this embodiment, it is possible to appropriately generate command values ​​for the operation of removing the finger from the object and the operation of moving the object while holding it, so that each operation can be performed appropriately.

[0097] 12 is a flowchart of the process of re-contact determination according to this embodiment. Note that the following process is performed after the robot's fingertip touches an object by remote control by the operator, the center coordinates are calculated, and the fingertip position is saved.

[0098] (Step S201) The control unit 27 determines whether the operation is a finger lifting operation or an object moving operation based on the finger position and normal direction of the operator as viewed from the fingertip position pi coordinate-transformed based on the change in the center coordinates. If the operation is a finger lifting operation (step S201; finger lifting operation), the control unit 27 proceeds to processing of step S202. If the operation is an object moving operation (step S201; object moving operation), the control unit 27 proceeds to processing of step S208.

[0099] (Step S202) The contact point position calculation unit 242 calculates the fingertip position pi s Set.

[0100] (Step S203) The control unit 27 calculates the fingertip position pi and the fingertip position pi s or the difference between the fingertip position pi and the fingertip position pi s Based on the ratio of the command value to the object position, the control unit 27 determines whether the fingertip position based on the command value is inside the object. If the control unit 27 determines that the fingertip position based on the command value is inside the object (step S203; YES), the control unit 27 proceeds to the process of step S204. If the control unit 27 determines that the fingertip position based on the command value is not inside the object (step S203; NO), the control unit 27 proceeds to the process of step S207.

[0101] (Step S204) The control unit 27 calculates the fingertip position pi sThe finger for which this is set is excluded from the determination of whether it has been released from the object.

[0102] (Step S205) The control unit 27 determines whether or not the operation is to remove the finger from the object, for example, based on the displacement of the fingertip position. If the operation is to remove the finger from the object (step S205; YES), the control unit 27 proceeds to the process of step S207. If the operation is not to remove the finger from the object (step S205; NO), the control unit 27 proceeds to the process of step S208.

[0103] (Step S207) The control unit 27 excludes the finger determined to be released from the calculation of the balance of the gripping forces and the calculation of the center coordinate. Subsequently, the calculation unit 24 calculates the balance of the gripping forces and recalculates the center coordinate. After the processing, the control unit 27 proceeds to the processing of step S209.

[0104] (Step S208) The calculation unit 24 calculates the balance of the gripping forces and recalculates the center coordinates. After the process, the control unit 27 proceeds to the process of step S209. After the process, the control unit 27 proceeds to the process of step S109.

[0105] (Step S209) The control unit 27 generates a command value based on the calculation result of step S207 or step S208. Note that if the command value enters the object before the finger is released, the control unit 27 performs control so as to apply force to the object again.

[0106] By the above process, it is possible to appropriately determine whether or not the command value has entered the object before the finger is released.

[0107] The above-described processing and configuration make it possible for this embodiment to provide support for object manipulation without relying on object information, which is expected to improve the success rate. Furthermore, this embodiment can provide support when using a learning algorithm rather than a human operator, and is expected to improve learning efficiency.

[0108] In the above-described embodiment, processing procedure, and configuration, the operation input unit 3 is described as a data glove, but the present invention is not limited to this. Any sensor capable of detecting the position and movement of a finger may be used, and for example, a combination of an imaging device and a distance measuring device may be used.

[0109] In the above-described embodiment, processing procedure, and configuration, the operator may perform remote operation using both hands. Even in this case, the central coordinates may be set in the x, y, and z axis directions, for example, based on one of the fingers used for operation. In this case, the central coordinates may be the center or center of gravity of multiple points used for remote operation.

[0110] A program for implementing all or part of the functions of the operation control device 2 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the operation control device 2. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSI (Large Scale Integration) hardware (including circuitry) such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0111] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0112] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0113] 1...operation control system, 10...end effector, 2...operation control device, 3...operation input unit, 4...environment sensor, 5...image display unit, 11...finger unit, 12...force sensor, 13...tactile sensor, 14...drive unit, 15...control unit, 16...communication unit, 21...acquisition unit, 22...setting unit, 23...detection unit, 24...calculation unit, 25...communication unit, 26...storage unit, 27...control unit, 31...sensor, 32...output unit, 241...center coordinate calculation unit, 242...contact point position calculation unit, 243...coordinate conversion unit, 244...tangential component calculation unit

Claims

1. A method for controlling an operation of an end effector having a plurality of contact portions capable of manipulating an object, comprising: sending acting force information when an operator manipulates the object to the end effector; a calculation step of calculating a target acting force for gripping the object by the end effector from the acquired state of the operator and acting force information; a detection step of detecting a contact force at a contact point where the contact portion is in contact with the object; a control step of controlling the operation of the drive unit of the end effector so that the acting force acting on the object from each contact element matches a target acting force; The calculation step a central coordinate system is set whose origin is the central coordinate of the contact point command value of the end effector calculated from the acting force information; Calculating a contact point position of the end effector in the central coordinate system; converting the coordinates of the contact point position in accordance with the change in the central coordinate system calculated from the updated acting force information; a command value based on the coordinate-converted contact point position is used as a correction command value; Operation control method.

2. The calculation step calculating a contact tangential component at the contact point of a difference between the displacement of the acting force information and the displacement of the contact point position due to coordinate transformation; the contact tangential component is added to the coordinate-transformed contact point position and used as a correction command value; The operation control method according to claim 1 .

3. updating the contact point position using the contact tangential component, and then performing coordinate transformation on the contact point position in accordance with a change in the central coordinate system calculated from the next updated acting force information; The operation control method according to claim 2 .

4. The control step includes: After the end effector having a plurality of finger portions comes into contact with the object, it is determined whether the operator is giving an instruction to move away from the object or an instruction to move the object while holding it, and whether the operator's fingers are outside the surface of the object or inside the object, based on the relative vectors and normal directions of each of the plurality of finger portions in the central coordinate system. The operation control method according to any one of claims 1 to 3.

5. The control step includes: setting a tentative contact point position within a predetermined range on the surface of the object; determining whether the operator is issuing an instruction to move away from the object after the end effector has come into contact with the object, or whether the operator is attempting to cancel the end effector's release from the object before the end effector is released from the object, based on the virtual contact point position and a command value issued by the operator; The operation control method according to any one of claims 1 to 3.

6. The end effector has a plurality of fingers, The central coordinate system is based on one of the plurality of finger portions, and an axial direction from the reference finger portion passing through the central coordinate is set as a first axial direction, a second axial direction is set within a plane including the first axial direction, and a third axial direction is set with respect to the plane. The operation control method according to any one of claims 1 to 3.

7. the acting force information is a joint angle of a finger of the operator, the center coordinate system is a contact point command value of the end effector calculated by matching a joint angle of the end effector to a joint angle of the operator; The operation control method according to any one of claims 1 to 3.

8. An operation control device for an end effector having a plurality of contact portions capable of operating an object, an output unit that sends acting force information when an operator manipulates the object to the end effector; a calculation unit that calculates a target acting force for gripping the object by the end effector based on the acquired state of the operator and acting force information; a detection step of detecting a contact force at a contact point where the contact portion is in contact with the object; a control unit that controls the operation of the drive unit of the end effector so that the acting force acting on the object from each contact element matches a target acting force; The calculation unit a central coordinate system is set whose origin is the central coordinate of the contact point command value of the end effector calculated from the acting force information; Calculating a contact point position of the end effector in the central coordinate system; converting the coordinates of the contact point position in accordance with the change in the central coordinate system calculated from the updated acting force information; a command value based on the coordinate-converted contact point position is used as a correction command value; Operation control device.

9. A computer of an operation control device for an end effector having a plurality of contact portions capable of manipulating an object, sending, to the end effector, acting force information when an operator manipulates the object; calculating a target acting force for grasping the object by the end effector from the acquired state of the operator and acting force information; detecting a contact force at a contact point where the contact portion is in contact with the object; controlling the operation of the drive unit of the end effector so that the acting force acting on the object from each contact element matches a target acting force; In the calculation, a central coordinate system is set, the central coordinate of which is the contact point command value of the end effector calculated from the acting force information; calculating a contact point position of the end effector in the central coordinate system; converting the coordinates of the contact point position in accordance with the change in the central coordinate system calculated from the updated acting force information; a command value based on the coordinate-converted contact point position is used as a correction command value; program.

Images