Operation control system, operation control device, operation control method, and program

Abstract

To provide an operation control system, an operation control device, an operation control method, and a program capable of appropriately operating a robot.SOLUTION: An operation control system includes: an end effector operable with respect to a first object; an input unit inputted with an operation instruction of an operator with respect to the end effector; a first acquisition unit that acquires object information of the first object; a second acquisition unit that acquires object information of a second object operated by the operator; a third acquisition unit that acquires a contact position between the second object and the operator; a fourth acquisition unit that acquires a correlation between the first object and the second object; and a control unit that generates an operation instruction of the end effector from the contact position and the correlation.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an operation control system, an operation control device, an operation control method, and a program. [Background technology]

[0002] Remote control systems for remotely controlling robots are being developed. Such robots have hands with multiple fingers, and can grasp, for example, a target object in response to remote control. In such systems, contact information is used to control the robot when it comes into contact with the target object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-28392 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, there is a delay in the transmission of sensor information used to acquire contact information to the operator, which can lead to a mismatch between the visual and tactile information received by the operator, making it difficult to properly operate the robot, such as for delicate tasks.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an operation control system, an operation control device, an operation control method, and a program that can appropriately operate a robot. [Means for solving the problem]

[0006] (1) In order to achieve the above object, an operation control system according to one embodiment of the present invention is an operation control system including: an end effector operable with respect to a first object; an input unit to which an operator inputs operation instructions for the end effector; a first acquisition unit that acquires object information for the first object; a second acquisition unit that acquires object information for a second object operated by the operator; a third acquisition unit that acquires a contact position between the second object and the operator; a fourth acquisition unit that acquires a correlation between the first object and the second object; and a control unit that generates operation instructions for the end effector from the contact position and the correlation.

[0007] (2) In the operation control system according to one aspect of (1) above, the control unit may be configured to switch between a first mode in which the correlation between the first object and the second object is reflected in the operation input of the operator, and a second mode in which the operation input of the operator is converted into an output suitable for the correlation and the first object.

[0008] (3) In an embodiment of the operation control system according to (1) or (2) above, the second object may be a primitive object, an average three-dimensional object among the things that appear in the work envisioned as the task, an object that approximates the first object, a primitive shape that can encase the first object, and a six-axis inertial measurement unit.

[0009] (4) In an operation control system according to any one of the above (1) to (3), the second acquisition unit may include a palm camera that acquires an image of the shape of the operator's finger from the palm side of the finger, and a head camera that is attached to the operator's head. The third acquisition unit may use the image captured by the palm camera to estimate the contact position between the operator's finger and the second object, and use the image captured by the palm camera and the image captured by the head camera to track the position of the operator's fingertip that is not touching the second object.

[0010] (5) In an embodiment of the operation control system according to (4) above, the end effector includes a plurality of hand cameras positioned at positions corresponding to the positions of the palm cameras, finger units, six-axis sensors at the fingertips of the finger units, and pressure sensors that detect the force applied to the surface, and the control unit may estimate the contact point between the finger units and the first object using the detection values ​​detected by the six-axis sensors and the pressure sensors and the images captured by the hand cameras.

[0011] (6) In the operation control system according to any one of the above aspects (1) to (5), when the control unit detects that the finger unit of the end effector has moved away from the first object, the control unit may move the finger unit so that a fingertip representative point is aligned with a trajectory plane defined at a position a predetermined distance away from the first object.

[0012] (7) In order to achieve the above object, an operation control device according to one embodiment of the present invention is an operation control device that controls an operation control system having an end effector that can be operated on a first object, and is an operation control device that includes: an acquisition unit that acquires an operation instruction from an operator for the end effector; a first acquisition unit that acquires object information of the first object; a second acquisition unit that acquires object information of a second object operated by the operator; a third acquisition unit that acquires a contact position between the second object and the operator; a fourth acquisition unit that acquires a correlation between the first object and the second object; and a control unit that generates an operation instruction for the end effector from the contact position and the correlation.

[0013] (8) In order to achieve the above object, an operation control method according to one embodiment of the present invention is an operation control method for controlling an operation control system having an end effector operable with respect to a first object, wherein an acquisition unit acquires an operation instruction from an operator for the end effector, a first acquisition unit acquires object information of the first object, a second acquisition unit acquires object information of a second object operated by the operator, a third acquisition unit acquires a contact position between the second object and the operator, a fourth acquisition unit acquires a correlation between the first object and the second object, and a control unit generates an operation instruction for the end effector from the contact position and the correlation.

[0014] (9) In order to achieve the above object, a program according to one embodiment of the present invention is a program that causes a computer of an operation control device that controls an operation control system having an end effector operable with respect to a first object to acquire an operation instruction from an operator for the end effector, acquire object information of the first object, acquire object information of a second object operated by the operator, acquire a contact position between the second object and the operator, acquire a correlation between the first object and the second object, and generate an operation instruction for the end effector from the contact position and the correlation. [Effects of the Invention]

[0015] According to the above (1) to (9), the robot can be operated appropriately. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining an overview of remote control of a robot and an operating environment. [Figure 2] This is an example of a palm camera attached to the palm of the operator's hand. [Figure 3] 1 is a diagram illustrating an example of the configuration of an operation control system according to an embodiment. [Figure 4] FIG. 1 illustrates an example of a primitive object. [Figure 5] FIG. 10 is an image diagram of a separation primitive. [Figure 6] 10 is a flowchart of a processing procedure of an operation control system according to an embodiment. [Figure 7] This is an image of the orbital plane, region, etc. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0018] (Summary) When remotely operating a robot's hand (end effector), the operator is present in the remote operation space and is not holding the target object, while the robot is present in the operation space and is holding the target object, and remote operation commands are transmitted to the robot. This makes it difficult for the operator to understand the size of the target object, and the current situation is that they are forced to make control inputs that are like grasping a cloud.

[0019] To obtain contact information between a robot's hand and an object, the robot attaches multiple skin sensors or force sensors to the inside of the finger pads or palms, and transmits the values ​​detected by the sensors to the operator. However, the development and installation costs of such sensors are high. Furthermore, when transmitting sensor information to the operator, there is a delay in the transmission of the sensor information used to obtain contact information.

[0020] In contrast to this, in this embodiment, the manner in which the operator touches the robot is conveyed to the robot via a primitive object, thereby controlling the robot fingertips without the robot transmitting information about how it touched the object to the operator. Primitive objects will be described later.

[0021] [Remote control and operating environment overview] First, an overview of remote control and the operating environment will be explained. FIG. 1 is a diagram for explaining an overview of the remote operation of a robot and the operation environment. As shown in FIG. 1, in the remote operation space, an operator Us, for example, wears a display device 31 and a head camera 33 on his head and wears palm cameras 32 (32L, 32R) on his hands. An environmental sensor 9 may be installed in the robot working space. The environmental sensor 9 may be attached to the robot 2. The robot 2 also includes, for example, an operation control device 5, an arm 21, and an end effector 22 (a first end effector 22L, a second end effector 22R). In addition, hand cameras 26 (26L, 26R) are attached to the end effector 22 of the robot 2.

[0022] The target object obj (first object) is, for example, a plastic bottle. The operator Us remotely controls the robot 2 and manipulates the target object obj, for example, by grasping the primitive object 4 (second object) with the hand wearing the palm camera 32 through the display device 31.

[0023] [Palm Camera] FIG. 2 shows an example of a palm camera attached to the palm of an operator. In the example of FIG. 2, palm camera 32 is attached to the inside of the operator's palm. First palm camera 321 is placed, for example, near the proximal phalanx of the index finger (including the thenar eminence) and between the index finger and the thumb. Second palm camera 322 is placed, for example, near the proximal phalanx of the little finger (including the thenar eminence) and on the outer side, not the ring finger side. Third palm camera 323 is installed, for example, near the thenar eminence of the palm. Fourth palm camera 324 is placed, for example, near the proximal phalanx of the ring finger (including the thenar eminence). Fifth palm camera 325 is placed, for example, near the proximal phalanx of the middle finger (including the thenar eminence).

[0024] First palm camera 321 to third palm camera 323 are attached to mount 327 made of, for example, an elastic material. First palm camera 321 to third palm camera 323 capture images of the hand and other parts from precision grip to force grip. The fourth palm camera 324 and the fifth palm camera 325 are fixed to mounts 328 made of a material that is resistant to deformation so that the distance between the cameras does not change. The fourth palm camera 324 and the fifth palm camera 325 are RGB (red-green-blue) stereo cameras. The operation control device 5 estimates depth information using images captured by the stereo cameras.

[0025] In the case of the camera configuration and arrangement shown in Figure 2, it is preferable that the cameras required for photographing the fingers in palm camera 32 are first palm camera 321 to third palm camera 323. The reason is that first palm camera 321 can photograph the index finger and middle finger, while second palm camera 322 can photograph the ring finger and little finger. Although first palm camera 321 and second palm camera 322 can also photograph depending on the state of the thumb, the thumb has a wide range of motion. For this reason, it is preferable to provide third palm camera 323 in order to photograph the thumb when it is about to grasp or when it is actually grasped. However, depending on the size of the object and the work involved, it may be possible to capture the thumb with first palm camera 321 and second palm camera 322. In such cases, palm camera 32 does not need to include third palm camera 323. Furthermore, depending on the work involved, palm camera 32 may include one of first palm camera 321 to third palm camera 323, or one of the three may be used. The camera placement and mount 327 material described in Figure 2 are examples and are not limited to these. For the shooting range of palm camera 32 and examples of other camera placements, see, for example, Japanese Patent Application No. 2023-199232.

[0026] [Operation Control System] Fig. 3 is a diagram showing an example of the configuration of an operation control system according to this embodiment. As shown in Fig. 3, the operation control system 1 includes, for example, a robot 2, an HMI 3 (input unit), a primitive object 4, and an operation control device 5. The operation control system 1 may also include an environmental sensor 9.

[0027] The robot 2 includes, for example, an arm 21, an end effector 22, an actuator 23, a pressure sensor 24, an encoder 25, a hand camera 26, a six-axis sensor 27, and a communication unit 28. The robot 2 also includes, for example, a communication unit, a power supply unit, a body, legs, a head, etc.

[0028] The HMI 3 includes, for example, a display device 31, a palm camera 32, and a head camera 33. Each part of the HMI 3 includes, for example, a communication unit, a power supply unit, and the like.

[0029] The primitive object 4 includes, for example, a sensor 41 and a communication unit 42 .

[0030] The environmental sensor 9 includes, for example, a sensor 91 and a communication unit 92. The environmental sensor 9 also includes a power supply unit (not shown) and the like.

[0031] The operation control device 5 includes, for example, a first acquisition unit 51 (acquisition unit), a second acquisition unit 52, a third acquisition unit 53, a fourth acquisition unit 54, a hand posture estimation unit 55, an object posture estimation unit 56, a control unit 57, an output unit 58, a storage unit 59, and an image generation unit 60. The operation control device also includes a power supply unit and the like.

[0032] (robot) The robot 2 transmits and receives various information to and from the operation control device 5 via a wired or wireless network NW. The robot 2 may include two sets of an arm 21 and an end effector 22. The robot 2 operates in accordance with operation instructions generated by the operation control device 5 based on instructions entered by an operator. The robot 2 may also include a drive circuit for driving an actuator 23.

[0033] One end of the arm 21 is connected to, for example, the body via a joint, and the other end is connected via a joint to an end effector 22. The arm 21 has joints that correspond to, for example, a human elbow. An actuator 23 and an encoder 25 are attached to each joint.

[0034] The end effector 22 has a plurality of fingers. Each finger has a joint. An actuator 23 and an encoder 25 are attached to each joint. A six-axis sensor 27 and a pressure sensor 24 are attached to the fingertips. A hand camera 26 is attached to the end effector 22 at a position corresponding to the position of the palm camera 32 attached to the palm of the operator as described with reference to FIG.

[0035] The actuator 23 is attached to each joint of the arm 21 and the end effector 22. The actuator 23 operates the arm 21 and the end effector 22 in accordance with an operation instruction obtained from the operation control device 5.

[0036] The pressure sensor 24 is, for example, a skin pressure sensor. The pressure sensor 24 is attached to the surface of the finger pad of the fingertip of the finger portion of the end effector 22, for example.

[0037] The encoder 25 is attached to each joint of the arm 21 and the end effector 22. The encoder 25 detects the movement of each joint.

[0038] The hand camera 26 is attached to the end effector 22. The hand camera 26 takes pictures in response to instructions from the operation control device 5, for example, at predetermined time intervals.

[0039] The six-axis sensor 27 is attached to the fingertip of each finger of the end effector 22. The six-axis sensor 27 is a sensor that measures forces (Fx, Fy, Fz) in three directions of the X, Y, and Z axes, and torques (Mx, My, Mz) around each axis.

[0040] The communication unit 28 acquires operation instructions from the operation control device 5. The communication unit 28 outputs the detection values ​​detected by the pressure sensor 24, the encoder 25, and the six-axis sensor 27 to the operation control device 5. The communication unit 28 outputs image data captured by the hand camera 26 to the operation control device 5.

[0041] (environmental sensor) The environmental sensor 9 transmits and receives various information to and from the operation control device 5 via a wired or wireless network NW. The sensor 91 is, for example, an RGB-D camera, and acquires RGB (red, green, blue) information and depth information D. Note that the information is acquired, for example, at predetermined time intervals. The communication unit 92 outputs the detection value detected by the sensor 91 to the operation control device 5. The data output by the environmental sensor 9 includes identification information that allows the environmental sensor 9 to be identified.

[0042] (HMI) The HMI 3 is worn by the operator. The display device 31 is worn on the head of the operator. The display device 31 is, for example, a see-through goggle equipped with a display unit. The display device 31 displays an image acquired from the operation control device 5 on the display unit. The operator gives work instructions by operating primitive objects through the display device 31 while viewing the image displayed on the display. Palm camera 32 is attached to the operator's hand as shown in Fig. 2. Palm camera 32 takes pictures in response to instructions from operation control device 5, for example, at predetermined time intervals. The head camera 33 is attached to the head of the operator. The head camera 33 takes pictures in accordance with instructions from the operation control device 5, for example, at predetermined time intervals.

[0043] (Primitive Objects) The primitive object 4 is an average three-dimensional object that appears in the work assumed as a task. The primitive object 4 is an object that approximates the target object of the robot 2, and has a primitive shape that can enclose the target object. The sensor 41 is, for example, a six-axis IMU (Inertial Measurement Unit). The communication unit 42 outputs the detection value detected by the sensor 41 to the operation control device 5.

[0044] (Operation control device) The operation control device 5 transmits and receives various information to and from the robot 2 via a wired or wireless network NW. The operation control device 5 receives various information from the HMI 3, the primitive objects 4, and the environmental sensors 9 via the wired or wireless network NW.

[0045] The first acquisition unit 51 acquires first object information of the first target object. The first acquisition unit 51 also acquires an operation instruction to the end effector 22 from the operator.

[0046] The second acquisition unit 52 acquires second object information of a second target object operated by the operator.

[0047] The third acquisition unit 53 acquires the contact position between the second target object and the operator. The third acquisition unit 53 estimates the contact point and the surface normal from the intersection of the planes drawn on the surface of the point cloud, which is obtained from the posture of the manipulation object estimated by the object posture estimation unit 56 and the posture of the operator's hand estimated by the hand posture estimation unit 55. The third acquisition unit 53 may calculate the normal by searching for the representative point with the shortest distance between polygons representing the shape using a gradient method based on the results of estimating the hand shape and object shape. As described below, the third acquisition unit 53 includes a model configured, for example, by a network (see, for example, Japanese Patent Application No. 2023-199232). In this way, the third acquisition unit 53 estimates the contact position between the operator's finger and the primitive object 4 using the image captured by the palm camera 32. Furthermore, the third acquisition unit 53 uses the image captured by the palm camera 32 and the image captured by the head camera 33 to track the position of the operator's fingertip that is not touching the primitive object 4.

[0048] The fourth acquisition unit 54 acquires the correlation between the first target object and the second target object.

[0049] The hand posture estimation unit 55 uses image data acquired from the head camera 33 to estimate the distance from the hand and the labels of the hand parts. The hand posture estimation unit 55 then converts the estimation results into the initial shape and posture of the fingers, the labels of each finger, and a point cloud of the fingers. A point cloud is a collection of points in three-dimensional space, and each point has coordinates. The hand posture estimation unit 55 also analyzes the image data from the first palm camera 321 to the third palm camera 323 among the image data acquired from the palm camera 32 to estimate the posture of the operator's hand. The hand posture estimation unit 55 is, for example, a model configured as a network (see, for example, Japanese Patent Application No. 2023-199232).

[0050] The object posture estimation unit 56 estimates the distance from the manipulation object using image data acquired from the head camera 33. Then, the object posture estimation unit 56 converts the estimation result into the initial position of the manipulation object. The object posture estimation unit 56 also estimates the posture of the manipulation object using image data from the fourth palm camera 324 to the fifth palm camera 325 among the image data acquired from the palm camera 32. The object posture estimation unit 56 is, for example, a model configured by a network (see, for example, Japanese Patent Application No. 2023-199232).

[0051] The control unit 57 generates operation instructions for the end effector 22 based on the contact position and correlation. The control unit 57 predicts the posture change of the primitive object, for example, up to several hundred milliseconds in advance, predicts the area where the real object will not fall from the RoboHand, and guides (supports) the fingertips of the robot 2 that are not touching the real object on a virtual trajectory plane. For details about the trajectory plane, see, for example, Japanese Patent Application No. 2023-026162. Depending on the correlation between the target object and the primitive object 4, the control unit 57 switches between a first mode, which reflects the correlation in the operator's operation input, and a second mode, which converts the operator's operation input into an output appropriate for the correlation and the target object. The control unit 57 estimates the contact point between the finger of the robot 2 and the virtual object using the detection values ​​detected by the 6-axis sensor 27 and the pressure sensor 24 and the image captured by the hand camera 26. When the control unit 57 detects that the finger of the robot 2 has left the target object, it moves the finger so that the fingertip representative point is aligned with a virtual trajectory plane defined at a position where the finger is a predetermined distance away from the target object. The fingertip representative point is, for example, the center point of the finger pad.

[0052] The output unit 58 outputs the operation instruction generated by the control unit 57 to the robot 2.

[0053] The memory unit 59 pre-calculates and stores a database of wrist positions and contact positions at which the primitive object 4 can be grasped. The database may be stored on the cloud. The memory unit 59 stores, for example, programs used by each unit of the operation control device 5, mathematical formulas, thresholds (described later), identification information for the robot 2, identification information for the environmental sensor 9, identification information for the palm camera 32, and identification information for the head camera 33.

[0054] The image generating unit 60 generates a display image required for remote control to be displayed on the display device 31 using the acquired image data.

[0055] The configuration example shown in FIG. 3 is just an example, and the present invention is not limited to this.

[0056] [Primitive Objects] FIG. 4 is a diagram showing an example of a primitive object. The objects with the codes g101 and g102 are "cones." The object with the code g103 is "cylinders." The objects with the codes g104 and g105 are "wedges." The object with the code g106 is "spheres." The object with the code g107 is "torus." The objects with the codes g108 and g109 are "boxes." The objects with the codes g110 and g111 are "pyramids." The example shown in FIG. 4 is just an example, and the present invention is not limited to this.

[0057] FIG. 5 is a conceptual diagram of a separation primitive. The image g201 is an example of the object shown in FIG. The image with code g211 is a chair with a backrest. The image g212 is an example of decomposing the chair g211 into basic three-dimensional objects. In this example, the chair can be separated into four cylinders and two cubes (boxes).

[0058] In this manner, in this embodiment, the 3D shape of a complex object is approximately separated into primitive objects in advance. If the shape of the target object is complex, the primitive objects of the part of the target object to be operated are used. Note that, for example, the operation control device 5 stores the relationship between the object of the complex shape and the separated primitive objects, and selects and instructs the primitive objects. If the object to be operated has a simple shape, it is replaced with a single primitive object.

[0059] The manipulation control device 5 detects the direction of gravity and the motion vector of the object when the primitive object is being manipulated, using the detection values ​​of the sensor 41.

[0060] [Operator operation] As described above, the operator wears a palm camera 32 that can precisely capture the shapes of objects and their hands, and a head camera 33 that can capture images of the palm camera 32 from a first-person perspective. Note that the HMI 3 does not have a functional unit that reproduces tactile sensations, etc., based on force feedback from the robot 2.

[0061] Using image data from the palm camera 32 and head camera 33 worn by the operator, the operation control device 5 precisely tracks the movement of the primitive object 4 in the hand from the measurement and estimation results of the object's motion vector and gravity.The operation control device 5 then estimates the contact position of the fingers using the photographic data from the palm camera 32. The manipulation control device 5 calculates the orientation of the primitive object 4 in the wrist coordinate system of the human hand. The manipulation control device 5 also obtains the finger angles of the human hand using image data from the palm camera 32 and head camera 33. Furthermore, the operation control device 5 uses the head camera 33 and palm camera 32 to track the tips of the operator's fingers that are not touching the primitive object 4 (are away from the object).

[0062] [Robot side] As described above, the robot 2 has hand cameras 26 placed on the end effector 22 at positions corresponding to the positions of the palm cameras 32. Therefore, the number of hand cameras 26 is unlimited. Furthermore, the end effector 22 has six-axis sensors 27 on the fingertips, pressure sensors 24 on the surface, and encoders 25 at the joints.

[0063] With this configuration, the operation control device 5 estimates the six-dimensional pose of the real object using the images captured by the hand camera 26. For details on six-dimensional object pose estimation, see, for example, Reference 1. The operation control device 5 also estimates the contact point of the finger with the object using the detection values ​​of the fingertip 6-axis sensor 27 and pressure sensor 24, and the image captured by the hand camera 26. The actual object posture is calculated in the wrist coordinate system of the hand of the robot 2. Information on the finger angle of the hand of the robot 2 is also obtained from the encoder 25. When the finger of the robot 2 is released from the object, the finger of the robot 2 moves with the fingertip representative point aligned on a trajectory plane defined at a predetermined distance from the object.

[0064] Reference 1: Yuto Harada, Tatsuya Aoki, et al., "6D Pose Estimation of Objects from Multiple Viewpoints in Virtual Space," Proceedings of the 2021 National Conference of the Japanese Society for Artificial Intelligence, p.2J1GS8a04-2J1GS8a04, 2021

[0065] [Control conditions] Next, control conditions and processing examples will be described. FIG. 6 is a flowchart of the processing procedure of the operation control system according to this embodiment.

[0066] (Step S1) The operation control device 5 uses data acquired from the HMI 3 to determine whether or not the operator has touched the primitive object 4. If the operator has touched the primitive object 4 (step S1; YES), the operation control device 5 proceeds to the processing of step S2. If the operator has not touched the primitive object 4 (step S1; NO), the operation control device 5 repeats the processing of step S1. Note that the HMI 3 may detect whether or not the operator's finger has touched the primitive object 4.

[0067] (Step S2) The operation control device 5 starts to control the movement of the robot 2 to follow the operation input instruction.

[0068] (Step S3) Using the acquired data, the operation control device 5 controls the posture of the real object and the wrist of the hand of the robot 2 so that they approach the posture of the primitive object 4 and the wrist of the operator's hand.

[0069] (Step S4) Using the acquired data, the operation control device 5 implements a control plan to move the finger of the robot 2 closer to a target that is a human finger contact point on the primitive object 4 converted to match the scale of the real object. Furthermore, the operation control device 5 distributes force. The size ratio between the target object and the primitive object 4 (the correlation between the target object and the primitive object 4) and the like are stored in the storage unit 59. The fourth acquisition unit 54 may acquire the correlation between the target object and the primitive object 4, or may read it out from the storage unit 59.

[0070] (Step S5) The operation control device 5 uses the data acquired from the HMI 3 to determine whether or not the operator's finger has been removed from the primitive object 4. If the operator's finger has been removed from the primitive object 4 (Step S5; YES), the operation control device 5 proceeds to the process of Step S6. If the operator's finger has not been removed from the primitive object 4 (Step S5; NO), the operation control device 5 returns the process to Step S2. Note that the HMI 3 may detect whether or not the operator's finger has touched the primitive object 4.

[0071] (Step S6) The operation control device 5 generates an operation instruction to move the fingertip of the robot 2 to a trajectory plane that is a predetermined distance away from the surface of the real object. The robot 2 moves the fingertip in accordance with this operation instruction. At this time, the robotic finger does not need to assume the same posture as the polygon formed by the operator's fingertip contact point and the tip of the operator's fingertip that has left the object; the finger that has left the real object simply needs to remain in the position on the trajectory plane closest to the human fingertip polygon.

[0072] (Step S7) As shown in Fig. 8, the operation control device 5 predicts the posture change of the primitive object up to, for example, several hundred milliseconds in advance, predicts the area where the real object will not fall out of the RoboHand, and performs guidance control (support control) on the fingertip g301 of the robot 2 that is not touching the real object on the orbital plane g302. Fig. 7 is an image diagram of the orbital plane, areas, etc. Note that the line g303 is the surface of the target object, and the area g304 is the area where the object will not fall out of the RoboHand.

[0073] Note that the above processing does not perform retargeting to align the operator's fingertips with those of robot 2. Note that retargeting takes into consideration the difference between the positions of the operator's fingertips and those of robot 2, since the sizes of the operator's hand and the hand of robot 2 are different.

[0074] As described above, the operator operates a primitive placed in front of him / her through the display device 31, which is a see-through goggle. Note that the primitive object 4 can use primitives of a plurality of different shapes so as to be able to approximate the target object of the robot 2. The robot 2 is controlled by switching between two control modes: a mode in which the robot 2 pinpoints and touches the position on the primitive object 4 touched by the operator, and an area mode in which the robot can touch anywhere on the primitive area. The primitive object 4 contains a 6-axis sensor and can output orientation information. Where on the primitive object 4 the operator is touching is detected using an image captured by a palm camera 32 (see, for example, Japanese Patent Application No. 2023-199232). The gap between the shape of the object to be grasped by the robot 2 and the operator's primitive object 4 can be detected by differences in the object mesh, and parts with extreme shape differences can be compensated for by control on the robot 2 side, or the operator can be visually informed using a display device 31 that control is difficult.

[0075] As described above, in this embodiment, the posture of the primitive object 4 that can envelop the object to be manipulated is output as the recognition result. In this embodiment, the wrist positions and contact positions at which the primitive object 4 can be grasped, for example, are calculated in advance and stored in a database, and the arm (wrist posture) and hand (contact position) are guided and controlled based on these. In this embodiment, the operator is allowed to manipulate the primitive object 4 (which approximates the target object of the robot 2) during remote operation. In this embodiment, the deviation in fingertip position between the operator's manipulation of the primitive object 4 and the robot 2's manipulation is detected, and further support and control are provided for the movement of the robot 2's fingertip.

[0076] As a result, according to this embodiment, the operator can instantly perceive contact information of an object at hand, thereby reducing the sensor placement on the robot 2 side and the delay in information transmission between the robot 2 and the operator, and reducing development costs for information reproduction on the operator side.

[0077] In the above example, the task is, for example, grasping an object, but may be other tasks such as picking up an object.

[0078] A program for implementing all or part of the functions of the operation control device 5 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the operation control device 5. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by hardware (including circuitry) using LSI (Large Scale Integration) such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip), or may be realized by a combination of software and hardware.

[0079] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0080] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0081] 1...operation control system, 2...robot, 3...HMI, 4...primitive object, 5...operation control device, 9...environmental sensor, 21, 21L, 21R...arm, 22, 22L, 22R...end effector, 23...actuator, 24...pressure sensor, 25...encoder, 26...hand camera, 27...6-axis sensor, 28...communication unit, 31...display device, 32...palm camera, 321...first palm camera , 322...Second palm camera, 323...Third palm camera, 324...Fourth palm camera, 325...Fifth palm camera, 33...Head camera, 41...Sensor, 42...Communication unit, 91...Sensor, 92...Communication unit, 51...First acquisition unit, 52...Second acquisition unit, 53...Third acquisition unit, 54...Fourth acquisition unit, 55...Hand posture estimation unit, 56...Object posture estimation unit, 57...Control unit, 58...Output unit, 59...Memory unit, 60...Image generation unit

Claims

1. an end effector operable with respect to a first object; an input unit to which an operator inputs an operation instruction for the end effector; a first acquisition unit that acquires object information of the first object; a second acquisition unit that acquires object information of a second object operated by the operator; a third acquisition unit that acquires a contact position between the second object and the operator; a fourth acquisition unit that acquires a correlation between the first object and the second object; a control unit that generates an operation instruction for the end effector based on the contact position and the correlation; An operation control system comprising:

2. The control unit a first mode in which the correlation between the first object and the second object is reflected in an operation input by the operator to create the object; a second mode in which the operation input of the operator is converted into an output suitable for correlation and the first object; Switch and control The operation control system of claim 1 .

3. The second object is a primitive object, which is an average three-dimensional object among things that appear in a work assumed as a task, which is an object that approximates the first object, which has a primitive shape that can enclose the first object, and which is equipped with a six-axis inertial measurement unit; 3. The operation control system according to claim 1 or 2.

4. The second acquisition unit a palm camera that captures an image of the shape of the operator's finger from the pad side of the finger; a head camera attached to the head of the operator, The third acquisition unit using the image captured by the palm camera, estimating a contact position between the operator's finger and the second object; Using the image captured by the palm camera and the image captured by the head camera, the position of the fingertip of the operator who is not touching the second object is tracked.

3. The operation control system according to claim 1 or 2.

5. The end effector a plurality of hand cameras arranged at positions corresponding to the positions at which the palm camera is arranged; A finger portion and a six-axis sensor at the tip of the finger; a pressure sensor that detects a force applied to the surface; the control unit estimates a contact point between the finger portion and the first object using detection values ​​detected by the six-axis sensor and the pressure sensor and an image captured by the hand camera. The operation control system of claim 4.

6. The control unit When it is detected that a finger provided on the end effector has been separated from the first object, the finger is moved so that a fingertip representative point is aligned with a trajectory plane defined at a position where the finger is separated from the first object by a predetermined distance.

3. The operation control system according to claim 1 or 2.

7. A manipulation control device for controlling a manipulation control system having an end effector operable with respect to a first object, an acquisition unit that acquires an operation instruction from an operator for the end effector; a first acquisition unit that acquires object information of the first object; a second acquisition unit that acquires object information of a second object operated by the operator; a third acquisition unit that acquires a contact position between the second object and the operator; a fourth acquisition unit that acquires a correlation between the first object and the second object; a control unit that generates an operation instruction for the end effector based on the contact position and the correlation; An operation control device comprising:

8. 1. A manipulation control method for controlling a manipulation control system having an end effector operable with respect to a first object, comprising: an acquisition unit acquires an operation instruction from an operator for the end effector; a first acquisition unit acquires object information of the first object; a second acquisition unit acquires object information of a second object operated by the operator; a third acquisition unit acquires a contact position between the second object and the operator; a fourth acquisition unit acquires a correlation between the first object and the second object; a control unit that generates an operation instruction for the end effector based on the contact position and the correlation; Operation control method.

9. a computer of an operation control device that controls an operation control system having an end effector operable with respect to a first object; acquiring an operation instruction from an operator for the end effector; acquiring object information of the first object; acquiring object information of a second object operated by the operator; acquiring a contact position between the second object and the operator; obtaining a correlation between the first object and the second object; generating an operation instruction for the end effector based on the contact position and the correlation; program.

Images