Holding control system, holding control method, and program

Abstract

To provide a holding control system which makes it possible to grasp a relationship between an object and an end effector more accurately and thereby operate the object more accurately from a remote site through a robot, and to provide a holding control method and a program.SOLUTION: A holding control system includes: an end effector having a plurality of fingers which can hold an object; an input unit into which an operation command from an operator to the end effector is input; a first acquisition unit which acquires environmental information of the object; a second acquisition unit which acquires position information of the object on a position where the fingers contact when the object is held by the end effector; and a control unit which controls the second acquisition unit to acquire the position information including an area of the object held by the end effector based on the environmental information and the operation command.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a grip control system, a grip control method, and a program. [Background technology]

[0002] In recent years, expectations for remote-controlled robots have been rising due to the effects of labor shortages, etc. In relation to this, techniques for enabling an operator to remotely control a robot are known (see, for example, Patent Documents 1 to 4). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-065971 [Patent Document 2] JP 2011-93067 A [Patent Document 3] International Publication No. 2022 / 190537 [Patent Document 4] Patent Publication No. 2022-121820 [Patent Document 5] JP 2022-156794 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, a camera is mounted on the head of the robot, and an operator remotely controls the robot based on the image from the camera. However, in the conventional technology, there is a problem that it is difficult to grasp the relative relationship (position, distance, angle, etc.) between the object to be operated and the end effector (also called the robot hand) of the robot only with the camera mounted on the head of the robot. Although it has been considered to mount a camera at the wrist position of the robot and use the camera, the fingers of the end effector (especially the thumb) tend to get in the way. Furthermore, it has been considered to mount a camera on the palm of the robot and use the camera, but it is difficult to grasp the relationship in the height direction of the object (depth direction of the camera). Due to such problems, there are cases where the operator cannot fully grasp whether or not the robot's fingers are accurately aligned with the desired position.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a gripping control system, a gripping control method, and a program that can more accurately manipulate an object remotely via a robot by more accurately understanding the relationship between the object and the end effector. [Means for solving the problem]

[0006] The gripping control system, gripping control method, and program according to the present invention employ the following configuration. (1) A first aspect of the present invention is a grasping control system comprising: an end effector having a plurality of fingers capable of grasping an object; an input unit to which an operator inputs an operational command to the end effector; a first acquisition unit that acquires environmental information which is information indicating an environment in which the object exists; a second acquisition unit that acquires positional information including a position of the object that the fingers contact when the object is grasped by the end effector; and a control unit that controls the second acquisition unit to acquire the positional information including an area of ​​the object to be grasped by the end effector based on the environmental information acquired by the first acquisition unit and the operational command input to the input unit.

[0007] (2) A second aspect of the present invention is the first aspect, wherein the area of ​​the object is a graspable area, which is an area in which the forces applied to the object from each of the multiple fingers are balanced when the object is grasped by the end effector.

[0008] (3) A third aspect of the present invention is that, in the second aspect, the control unit controls the second acquisition unit to acquire the position information including the graspable area observed from a straight line that is perpendicular to a polygonal two-dimensional space connecting the fingertips of each of the multiple fingers and passes through a grasping center that is the center when the object is grasped by the end effector.

[0009] (4) A fourth aspect of the present invention is that in the first or second aspect, the first acquisition unit is a first camera that images the environment and acquires image data of the imaged environment as the environment information, and the second acquisition unit is a second camera that images the position of the object touched by the finger from a viewpoint different from that of the first camera and acquires image data of the imaged position as the position information.

[0010] (5) A fifth aspect of the present invention is a first or second aspect, wherein the first acquisition unit is a first camera that captures an image of the environment and acquires image data of the captured environment as the environmental information, and the second acquisition unit uses a machine learning model to generate image data that is generated when the object touched by the finger is virtually observed from a viewpoint different from that of the first camera, and acquires the generated image data as the position information.

[0011] (6) A sixth aspect of the present invention is a robotic device according to the first or second aspect, further comprising a second control unit that controls the end effector to grasp the object based on the position information including an area of ​​the object.

[0012] (7) A seventh aspect of the present invention is a grasping control method including: an operator inputting, into an input unit, an operation command for an end effector having a plurality of fingers capable of grasping an object; a first acquisition unit acquiring environmental information which is information indicating an environment in which an object that can be grasped by the end effector exists; a second acquisition unit acquiring positional information including a position of the object that the fingers contact when the object is grasped by the end effector; and a control unit controlling the second acquisition unit to acquire the positional information including an area of ​​the object to be grasped by the end effector based on the environmental information acquired by the first acquisition unit and the operation command input to the input unit.

[0013] (8) An eighth aspect of the present invention is a program to be executed on a computer having an input unit to which an operator inputs operational commands to an end effector having multiple fingers capable of grasping an object, the program including: acquiring environmental information that is information indicating the environment in which the object exists; acquiring positional information including a position of the object that the fingers contact when the object is grasped by the end effector; and acquiring the positional information including an area of ​​the object to be grasped by the end effector based on the acquired environmental information and the operational command input to the input unit. Effect of the Invention

[0014] According to the above aspect, by more accurately grasping the relationship between the object and the end effector, it is possible to more accurately manipulate the object remotely via the robot. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating an example of a usage scene of the gripping control system 1 according to the first embodiment. [Diagram 2] FIG. 1 is a functional configuration diagram of a gripping control system 1 according to a first embodiment. [Diagram 3] 2 is a sequence diagram showing a flow of a series of processes of the gripping control system 1 according to the first embodiment. FIG. [Figure 4] FIG. 13 is a diagram showing an example of a graspable area Ra. [Diagram 5] 13 is a diagram for explaining how the graspable area Ra is imaged by the external environment camera 50 before it is moved. FIG. [Figure 6] 13 is a diagram for explaining how the graspable area Ra is imaged by the external environment camera 50 after it has been moved. FIG. [Figure 7] 4 is a flowchart showing the flow of a series of processes of the HMI 10 according to the first embodiment. [Figure 8] 4 is a flowchart showing the flow of a series of processes of the control device 100 according to the first embodiment. [Figure 9] FIG. 13 is a diagram for explaining an example of details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 10] FIG. 13 is a diagram for explaining an example of details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 11] FIG. 13 is a diagram for explaining an example of details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 12] FIG. 13 is a diagram for explaining an example of details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 13] FIG. 13 is a diagram for explaining another example of the details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 14] FIG. 13 is a diagram for explaining another example of the details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. [Figure 15] FIG. 1 is a diagram for explaining an overview of NeRF. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, embodiments of a gripping control system, a gripping control method, and a program according to the present invention will be described with reference to the drawings.

[0017] First Embodiment [Scenes where the grip control system can be used] Fig. 1 is a diagram showing an example of a usage scene of the gripping control system 1 according to the first embodiment. As shown in Fig. 1, the gripping control system 1 includes, for example, a human machine interface 10 (hereinafter referred to as HMI 10) used by an operator OP, a robot 20 remotely controlled by the operator OP, an external environment camera 50 that observes the robot working while moving freely in the same environment (or the same space) as the robot 20, and a control device 100 for controlling the robot 20. These are connected via a network NW. The network NW includes a LAN (Local Area Network) and a WAN (Wide Area Network), etc.

[0018] The HMI 10 is, for example, an interface used to remotely control the robot 20, and includes a wearable display 11 (also called a wearable device) worn by an operator OP, a remote controller 12, and the like. The HMI 10 is an example of an "input unit."

[0019] The wearable display 11 is a display for displaying still images and moving images (video) captured by the camera 21 and the external environment camera 50 mounted on the robot 20 under remote control. For example, the wearable display 11 is goggles, a headset, a head-mounted display, smart glasses, etc. that are worn on the head. The wearable display 11 may be a device that uses technology such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and projection mapping.

[0020] The remote controller 12 is a controller for moving the end effectors 23 (right end effector 23R, left end effector 23L) corresponding to the hands of the robot 20, and the arms 22 (right arm 22R, left arm 22L) connecting the torso of the robot 20 and the end effectors 23. The remote controller 12 may be a controller on which switches, buttons, joysticks, etc. are arranged as shown in the figure, or may be a glove-type controller. For example, the operator OP operates the robot 20 remotely by operating the remote controller 12 with his / her hand while checking the image displayed on the wearable display 11.

[0021] The robot 20 is typically a humanoid robot capable of grasping or manipulating an object OB by means of the end effector 23, but is not limited to this and may be any type of robot as long as it can grasp or manipulate an object OB. For example, the robot 20 may be a quadrupedal animal-type robot, an industrial robot, a military robot, or any other type of robot.

[0022] The end effector 23 may be referred to as a robot hand. The end effector 23 may be provided with, for example, several fingers (a thumb F1, an index finger F2, a middle finger F3, and a ring finger F4, which will be described later) as grippers.

[0023] The robot 20 is equipped with, for example, a camera 21 for imaging the external environment seen by the robot 20, and a control device 100 for controlling the operation of the robot 20, and performs a desired task according to the actions determined by the control device 100.

[0024] The task may be, for example, gripping an object OB with the end effector 23, transferring the object OB to the other end effector 23, or moving the object OB. Note that the tasks are not limited to these, and any task may be set. In this embodiment, a task of "grasping an object OB" will be described as an example.

[0025] The camera 21 is installed in a part of the body of the robot 20 (typically, the head). The camera 21 may be, for example, a depth camera (3D camera). For example, the camera 21 captures an image of a scene in which the object OB is grasped or operated by the end effector 23, and transmits image data of the scene to the control device 100, or transmits the image data to the HMI 10 via the control device 100. For convenience, the image data captured by the camera 21 will be referred to as "head viewpoint image data" in the following description. The camera 21 is an example of a "first acquisition unit" or an "acquisition unit". The head viewpoint image data is an example of "environmental information".

[0026] The external environment camera 50 may be, for example, a camera having tires or caterpillars and capable of moving forward, backward, turning left and right. The external environment camera 50 may also be a camera capable of moving on rails laid in advance. The external environment camera 50 may be, for example, a depth camera (3D camera) like the camera 21. For example, the external environment camera 50 captures a scene in which the object OB is grasped or operated by the end effector 23 from a viewpoint different from that of the camera 21. The external environment camera 50 then transmits image data of the scene to the control device 100, or transmits the image data to the HMI 10 via the control device 100. Hereinafter, for convenience, the image data captured by the external environment camera 50 will be referred to as "external viewpoint image data" in the following description. The external environment camera 50 is an example of a "second acquisition unit". The external viewpoint image data is an example of "position information".

[0027] The control device 100 is typically mounted on the robot 20 and controls the robot 20 based on operation commands received from the HMI 10. The control device 100 may also be installed at a location far away from the robot 20 and control the robot 20 remotely via a network NW.

[0028] [Functional configuration of the grip control system] 2 is a functional configuration diagram of the gripping control system 1 according to the first embodiment. The HMI 10 may further include a stationary display 13, a processing unit 14, a storage unit 15, a communication interface 16, etc., in addition to the above-mentioned wearable display 11 and remote controller 12.

[0029] The stationary display 13 displays images generated by the processing unit 14, a GUI (Graphical User Interface) for receiving various input operations from the operator OP, etc. For example, the stationary display 13 is an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, etc.

[0030] The processing unit 14 displays images on the wearable display 11 or the stationary display 13, and transmits operation commands for the robot 20 (end effector 23) input by the operator OP to the remote controller 12 to the control device 100 via the communication interface 16.

[0031] The processing unit 14 performs various processes by executing programs or instructions stored in the storage unit 15 using a central processing unit (CPU) or a graphics processing unit (GPU). For example, the processing unit 14 may be realized by hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), or may be realized by a combination of software and hardware.

[0032] The storage unit 15 is realized by, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), etc. The storage unit 15 stores firmware, application programs, calculation results of the processing unit 14, etc.

[0033] The communication interface 16 communicates with the control device 100 via the network NW. The communication interface 16 includes, for example, a wireless communication module including a receiver and a transmitter, and a NIC.

[0034] The robot 20 further includes an actuator 24, a state sensor 25, and a processing unit 26 in addition to the above-mentioned camera 21, arm 22 (22R, 22L), and end effector 23 (23R, 23L).

[0035] The actuator 24 drives each part of the robot 20 (such as the arm 22 and each finger F of the end effector 23) under the control of the processing unit 26. The actuator 24 includes, for example, an electromagnetic motor, a gear, an artificial muscle, and the like.

[0036] The status sensor 25 is a sensor that detects the status (e.g., joint angle, angular velocity, torque, etc.) of the robot 20. The status sensor 25 includes, for example, a rotary encoder that detects the degree of rotation of the joints of the robot 20 and the end effector 23, a tension sensor that detects the tension of the wire for rotating the joint, a torque sensor that detects the torque applied to the joint shaft, an acceleration sensor or a gyro sensor that detects the posture of the robot 20, etc.

[0037] The processing unit 26 controls the actuator 24 based on a control command generated by the control device 100. The processing unit 26 may be realized by a CPU, a GPU, or the like. The processing unit 26 may also be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0038] The control device 100 includes, for example, a communication interface 110, a processing unit 120, and a storage unit 130.

[0039] The communication interface 110 communicates with an external device via a network NW, and with the robot 20 via a communication line such as a bus. The external device is, for example, the HMI 10 or the external environment camera 50. The communication interface 110 includes, for example, a wireless communication module including a receiver and a transmitter, a NIC, and the like.

[0040] The processing unit 120 includes, for example, an environmental polygon calculation unit 121, a gripping posture estimation unit 122, a contributing finger selection unit 123, a line of sight area calculation unit 124, a movement amount calculation unit 125, an image generation unit 126, and an output control unit 127. The output control unit 127 is an example of a "control unit" or a "second control unit." The line of sight area calculation unit 124 is an example of a "calculation unit."

[0041] The environment polygon calculation unit 121 acquires head viewpoint image data of the working environment (also called working space) of the robot 20 captured by the camera 21 of the robot 20, and acquires external viewpoint image data of the working environment of the robot 20 captured by the external environment camera 50, via the communication interface 110. The working environment of the robot 20 may include the object OB and the fingers of the end effector 23 that are trying to grasp the object OB. Furthermore, the environment polygon calculation unit 121 may acquire sensor data including a detection value detected by the state sensor 25 of the robot 20, via the communication interface 110.

[0042] Based on these image data and sensor data, the environmental polygon calculation unit 121 calculates polygons (or polygon meshes) that approximate the working environment of the robot 20 to a polygonal shape. The polygons may include information such as the shape, weight, and center of gravity position of the object OB.

[0043] The gripping posture estimation unit 122 estimates the posture of the end effector 23 when gripping the object OB, based on image data or sensor data. For example, when the sensor data includes angles of each of a plurality of joints provided in the end effector 23, the gripping posture estimation unit 122 may estimate the posture of the end effector 23 based on the angle of each joint.

[0044] The contributing finger selection unit 123 selects the finger that contributes most to grasping the object OB (hereinafter referred to as the contributing finger) from among the multiple fingers provided on the end effector 23, based on the head viewpoint image data acquired from the camera 21 of the robot 20 and the operation command for the robot 20 from the operator OP input to the HMI 10.

[0045] The line-of-sight area calculation unit 124 calculates an area on the surface of the object OB (the polygon surface of the object OB) that should be more focused and visible, based on the contributing fingers.

[0046] The movement amount calculation unit 125 calculates the amount of movement from the current position (current viewpoint) of the external environment camera 50 to a target position (another viewpoint) at which the entire area to be viewed can be observed, based on the area to be viewed.

[0047] The image generation unit 126 uses a machine learning model to generate or synthesize image data (external viewpoint image data) of the working environment of the robot 20 observed from an arbitrary viewpoint.

[0048] The output control unit 127 generates control commands for controlling the actuators 24 of the robot 20 based on the contributing fingers, operation commands, and various image data. Then, the output control unit 127 transmits the control commands to the robot 20 via the communication interface 110. When the processing unit 26 of the robot 20 receives the control commands from the control device 100, it controls the actuators 24 based on the control commands. In this way, the robot 20 executes a task such as grasping and manipulating an object OB.

[0049] Furthermore, the output control unit 127 may generate a movement command for moving the external environment camera 50 to another viewpoint different from the current viewpoint observing the working environment of the robot 20, based on the contributing fingers, the operation command, and various image data. In this case, the output control unit 127 transmits the movement command to the external environment camera 50 via the communication interface 110. Upon receiving the movement command from the control device 100, the external environment camera 50 moves to the other viewpoint in accordance with the control command, observes the working environment of the robot 20 from the other viewpoint, and generates external viewpoint image data.

[0050] Furthermore, in addition to or instead of moving the external environment camera 50 to a different viewpoint, the output control unit 127 may cause the image generation unit 126 to generate external viewpoint image data capturing the working environment of the robot 20 from a different viewpoint.

[0051] The above components of the processing unit 120 are realized, for example, by a CPU, a GPU, or the like executing a program or instructions stored in the storage unit 130. Some or all of these components may be realized by hardware such as an LSI, an ASIC, or an FPGA, or may be realized by a combination of software and hardware.

[0052] The storage unit 130 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, a RAM, etc. The storage unit 130 stores firmware, application programs, the results of calculations by the processing unit 120, and the like.

[0053] [Processing flow of the grip control system] Hereinafter, a series of process flows of the grip control system 1 according to the first embodiment will be described with reference to a sequence diagram. Fig. 3 is a sequence diagram showing a series of process flows of the grip control system 1 according to the first embodiment.

[0054] First, the operator OP wears the wearable display 11 of the HMI 10, and searches for an object OB to be grasped on the image (in the image space) while visually checking the image displayed on the wearable display 11 (step S100). This image is an image of the working environment as seen by the robot 20 (i.e., an image of the working environment of the robot 20 captured by the camera 21).

[0055] When the object OB to be grasped by the operator OP is discovered on the image, the HMI 10 transmits position data indicating the position in the image space of the object OB that the operator OP was looking at when discovering it to the control device 100 mounted on the robot 20 (step S102).

[0056] When the control device 100 mounted on the robot 20 receives the position data of the object OB from the HMI 10, it identifies the object OB to be grasped from among one or more objects existing in the working environment of the robot 20 (step S104).

[0057] Next, the operator OP remotely controls the arm 22 of the robot 20 using the remote controller 12 of the HMI 10 in order to move the end effector 23 of the robot 20 closer to the object OB (step S106).

[0058] When the arm 22 is operated by the remote controller 12, the HMI 10 transmits signal data indicating the operation content input to the remote controller 12, i.e., posture data indicating the posture (pose) of the arm 22 instructed by the operator OP via the remote controller 12, to the control device 100 (step S108).

[0059] When the control device 100 receives the posture data of the arm 22 from the HMI 10, it determines the posture of the target wrist (the joint between the arm 22 and the end effector 23) of the robot 20 (step S110). Furthermore, the control device 100 calculates a polygon that approximates the working environment of the robot 20 to a polygonal shape based on the head viewpoint image data representing the working environment of the robot 20 captured by the camera 21.

[0060] Next, the operator OP remotely controls the fingertips of the end effector 23 of the robot 20 using the remote controller 12 of the HMI 10 in order to cause the end effector 23 of the robot 20 to grip the object OB (step S112).

[0061] When the fingertip of the end effector 23 is operated by the remote controller 12, the HMI 10 transmits signal data indicating the operation content input to the remote controller 12, i.e., data regarding the finger F that the operator OP has been instructed to move (hereinafter referred to as gripping data), to the control device 100 (step S114).

[0062] The gripping data may include, for example, identification data that can identify the finger F that the operator OP has instructed to move among the multiple fingers F provided on the end effector 23, and the joint angles of that finger F.

[0063] When the control device 100 receives the grasping data from the HMI 10, it selects a contributing finger (the finger F that contributes most to grasping the object OB) from among the multiple fingers F provided on the end effector 23 based on the head viewpoint image data representing the working environment of the robot 20 captured by the camera 21 and the grasping data (step S116).

[0064] For example, the control device 100 selects, as a contributing finger, one or more of the following conditions being satisfied: Note that these conditions are merely examples.

[0065] The finger that the operator OP is trying to move (a finger that the operator OP can see and can also operate or control, but which the operator OP wants to see in more detail). A finger that the operator OP is trying to move and that the operator OP is looking at through the screen of the wearable display 11 (a finger that is visible to the operator OP and can also be operated or controlled, but which the operator OP would like to see in more detail). Fingers that are difficult to align in height when viewed from the operator OP (even if the operator OP can see them, they are difficult to operate or control, so the operator wants to see those fingers). A finger that is occluded when viewed from the operator's OP (it cannot be seen from the operator's OP in the first place, so we want to see through it). The operator OP has already caused the robot 20 to grasp the object OB, and is about to remove the finger and place it again. The finger with the most distributed force in the predicted pair of grasp points (the finger that will exert the most force in the grasp). A finger that is predicted to get caught on the surface of the object OB when the operator OP changes his / her fingertip position. The finger located furthest from the operator OP in the grasping taxonomy (fingers used for grasping). - Before grasping) The finger is moved with intent and the gap between the finger and the object OB is poorly visible from the camera 21. - Before grasping) The gap between the fingers and the object OB where a large force is expected to be used when grasping the object OB with intentionally moved fingers. - After grasping) When the fingers are moved intentionally, the gap between the fingers and the object OB can easily cause the posture of the object OB to become unstable.

[0066] The above-mentioned grasping taxonomy is a classification system of patterns of fingers F used when the end effector 23 grasps an object OB, or a database of the classification system. For example, the grasping taxonomy includes medium-wrap, power-sphere, tripod, palmar-pinch, prismatic-2fingers, prismatic-3fingers, and the like.

[0067] A medium-wrap is a taxonomy in which an object OB is a horizontal cylinder, and all fingers F are used to grasp the horizontal cylinder.

[0068] Power-sphere is a taxonomy in which an object OB is a sphere, and the sphere is grasped with all fingers F so that the force (contact force) applied to the sphere from each finger F is directed toward the center point of the sphere.

[0069] Tripod is a taxonomy in which an object OB is a vertical cylinder that is grasped with three fingers F.

[0070] A palmar pinch is a taxonomy in which an object OB is a rectangular parallelepiped, and the rectangular parallelepiped is grasped by pinching it between two fingers F.

[0071] Prismatic-2fingers is a taxonomy in which an object OB is a rectangular prism that is wider than that of a palmer pinch, and the ends of the rectangular prism are grasped by pinching them with three fingers F.

[0072] Prismatic-3fingers is a taxonomy in which an object OB is a rectangular parallelepiped that is wider than that of prismatic-2fingers, and the ends of the rectangular parallelepiped are grasped by pinching them with four fingers F.

[0073] In this way, for each grip taxonomy, there are fingers F that are used to grip an object OB and fingers F that are not used, so the control device 100 may select, as a contributing finger, a finger F that satisfies the above various conditions from among the fingers F that are determined in advance to be used by the grip taxonomy. Note that the grip taxonomy is not limited to the above examples, and finger F patterns may also be organized for an object OB having a shape such as a triangular prism or a cone.

[0074] Next, the control device 100 calculates an area on the surface of the object OB (the polygon surface of the object OB) that should be viewed in more focus, and further calculates the amount of movement of the external environment camera 50 to observe the entire area to be viewed (step S118).

[0075] The area to be viewed includes a graspable area Ra. The graspable area Ra is a partial area on the object OB that can be touched by each of the fingers F such that the forces applied to the object OB from each of the multiple fingers F, including the contributing fingers, are balanced when the fingers F are brought into contact with the object OB.

[0076] FIG. 4 is a diagram showing an example of the graspable area Ra. In the example of FIG. 4, in a state where the smartphone is grasped by the thumb F1, index finger F2, and middle finger F3 of the end effector 23 (i.e., prismatic two fingers), the grasping position of the thumb F1 is adjusted so that the object OB can be grasped stably while the forces applied from each finger F are balanced. An arrow g101 indicates a force acting on the surface of the object OB from the index finger F2. An arrow g102 indicates a force acting on the surface of the object OB by the middle finger F3. That is, the index finger F2 is in contact with the surface of the object OB indicated by the arrow g101, and the middle finger F3 is in contact with the surface of the object OB indicated by the arrow g102. A reference symbol g103 indicates a position (initial grasping position) where the thumb F1 first comes into contact.

[0077] Even if the finger cannot be stably grasped at the initial grasp position, the control device 100 can move the thumb F1 to a grasp position where the finger can be stably grasped by moving the thumb F1 on the graspable area Ra. The graspable area Ra may be calculated, for example, by a method described in Japanese Patent Application Laid-Open No. 2022-156794.

[0078] The area to be viewed may include a contributing finger contact area Rb in addition to or instead of the graspable area Ra.

[0079] The contributing finger contact area Rb is a partial area on the object OB that can be contacted by a contributing finger.

[0080] The amount of movement of the external environment camera 50 may include, for example, the distance from the current position (current viewpoint) of the external environment camera 50 to the target position (another viewpoint), the angle (elevation angle, azimuth angle) of the external environment camera 50 at the target position, the focal length, etc.

[0081] Returning to the explanation of the sequence diagram, the control device 100 then transmits a command based on the calculated amount of movement (hereinafter referred to as a movement command) to the external environment camera 50 (step S120). The movement command may include, for example, the coordinates of a target position (another viewpoint) at which the entire area to be viewed can be observed, and the amount of movement to the target position (another viewpoint).

[0082] Next, the external environment camera 50 moves to a target position (another viewpoint) in accordance with the movement command, and captures an image of an area to be viewed from the target position (step S122). As a result, external viewpoint image data is generated that represents the working environment of the robot 20 captured from the target position of the viewpoint different from the current viewpoint.

[0083] FIG. 5 is a diagram explaining how the graspable area Ra is imaged by the external environment camera 50 before it is moved, and FIG. 6 is a diagram explaining how the graspable area Ra is imaged by the external environment camera 50 after it is moved.

[0084] R50 in the figure represents the area captured by the external environment camera 50, that is, the FOV (Field of View). In the example of FIG. 5, the entire graspable area Ra, which is one of the areas to be viewed, is not included in the area R50. Therefore, the control device 100 calculates a target position (another viewpoint) such that the entire graspable area Ra is included in the area R50, which is the FOV of the external environment camera 50, and further calculates the amount of movement to the target position. Then, the control device 100 transmits a movement command including the calculation result to the external environment camera 50.

[0085] The external environment camera 50 moves to a target position in accordance with the movement command and captures an image of the object OB from the target position. As a result, as in the example of Fig. 6, external viewpoint image data is generated in which a graspable area Ra having a larger area than that in the example of Fig. 5 is captured.

[0086] Returning to the explanation of the sequence diagram, next, the external environment camera 50 transmits external viewpoint image data to the control device 100 (step S124).

[0087] Next, the control device 100 generates operator assistance information or an automatic control command based on the external viewpoint image data received from the external environment camera 50 (step S126).

[0088] The operator assistance information is information for assisting the operator OP in remote operation, and includes, for example, at least external viewpoint image data. More specifically, the operator assistance information may include an external viewpoint image overlapping a part of a head viewpoint image, an external viewpoint image and a head viewpoint image displayed in separate windows, or only an external viewpoint image. Furthermore, the operator assistance information may include, in addition to or instead of the external viewpoint image data, the coordinates of a target position included in a movement command and a movement amount to the target position.

[0089] The automatic control command is a command for the control device 100 to automatically control the robot 20 by itself without remote operation by the operator OP, and may include, for example, the next grasping taxonomy to be adopted, identification data of the finger F to be moved next, the joint angle of the finger F, etc.

[0090] For example, after generating the operator assistance information, the control device 100 transmits the operator assistance information to the HMI 10 (step S128).

[0091] On the other hand, when the control device 100 generates an automatic control command, it controls each finger F of the end effector 23 in accordance with the automatic control command.

[0092] When the HMI 10 receives the operator assistance information from the control device 100, the HMI 10 displays the operator assistance information on the wearable display 11 or the stationary display 13 (step S130).

[0093] In response to this, the operator OP, while visually checking the image (e.g., an external viewpoint image or a target position) displayed on the wearable display 11 or the stationary display 13, searches again for the object OB to be grasped on the image (in the image space), and repeats the series of processes using the remote controller 12 until the end effector 23 grasps the object OB.

[0094] [HMI processing flow] Hereinafter, the process flow of each of the HMI 10 and the control device 100 included in the gripping control system 1 will be described with reference to flowcharts. First, the process of the HMI 10 will be described, and then the process of the control device 100 will be described. Fig. 7 is a flowchart showing the process flow of a series of processes of the HMI 10 according to the first embodiment.

[0095] First, the processing unit 14 of the HMI 10 waits until the operator OP gazes at the object OB to be grasped on the image displayed on the wearable display 11 (step S200), and when the operator OP gazes at the object OB, it transmits position data indicating the position of the gazed-at object OB in the image space to the control device 100 via the communication interface 16 (step S202).

[0096] Next, the processing unit 14 waits until the operator OP operates the arm 22 using the remote controller 12 (step S204), and when the arm 22 is operated by the remote controller 12, transmits posture data indicating the posture (pose) of the arm 22 instructed by the operator OP via the remote controller 12 to the control device 100 via the communication interface 16 (step S206).

[0097] Next, the processing unit 14 waits until the operator OP operates the fingertip of the end effector 23 using the remote controller 12 (step S208), and when the fingertip of the end effector 23 is operated by the remote controller 12, it transmits gripping data (including identification data and joint angles of the finger F) related to the finger F that the operator OP has instructed to move to the control device 100 via the communication interface 16 (step S210).

[0098] In addition, the processing unit 14 may transmit to the control device 100, in addition to or instead of the finger F operated by the operator OP using the remote controller 12, gripping data regarding the finger F that the operator OP intends to operate, even if the operator OP is not operating it.

[0099] For example, the processing unit 14 may estimate the finger F that the operator OP intends to operate based on the above-mentioned gripping taxonomy, or may estimate the finger F that the operator OP intends to operate using a pre-trained machine learning model.

[0100] Next, the processing unit 14 determines whether the communication interface 16 has received operator assistance information from the control device 100 (step S212), and if the operator assistance information has been received by the communication interface 16, causes the wearable display 11 or the stationary display 13 to display an image based on the operator assistance information (step S214).

[0101] Next, the processing unit 14 determines whether or not the operator OP gazes again at the object OB (step S216).

[0102] When the operator OP gazes at the object OB again, the processing unit 14 returns the process to S208 and repeats the transmission of the grip data.

[0103] If the operator OP does not gaze at the object OB again and does not operate the end effector 23 again, the processing unit 14 ends the processing of this flowchart.

[0104] [Control device processing flow] FIG. 8 is a flowchart showing the flow of a series of processes of the control device 100 according to the first embodiment.

[0105] First, the environmental polygon calculation unit 121 of the processing unit 120 waits until the communication interface 110 receives position data of the object OB from the HMI 10 (step S300), and when the position data is received by the communication interface 110, it identifies the object OB to be grasped from among one or more objects present in the working environment of the robot 20 based on the position data (step S302).

[0106] Next, the gripping posture estimation unit 122 waits until the communication interface 110 receives posture data of the arm 22 from the HMI 10 (step S304), and when the posture data of the arm 22 is received by the communication interface 110, it determines the posture of the target wrist of the robot 20 (the joint between the arm 22 and the end effector 23) (step S306).

[0107] Next, the environment polygon calculation unit 121 calculates a polygon (or a polygon mesh) that approximates the working environment of the robot 20 to a polygonal shape, based on the head viewpoint image data generated by the camera 21 (step S308).

[0108] Next, the contributing finger selection unit 123 waits until the communication interface 110 receives the gripping data from the HMI 10 (step S310), and when the gripping data is received by the communication interface 110, selects a contributing finger from among the multiple fingers F provided on the end effector 23 based on the head viewpoint image data and the gripping data (step S312).

[0109] Next, the line-of-sight area calculation unit 124 calculates an area to be seen on the surface of the object OB (polygon surface of the object OB) based on the contributing fingers (step S314).

[0110] Next, the movement amount calculation unit 125 calculates the amount of movement from the current position (current viewpoint) of the external environment camera 50 to a target position (another viewpoint) at which the entire area to be viewed can be observed, based on the area to be viewed (step S316).

[0111] 9 to 12 are diagrams for explaining an example of the details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. In FIG. 9, GC represents the grip center, and R21 represents the area on the object OB captured by the camera 21, that is, the FOV of the camera 21. BA represents the blind spot range corresponding to the line segment between the grip center CC and the center of the fingertip. The center of the fingertip is the center of the polygon of each finger.

[0112] The grip center CC is a point on the polygon of the object OB where the extension lines of the fingertips of multiple fingers F that are determined to be used for gripping the object OB by the grip taxonomy intersect, and is the center of the surface formed by the fingertips of these fingers. For example, when the grip taxonomy is tripod, the grip center CC is the intersection point of the extension lines of the fingertips of the thumb F1, index finger F2, and middle finger F3.

[0113] 10, the movement amount calculation unit 125 first calculates a polygonal two-dimensional space (hereinafter referred to as a gripping plane GF) connecting the thumb F1 and the index finger F2, middle finger F3, and ring finger F4 opposed to the thumb F1. Then, the movement amount calculation unit 125 sets a target position on a line perpendicular to the gripping plane GF and passing through the gripping center CC, and calculates a movement amount for moving the external environment camera 50 on the line perpendicular to the gripping plane GF.

[0114] When the external environment camera 50 moves according to this target position and movement amount, the area that was not visible to the camera 21 due to the blind spot range BA can be made into an area R50 indicating the FOV of the external environment camera 50, as shown in Figures 11 and 12.

[0115] 13 and 14 are diagrams for explaining another example of the details of how the external environment camera 50 is moved to view the contributing finger contact area Rb. Rp in FIG. 13 represents an area that is not visible from the camera 21 due to the blind spot range BA, but can be predicted from the polygonal shape of the object OB. This area Rp may be calculated as an area that should be visible. In this case, as shown in FIG. 14, the area Rp that can be predicted from the polygonal shape can be further set as an area R50 indicating the FOV of the external environment camera 50.

[0116] More specifically, the area to be viewed may be determined according to the following conditions:

[0117] In a first mode in which the prerequisite that the external environment camera 50 is located closer to the object OB than the camera 21 is satisfied, while still improving the visibility of the area visible from the camera 21, (a) before the fingertip comes into contact with the object OB, (i) when the position (point of gaze) gazed upon by the operator OP is on the object OB, the focus is set so that the entire thumb F1 and the opposing fingers (which change to the index finger F2, middle finger F3, and ring finger F4 depending on the taxonomy) are displayed, or (ii) when the point of gaze moves onto one of the fingers F, or the focus is set (zoomed in) around the finger F that is predicted to be operated by the operator OP.

[0118] Under the above preconditions and further in the first mode, (b) after the fingertip has come into contact with the object OB, (i) if the operation is not an in-hand operation, zoom out so that the entire object OB and finger F are included, (ii) if the operation is not an in-hand operation and the object OB is brought closer to a specific location in the environment after being grasped, move the external environment camera 50 so as to view the center of the opposing axis between the environment and object OB, (iii) if the operation is an in-hand operation, if the point of gaze is on the object OB, focus on the entire object OB and finger F, or (iv) if the operation is an in-hand operation and the point of gaze moves onto finger F, or focus on the vicinity of finger F that is predicted to be operated by operator OP.

[0119] In a second mode in which it is desired to improve visibility of a blind spot area that is not visible to the camera 21, the self-occlusion occurs and focusing is performed in the same manner as above ((a) or (b)) from the side that is difficult for the camera 21 to see.

[0120] Returning to the explanation of the flowchart, the output control unit 127 then transmits a movement command based on the amount of movement to the external environment camera 50 via the communication interface 110 (step S318).

[0121] Next, the output control unit 127 waits until image data of the working environment of the robot 20 captured by the external environment camera 50 that has moved to the target position in accordance with the movement command, i.e., external viewpoint image data at the target position, is received by the communication interface 110 (step S320), and when the external viewpoint image data at the target position is received by the communication interface 110, operator assistance information or automatic control commands are generated based on this external viewpoint image data (step S322).

[0122] Next, if operator assistance information has been generated, the output control unit 127 transmits the operator assistance information to the HMI 10 via the communication interface 110, and if an automatic control command has been generated, the output control unit 127 transmits the automatic control command to the robot 20 via the communication interface 110 (step S324).

[0123] When the processing unit 26 of the robot 20 receives the automatic control command, it controls the actuator 24 based on the automatic control command, thereby causing the object OB to be grasped by the end effector 23. This ends the processing of this flowchart.

[0124] According to the first embodiment described above, the gripping control system 1 includes an end effector 23, an HMI 10 (an example of an “input unit”), a camera 21 (an example of a “first acquisition unit” or “acquisition unit”) mounted on the robot 10, an external environment camera 50 (an example of a “second acquisition unit”), and a control device 100.

[0125] The end effector 23 has a plurality of fingers F capable of gripping an object OB. An operation command from an operator OP to the end effector 23 is input to the HMI 10.

[0126] The camera 21 mounted on the robot 10 generates head viewpoint image data (an example of "environmental information"), which is image data obtained by capturing an image of the environment in which the object OB exists (the working environment of the robot 10).

[0127] The external environment camera 50 generates different viewpoint image data (an example of “position information”), which is image data obtained by capturing an image of the environment in which the object OB exists (the working environment of the robot 10) from a viewpoint different from that of the camera 21.

[0128] The processing unit 120 of the control device 100 selects, based on the head viewpoint image data and the operation command, from among the multiple fingers F of the end effector 23, a contributing finger that is a finger that contributes most to gripping the object OB.

[0129] The processing unit 120 of the control device 100 calculates a region to be more focused and viewed based on the contributing fingers. The region to be viewed is a region on the object OB that the contributing fingers may contact, and includes the graspable region Ra and the contributing finger contact region Rb.

[0130] The processing unit 120 of the control device 100 moves the external environment camera 50 so as to capture an image of the area to be viewed, thereby generating different viewpoint image data including the area to be viewed.

[0131] The processing unit 120 of the control device 100 generates operator assistance information based on different viewpoint image data including the area to be viewed captured by the external environment camera 50, and causes the HMI 10 to display this operator assistance information.

[0132] Through such processing, the operator OP can more accurately grasp the relationship between the object OB and the end effector 23. As a result, the operator OP can more accurately operate the object OB remotely via the robot 20.

[0133] In addition, the processing unit 120 of the control device 100 may generate an automatic control command based on another viewpoint image data including the area to be viewed captured by the external environment camera 50, and control the end effector 23 to grasp the object OB based on this automatic control command.

[0134] In this case, the control device 100 can remotely control the object OB via the robot 20 more accurately.

[0135] <Second embodiment> The second embodiment will be described below. In the above-mentioned first embodiment, the external environment camera 50 is moved to a target position (another viewpoint) where the entire area to be viewed can be observed, and the external environment camera 50 is controlled to capture an image of the object OB from the target position. In contrast, the second embodiment differs from the above-mentioned first embodiment in that a machine learning model is used to generate or synthesize image data (external viewpoint image data) of the working environment of the robot 20 virtually observed from the target position. The following description will focus on the differences from the first embodiment, and the description of the points in common with the first embodiment will be omitted. In the description of the second embodiment, the same parts as those in the first embodiment will be described with the same reference numerals.

[0136] The machine learning model used to generate or synthesize the external viewpoint image data may be implemented by a neural network including multiple intermediate layers. In this case, a program or algorithm that defines the machine learning model is stored in the storage unit 130. More specifically, as the program or algorithm that defines the machine learning model, the storage unit 130 stores various information such as coupling information on how units included in each of the multiple layers that make up the neural network are coupled to each other, and coupling coefficients that are assigned to data input and output between the coupled units.

[0137] The coupling information includes, for example, the number of units included in each layer, information specifying the type of unit to which each unit is coupled, activation functions that realize each unit, and gates provided between units in the hidden layer. The activation functions of the units may be, for example, a rectified linear function (ReLU function), a sigmoid function, a step function, a Softplus function, or other functions. The gates selectively pass or weight data transmitted between units according to, for example, a value (e.g., 1 or 0) returned by the activation function. The coupling coefficients include, for example, weights that are assigned to output data when data is output from a unit in a certain layer to a unit in a deeper layer in the intermediate layer of a neural network. The coupling coefficients may include bias components specific to each layer.

[0138] For example, the machine learning model is a model that has been trained in advance to output information about a three-dimensional structure that a human being can recognize when observing the target scene from a specified viewpoint and direction when an arbitrary viewpoint from which the target scene can be observed is specified and a line of sight direction when observing the scene from the specified viewpoint and direction. Such a machine learning model can be implemented, for example, by NeRF (Neural Radiance Fields).

[0139] NeRF is a method proposed primarily to achieve the task of generating an image from an arbitrary (new) viewpoint from images from multiple viewpoints (called novel view synthesis).

[0140] FIG. 15 is a diagram for explaining the outline of NeRF. When a position vector x(→)=(x,y,z) indicating the three-dimensional position of a certain viewpoint and a direction vector d(→)=(θ,φ) indicating the direction of the line of sight when observing a scene from that viewpoint are specified, NeRF determines N sampling points on the line of sight and outputs a color vector c(→)=(R,G,B) indicating the color of each sampling point and a density σ. An arrow in parentheses (→) represents a vector. For example, x may represent the width of the three-dimensional space, y may represent the depth, and z may represent the height. One of θ and φ may represent the elevation angle, and the other of θ and φ may represent the azimuth angle. R represents red, G represents green, and B represents blue. N is an arbitrary natural number.

[0141] For example, the image generation unit 126 inputs to NeRF a position vector x(→) indicating a target position at which the entire area to be viewed can be observed, a direction vector d(→) indicating the line of sight direction when observing the area to be viewed from the target position, and a posture vector l(→) of the joint of the end effector 23.

[0142] NeRF outputs a color vector c(→) and density σ in response to inputs x(→), d(→), and l(→).

[0143] The image generation unit 126 performs volume rendering of the color vector c(→) and density σ output by NeRF in the line of sight direction (directional vector d(→)) to generate external viewpoint image data of the scene that would be obtained when observed from the target position indicated by the position vector x(→).

[0144] The output control unit 127 generates operator support information or an automatic control command based on the external viewpoint image data generated by the image generation unit 126 .

[0145] According to the second embodiment described above, similarly to the first embodiment, the operator OP can more accurately grasp the relationship between the object OB and the end effector 23. As a result, the operator OP can more accurately operate the object OB remotely via the robot 20. Also, the control device 100 can more accurately operate the object OB remotely via the robot 20.

[0146] <Other embodiments> Other embodiments will be described below. In the above-mentioned embodiment, the HMI 10, the external environment camera 50, and the control device 100 included in the gripping control system 1 are described as separate devices independent of each other, but this is not limited to the above. For example, some or all of these devices may be integrated into one device. For example, the processing unit 14 of the HMI 10 may include an environment polygon calculation unit 121, a gripping posture estimation unit 122, a contributing finger selection unit 123, a line of sight area calculation unit 124, a movement amount calculation unit 125, an image generation unit 126, and an output control unit 127.

[0147] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0148] 1...Grasp control system, 10...HMI, 11...Wearable display, 12...Remote controller, 13...Fixed display, 14...Processing unit, 15...Memory unit, 16...Communication interface, 20...Robot, 21...Camera, 22...Arm, 23...End effector, 24...Actuator, 25...Status sensor, 26...Processing unit, 100...Control device, 110...Communication interface, 120...Processing unit, 130...Memory unit, 121...Environment polygon calculation unit, 122...Grip posture estimation unit, 123...Contributing finger selection unit 123, 124...Line of sight area calculation unit, 125...Movement amount calculation unit, 126...Image generation unit, 127...Output control unit

Claims

1. an end effector having a plurality of fingers capable of grasping an object; an input unit to which an operation command from an operator to the end effector is input; a first acquisition unit that acquires environmental information that is information indicating an environment in which the object exists; a second acquisition unit that acquires position information including a position of the object that is in contact with the finger when the object is grasped by the end effector; a control unit that controls the second acquisition unit to acquire the position information including an area of ​​the object gripped by the end effector, based on the environmental information acquired by the first acquisition unit and the operation command input to the input unit; and A gripping control system comprising:

2. The region of the object is a graspable region, which is a region in which forces applied to the object from each of the multiple fingers are balanced when the object is grasped by the end effector. The gripping control system of claim 1 .

3. the control unit controls the second acquisition unit to acquire the position information including the graspable area observed from a straight line that is perpendicular to a polygonal two-dimensional space connecting the fingertips of the plurality of fingers and passes through a grasp center that is a center when the object is grasped by the end effector. The gripping control system of claim 2 .

4. the first acquisition unit is a first camera that captures an image of the environment and acquires image data of the captured environment as the environment information; the second acquisition unit is a second camera that captures an image of a position of the object touched by the finger from a viewpoint different from that of the first camera and acquires image data of the captured position as the position information, The gripping control system according to claim 1 or 2.

5. the first acquisition unit is a first camera that captures an image of the environment and acquires image data of the captured environment as the environment information; The second acquisition unit generates image data generated when the object touched by the finger is virtually observed from a viewpoint different from that of the first camera using a machine learning model, and acquires the generated image data as the position information. The gripping control system according to claim 1 or 2.

6. a second controller configured to control the end effector to grasp the object based on the position information including a region of the object; The gripping control system according to claim 1 or 2.

7. An operator inputs, into an input unit, an operation command for an end effector having a plurality of fingers capable of grasping an object; a first acquisition unit acquiring environmental information which is information indicating an environment in which an object that can be grasped by the end effector exists; a second acquisition unit acquiring position information including a position of the object that is in contact with the finger when the object is grasped by the end effector; a control unit controls the second acquisition unit to acquire the position information including an area of ​​the object grasped by the end effector, based on the environmental information acquired by the first acquisition unit and the operation command input to the input unit; A gripping control method comprising:

8. A program to be executed by a computer having an input unit to which an operation command from an operator is input to an end effector having a plurality of fingers capable of grasping an object, acquiring environmental information that is information indicating an environment in which the object exists; acquiring position information including a position of the object contacted by the fingers when the object is grasped by the end effector; acquiring the position information including an area of ​​the object to be grasped by the end effector based on the acquired environmental information and the operation command input to the input unit; Programs including.

Images