Robot control system, robot control device, robot control method, and program

Abstract

This robot control system comprises: a robot having a plurality of joints; a sensor for acquiring a state quantity of the robot; a part designation unit for designating a robot part to be operated of the robot; a freedom degree determination unit for determining the degree of freedom to be used of the robot; and a control unit that receives inputs of the state quantity of the robot, the robot part, a target position of the robot part, and the degree of freedom to be used, and outputs a joint angle target for moving the robot part to the target position in a three-dimensional space.

Description

Robot control system, robot control device, robot control method, and program

[0001] The present invention relates to a robot control system, a robot control device, a robot control method, and a program. This application claims priority to Japanese Patent Application No. 2024-042665, filed March 18, 2024, the contents of which are incorporated herein by reference.

[0002] For example, methods for operating a robot equipped with a manipulator have been proposed. In all such operations, the position and orientation of a specific part (end effector) on the manipulator in three-dimensional space and the force applied from that part to an object are controlled. In a typical manipulator, the end effector is usually located near the tip of the manipulator (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-20259

[0004] In real-world situations, people manipulate objects using various parts of their body, such as their wrists or elbows. For this reason, controlling an end effector that is fixed to a specific part, as in conventional technology, is not always appropriate. This makes it difficult for conventional technology to achieve operations appropriate for a given task or scene.

[0005] The aspects of the present invention have been made in consideration of the above-mentioned problems, and aim to provide a robot control system, a robot control device, a robot control method, and a program that enable operations appropriate for tasks and scenes.

[0006] In order to solve the above problems, the present invention employs the following aspects: (1) A robot control system according to one aspect of the present invention is a robot control system including a robot having a plurality of joints, a sensor that acquires state quantities of the robot, a part designation unit that designates a part of the robot to be operated, a degree-of-freedom determination unit that determines degrees of freedom of the robot to be used, and a control unit that receives as input the state quantities of the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part of the robot to a target position in three-dimensional space.

[0007] (2) In the above aspect (1), the control unit may calculate the joint angle target by inverse kinematics, use mathematical optimization to solve the inverse kinematics, add a cost related to the target position of the part on the robot to an objective function, and change the weight of the cost related to the corresponding joint angle depending on the degree of freedom used.

[0008] (3) In the above aspect (1) or (2), the part designation unit may determine the parts on the robot using information necessary for determining the parts on the robot, and the information necessary for determining the parts on the robot may be at least one of objects around the robot, work content, objects around each of the candidate parts, and position information or posture information of each of the candidate parts.

[0009] (4) In any one of the above aspects (1) to (3), the degree of freedom determination unit may determine the available degrees of freedom using information necessary for determining the available degrees of freedom, and the information necessary for determining the available degrees of freedom may be at least one of the degrees of freedom of each candidate part, objects around the robot, work content, objects around each of the candidate parts, and position information or posture information of each of the candidate parts.

[0010] (5) In any one of the above aspects (1) to (4), the part on the robot may be at least one of the fingertips, finger pads, palm, wrist, and part on the arm of a hand provided on the robot.

[0011] (6) A robot control device according to one aspect of the present invention includes a part designation unit that designates a part of a robot having a plurality of joints to be operated, a degree of freedom determination unit that determines the degrees of freedom to be used of the robot, and a control unit that receives as inputs a state quantity of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs a joint angle target that moves the part of the robot to a target position in three-dimensional space.

[0012] (7) A robot control method according to one aspect of the present invention is a robot control method in which a part designation unit designates a part of a robot to be operated, the robot having a plurality of joints, a degree of freedom determination unit determines the degrees of freedom to be used of the robot, and a control unit receives as inputs a state quantity of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part of the robot to a target position in three-dimensional space.

[0013] (8) A program according to one aspect of the present invention causes a computer of a robot control device to specify a part of a robot having a plurality of joints to be operated, causes a degree of freedom determination unit to determine the degrees of freedom to be used of the robot, and causes a control unit to input the state quantities of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part of the robot to a target position in three-dimensional space.

[0014] According to the aspects of the present invention, it is possible to perform operations suited to tasks and scenes.

[0015] FIG. 1 is a diagram for explaining an example of a robot workspace. FIG. 2 is a diagram showing an example of the configuration of a robot control system according to a first embodiment. FIG. 3 is a diagram showing examples of parts in the first embodiment and examples of behavior when parts are changed. FIG. 4 is a diagram showing an example of a process for generating joint angle target values ​​according to the first embodiment. FIG. 5 is a flowchart of a process performed by a robot control device according to the first embodiment. FIG. 6 is a diagram showing an example of changing the degree of freedom according to a contact point, as viewed from the front of the robot. FIG. 7 is a diagram for explaining an overview of remote control of a robot. FIG. 8 is a diagram showing an example of the configuration of a robot control system according to a second embodiment.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component has been appropriately changed so that each component can be recognized. In all drawings used to explain the embodiments, components having the same function are designated by the same reference numerals, and repeated explanations will be omitted. In addition, "based on XX" in this application means "based on at least XX" and includes cases where the component is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where the component is based on XX after calculation or processing. "XX" is any element (for example, any information).

[0017] [Overview] In this embodiment, for example, in a robot with a hand equipped with multiple fingers, the tip or vicinity of the finger is not fixed as an end effector as in conventional technology, but the end effector is switched depending on the operation content and the scene. Thus, in this embodiment, an "end effector" is a part that can be used in a task. Furthermore, the usable part is, for example, at least one of the fingertips, finger pads, palm, wrist, and part on the arm of the end effector (hand) equipped in the robot.

[0018] First Embodiment In this embodiment, an example will be described in which an operator controls a robot by inputting work instructions, etc. FIG. 1 is a diagram illustrating an example of a robot workspace. As shown in FIG. 1, an environmental sensor 3 and a robot control device 6 are installed in the robot workspace. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 also includes a robot control device 6 and a manipulator 21 (a first manipulator 21L and a second manipulator 21R). The manipulator 21 includes, for example, hands 211 (211L, 211R). The operator operates an operation unit 69 ( FIG. 2 ) provided in the robot control device 6 to cause the robot 2 to manipulate a target object obj. The robot 2 performs work in the robot workspace under the control of the robot control device 6. Note that the robot 2 does not have to be a humanoid robot and may have a single arm.

[0019] [Configuration of Robot Control System] Next, a configuration example of the robot control system 1 will be described. Fig. 2 is a diagram showing a configuration example of the robot control system according to this embodiment. As shown in Fig. 2, the robot control system 1 includes, for example, a robot 2, an environmental sensor 3, and a robot control device 6.

[0020] The robot 2 includes, for example, a first manipulator 21L, a second manipulator 21R, a body 22, and a communication unit 23. The first manipulator 21L includes, for example, a hand 211L, an arm 212L, an actuator 213L, and a sensor 214L. The second manipulator 21R includes, for example, a hand 211R, an arm 212R, an actuator 213R, and a sensor 214R. The robot 2 may also include a power supply unit, legs, a head, etc., which are not shown.

[0021] In the following description, when there is no need to distinguish between the first manipulator 21L and the second manipulator 21R, they will also be referred to as manipulators 21. Similarly, when there is no need to distinguish between the hands 211L and 211R, they will also be referred to as hands 211, and when there is no need to distinguish between the arms 212L and 212R, they will also be referred to as arms 212. When there is no need to distinguish between the actuators 213L and 213R, they will also be referred to as actuators 213, and when there is no need to distinguish between the sensors 214L and 214R, they will also be referred to as sensors 214. The robot 2 transmits and receives various information to and from the robot control device 6 via a wired or wireless network NW.

[0022] The environmental sensor 3 includes, for example, a sensor 31 and a communication unit 32. The environmental sensor 3 also includes a power supply unit (not shown) and the like. The environmental sensor 3 transmits and receives various information to and from the robot control device 6 via a wired or wireless network NW.

[0023] The robot control device 6 includes, for example, an acquisition unit 61, a part designation unit 62, a degree of freedom determination unit 63, a control unit 64, a drive circuit 65, an output unit 67, a storage unit 68, and an operation unit 69. The robot control device 6 also includes a power supply unit (not shown). The robot control device 6 transmits and receives various types of information to and from the robot 2 and HDM 4 via a wired or wireless network NW. The robot control device 6 receives various types of information from the environmental sensor 3 via the wired or wireless network NW.

[0024] The configuration example shown in FIG. 2 is just an example, and the present invention is not limited to this.

[0025] [Functions of Each Device in the Robot Control System] Next, the functions of each device in the robot control system 1 will be described with reference to FIG. 2. (Robot 2) The hand 211 includes, for example, multiple fingers. Each finger includes a joint. The hand 211 may be a gripper or the like. One end of the arm 212 is attached to the hand 211 via a joint, and the other end is attached to the body 22 via a joint. An actuator 213 is attached to each joint. The sensor 214 is, for example, a six-axis sensor attached to the joint, a tactile sensor attached to the finger, or a force sensor. The six-axis sensor detects forces along three axes (x, y, z) and moments along three axes (α, β, γ).

[0026] The communication unit 23 transmits the detection value detected by the sensor 214 to the robot control device 6. The communication unit 23 receives the control signal or control instruction output by the robot control device 6. The data output by the robot 2 includes identification information that can identify the robot 2. The data acquired by the robot 2 includes identification information that can identify that the data is addressed to the robot 2. The robot 2 may include a drive circuit that drives the actuator 213.

[0027] (Environmental Sensor 3) The environmental sensor 3 is installed, for example, in the robot workspace, as shown in Fig. 1. The sensor 31 is, for example, an RGB-D camera, and acquires RGB (red, green, blue) information and depth information. The information is acquired, for example, at predetermined time intervals. The communication unit 32 outputs the detection value detected by the sensor 31 to the robot control device 6. The data output by the environmental sensor 3 includes identification information that allows the environmental sensor 3 to be identified.

[0028] (Robot control device 6) The acquisition unit 61 acquires a first sensor value detected by the sensor 214 from the robot 2. The acquisition unit 61 acquires a first sensor value detected by the environment sensor 3. The acquisition unit 61 acquires an operation result of the operator detected by the operation unit 69.

[0029] The part designation unit 62 designates the parts and number of parts on the robot 2 to be operated. The part designation unit 62 may designate the parts by acquiring information indicating the parts input by the operator operating the operation unit 69. Alternatively, the part designation unit 62 may designate the parts by estimating the relationship between the operator's operation intention (see, for example, Japanese Patent Application No. 2022-006498) and the operation target object based on information acquired from the environment sensor 3. The number of parts to be set is not limited to one, and may be multiple (for example, the thumb and index finger).

[0030] The degree of freedom determination unit 63 determines the degrees of freedom to be used for each designated part, for example, depending on the task and the environment. Note that the degree of freedom determination unit 63 may determine the degrees of freedom based on, for example, the detection results of the environmental sensor 3 or the estimated operation content, or may determine the degrees of freedom based on the degrees of freedom input by the operator operating the operation unit 69. For example, when the arm 212 of the robot 2 performs a task while being in contact with a wall, the contact with the wall may be detected based on the detection results of the force sensor or the environmental sensor 3 provided in the robot 2, or may be input by the operator operating the operation unit 69.

[0031] The control unit 64 receives as input the robot state quantities, the part on the robot 2 to be operated, the target position of the part on the robot 2 to be operated, and the degrees of freedom to be used, and calculates the target joint angles of the robot 2 to move the part on the robot 2 to the target position (target position in three-dimensional space). The control unit 64 generates a robot control command including the target joint angles, and outputs the generated robot control command to the drive circuit 65.

[0032] The drive circuit 65 generates a drive signal for controlling the robot 2 based on the robot control command generated by the control unit 64. Note that if the robot 2 is equipped with the drive circuit 65, the robot control device 6 does not need to be equipped with the drive circuit 65. Alternatively, the robot control device 6 and the robot 2 may each be equipped with a part of the drive circuit 65.

[0033] The output unit 67 outputs the drive signal output by the drive circuit 65 or the operation command generated by the control unit 64 to the robot 2 .

[0034] The storage unit 68 stores, for example, programs, mathematical formulas, thresholds, identification information of the robot 2, identification information of the environmental sensor 3, etc. used by each part of the robot control device 6.

[0035] [Examples of parts and examples of behavior when parts are changed] Next, examples of parts and examples of behavior when parts are changed will be described. Fig. 3 is a diagram showing examples of parts in this embodiment and examples of behavior when parts are changed. In Fig. 3, reference symbol g11 is the tip of the hand, reference symbol g12 is the hand 211, reference symbol g13 is the wrist, and reference symbol g14 is the arm 212.

[0036] Reference symbol g20 indicates an example of behavior when the tip g11 of the index finger is set as the part. In this case, the robot control device 6 issues a joint angle command centered on the tip g11 of the index finger, as shown by a circle g21. In this case, since the specified part is the fingertip, control is performed using the degrees of freedom of each of the arm and finger, for example.

[0037] Reference symbol g30 indicates an example of behavior when the wrist g13 is set as the part. In this case, the robot control device 6 issues a joint angle command centered on the wrist g13, as shown by a circle g31. In this case, since the specified part is the wrist, control is performed using the respective degrees of freedom of the arm and hand, for example, to achieve this.

[0038] [Joint angle target value] Next, an example of a method for generating a joint angle target value performed by the part designation unit 62, degree of freedom determination unit 63, and control unit 64 will be described. Fig. 4 is a diagram showing an example of a process for generating a joint angle target value according to this embodiment. The control unit 64 includes, for example, an inverse kinematics calculation unit 642. A determiner 641 corresponds to the part designation unit 62 and degree of freedom determination unit 63.

[0039] The decision unit 641 is, for example, a trained model. The first decision unit 641-1 corresponds to the part designation unit 62, and the second decision unit 641-2 corresponds to the degree of freedom determination unit 63. Note that the decision unit 641 outputs the end effector command value as it is, since the end effector command value is a command value indicating where to move the designated end effector part.

[0040] During learning, the first determinator 641-1 receives "information necessary to determine the location of the end effector (location on the robot)" and teacher data that is the correct output, and outputs "the location of the end effector." During use, the first determinator 641-1 receives "information necessary to determine the location of the end effector (location on the robot)" and outputs "the location of the end effector." Note that the "information necessary to determine the location of the end effector" is at least one of the following: objects in the vicinity of the robot, the work content, objects in the vicinity of each of the candidate locations, and position information or posture information of each of the candidate locations.

[0041] During learning, the second determinator 641-2 receives input of "information necessary to determine usable degrees of freedom" and teacher data that is the correct output, and outputs "usable degrees of freedom." During use, the second determinator 641-2 receives input of "information necessary to determine usable degrees of freedom" and outputs "usable degrees of freedom." Furthermore, the "information necessary to determine usable degrees of freedom" is, for example, determined by the second determinator 641-2 using the detection results of sensors (force sensors, torque sensors) provided in the robot 2 to determine contact with parts of the robot 2, and this information is used as the determination result. Note that the determination is made using the sensors provided in the robot 2 in this way, and the second determinator 641-2 may make the determination using other methods, not limited to the learned model.

[0042] The information required to determine the available degrees of freedom may be setting information set by the operator, or may be information based on the detection values ​​of the sensor 214 and the environmental sensor 3. Here, the available degrees of freedom refer to the degrees of freedom of the joints of the specified part, excluding, for example, directions in which movement is not possible due to the environment. For example, if there is a wall or other object around the specified part, moving the joint in that direction may result in a collision with the wall or other object. Therefore, the degree of freedom determination unit 63 sets the degrees of freedom based on, for example, the detection results detected by the environmental sensor 3.

[0043] The inverse kinematics calculation unit 642 calculates the objective function C of the following equation (1) using the input "end effector part, available degrees of freedom, and end effector command value." ａｌｌ is solved to calculate the joint angle target value.

[0044]

[0045] In addition, in the formula (1), W １ , W ２ is the weight of the joint angular acceleration task, and is expressed as a vector of degrees of freedom, so the magnitude of the weight expresses the degrees of freedom that can be used. １ and W ２ is a parameter to be updated. As shown in equation (1), the objective function is, for example, C １ and C related to joint angles ２The objective function is constructed from the following. Each cost is expressed by a weighted norm, for example, and the dimension of the weight vector corresponds to the dimension of the cost. For example, C ２ The dimension of the weight vector corresponds to the joint degrees of freedom, and by changing this for each joint, the degrees of freedom to be actively used are controlled. １ and C related to joint angles ２ The objective function is constructed from the following. Each cost is expressed by a weighted norm, for example, and the dimension of the weight vector corresponds to the dimension of the cost. For example, C ２ The dimension of the weight vector corresponds to the joint degrees of freedom, and by changing this for each joint, the degrees of freedom to be actively used are controlled.

[0046] As in Equation (1), the end effector part is the objective function of the quadratic programming problem, and the available degrees of freedom are reflected in the weights of the quadratic programming problem. That is, in this embodiment, mathematical optimization is used to solve the inverse kinematics, an objective function related to the end effector target is added according to the set part, and the weights of the objective function related to the joint angle are changed according to the determined degrees of freedom. Minimizing the norm of the first term in Equation (1) means placing (moving) the specified part to a certain position in three-dimensional space.

[0047] In the control, the coordinate system used is, for example, a robot coordinate system. Positions in different coordinate systems are converted into the robot coordinate system using a well-known method.

[0048] [Example of Processing Procedure] Next, a description will be given of an example of processing procedure performed by the robot control device 6. Fig. 5 is a flowchart of processing performed by the robot control device according to this embodiment.

[0049] (Step S1 ) The acquisition unit 61 acquires the operation result detected by the operation unit 69 .

[0050] (Step S2) The acquisition unit 61 acquires the first sensor value (robot state quantity) detected by the sensor 214 from the robot 2. The acquisition unit 61 acquires the first sensor value detected by the environment sensor 3.

[0051] (Step S3) The part designation unit 62 estimates the relationship between the operator's operation intention and the operation target object based on information acquired from, for example, the environment sensor 3, the operation unit 69, etc.

[0052] (Step S4) The part designation unit 62 sets the parts of the robot 2 to be operated and the number of parts on the robot 2 based on the estimated operation intention, the detection results of the environmental sensor 3, etc. Note that the part designation unit 62 may also designate parts by acquiring information indicating the parts designated by the operator by operating the operation unit 69, for example.

[0053] (Step S5) The degree of freedom determination unit 63 determines the degrees of freedom to be used among the joints of the specified part based on, for example, the specified part and the detection results of the environmental sensor 3.

[0054] (Step S6) The inverse kinematics calculation unit 642 of the control unit 64 calculates the joint angle target value using the "end effector part, available degrees of freedom, end effector command value" and equation (1).

[0055] (Step S7) The control unit 64 transmits the joint angle target value or a drive signal generated by inputting the joint angle target value to the drive circuit 65 to the robot 2 via the output unit 67.

[0056] [Example of changing the degree of freedom according to the contact point] Next, an example of changing the degree of freedom according to the contact point will be explained using Fig. 6. Fig. 6 is a front view of the robot, showing an example of changing the degree of freedom according to the contact point. Note that Fig. 6 is an example of the results of a simulation.

[0057] Image g50 in Fig. 6 is a diagram showing the degrees of freedom etc. before the contact point is changed. Image g60 in Fig. 6 is a diagram showing the degrees of freedom etc. after the contact point is changed. Reference symbol g71 indicates the fingertip joint, reference symbol g72 indicates the wrist joint, reference symbol g73 indicates the arm joint, reference symbol g74 indicates the first shoulder joint, and reference symbol g75 indicates the second shoulder joint. Reference symbol g76 indicates the contact point. Reference symbol g77 indicates the direction of the degrees of freedom at each joint.

[0058] Before contact, as shown by the dashed-line rectangle g51 in the diagram indicated by reference symbol g50, the degrees of freedom of the arm, hand, and fingers are all controlled and available without any restrictions. In contrast, after contact, as shown by the dashed-line rectangle g61 indicated by reference symbol g60, the area below the contact point g71 is used without moving the area above the contact point g71, so that the available degrees of freedom are restricted (applied). Note that such switching of the available degrees of freedom is determined by the degree-of-freedom determination unit 63, which determines whether or not an object has come into contact with the contact point based on the detection results of, for example, a sensor provided in the robot 2 or the environmental sensor 3.

[0059] Without the control of this embodiment, the shoulder, arm, and hand can move without any restrictions on the degree of freedom because contact points other than the fingertips are not taken into consideration, which causes a discrepancy between the actual measured values ​​obtained by the sensors equipped in the robot and the command values.

[0060] In contrast, when the control of this embodiment is performed, the control limits the degrees of freedom of the shoulders and the like in consideration of the contact points, and the joint angles can be controlled within a range in three-dimensional space. In this way, according to this embodiment, the difference between the command values ​​and the actual measured values ​​acquired by the sensors equipped in the robot 2 can be reduced.

[0061] In the above example, an example of controlling one manipulator 21 has been described, but when controlling two manipulators 21, the above-described control method is used to control each of the manipulators 21. The number of manipulators 21 may be three or more, and in that case, each of the manipulators 21 may be controlled by the above-described control method.

[0062] As described above, in this embodiment, the positions and number of end effectors are determined based on the judgment of a person or the robot control device 6. Then, in this embodiment, the degrees of freedom that can be used are actively determined based on the judgment of a person or the robot control device 6. Furthermore, in this embodiment, mathematical optimization is used to solve the inverse kinematics, an objective function related to the end effector target is added according to the determined positions and number, and the weight of the objective function related to the joint angle is changed according to the determined degrees of freedom.

[0063] As a result, according to this embodiment, by changing the end effector part depending on the situation, it becomes possible to perform operations suited to the task or scene. Furthermore, according to this embodiment, even when the degrees of freedom are limited due to contact with the environment (for example, when contacting or being close to a wall), it is possible to continue moving with the remaining degrees of freedom, thereby expanding the range of applications.

[0064] Second Embodiment Next, an example of remotely controlling a robot 2 will be described. FIG. 7 is a diagram for explaining an overview of remotely controlling a robot. As shown in FIG. 7 , in a remote control space, an operator Us wears, for example, an HMD (head-mounted display) 4 on his head and operation units 5 (5L, 5R) such as data gloves on his hands. An environmental sensor 3 and a robot control device 6A are installed in the robot workspace. Note that the environmental sensor 3 may be attached to the robot 2. The robot 2 also includes a robot control device 6 and a manipulator 21 (a first manipulator 21L and a second manipulator 21R). The manipulator 21 includes, for example, a hand 211 (211L, 211R). The operator Us remotely controls the robot 2 to manipulate a target object obj by, for example, moving the hand or fingers wearing the operation unit 5 while viewing an image displayed on the HMD 4. The robot 2 performs a task in the robot workspace in accordance with the remote control. The robot 2 does not have to be a humanoid robot and may have one arm.

[0065] [Configuration of Robot Control System] Next, a configuration example of a robot control system 1A will be described. Fig. 8 is a diagram showing a configuration example of a robot control system according to this embodiment. As shown in Fig. 8, the robot control system 1A includes, for example, a robot 2, an environment sensor 3, an HMD 4, an operation unit 5, and a robot control device 6A.

[0066] The HMD 4 includes, for example, an image display unit 41, a gaze detection unit 42, and a communication unit 43. The HMD 4 also includes a power supply unit (not shown) and the like. The HMD 4 transmits and receives various information to and from the robot control device 6A via a wired or wireless network NW.

[0067] The operation unit 5 includes, for example, a sensor 51 and a communication unit 52. The operation unit 5 transmits information to the robot control device 6A via a wired or wireless network NW.

[0068] The robot control device 6A includes, for example, an acquisition unit 61A, a part designation unit 62A, a degree of freedom determination unit 63A, a control unit 64, a drive circuit 65, an image generation unit 66, an output unit 67A, and a storage unit 68A. The robot control device 6 also includes a power supply unit (not shown). The robot control device 6A transmits and receives various information to and from the robot 2 and HDM 4 via a wired or wireless network NW. The robot control device 6A receives various information from the environmental sensor 3 and operation unit 5 via the wired or wireless network NW.

[0069] The acquisition unit 61A acquires the second sensor value detected by the sensor 51 of the operation unit 5 and the second sensor value detected by the gaze detection unit 42 of the HMD 4 in addition to the information acquired by the acquisition unit 61.

[0070] The part designation unit 62A designates the part and the number of parts of the robot 2 to be operated. The part designation unit 62A may designate a part by acquiring information indicating a part designated by the operator operating the operation unit 5. Alternatively, the part designation unit 62A may designate a part by estimating the relationship between the operator's operation intention and the operation target object based on information acquired from the environmental sensor 3, the operation unit 5, etc.

[0071] The degree of freedom determination unit 63A determines the degree of freedom to be used for each designated part, for example, depending on the task or the environment. Note that the degree of freedom determination unit 63A may determine the degree of freedom based on, for example, the detection results of the environmental sensor 3 or the estimated operation content, or may determine the degree of freedom based on the degree of freedom input by the operator, for example, by operating the operation unit 5.

[0072] The image generation unit 66 generates an image to be provided to the HMD 4 based on the image captured by the environmental sensor 3. Note that, as for a method of generating an image to be displayed on the HMD 4 and examples of the image, for example, the method described in Japanese Patent Application No. 2022-156322 may also be used.

[0073] The output unit 67A outputs the image data generated by the image generation unit 66 to the HMD 4 in addition to the information output by the output unit 67 .

[0074] The storage unit 68A stores the identification information of the HMD 4, the identification information of the operation unit 5, etc. in addition to the information stored in the storage unit 68.

[0075] With the above configuration, according to this embodiment, in remote operation, the end effector part can be changed depending on the situation, making it possible to perform operations suited to the task or scene, just as in the first embodiment. Furthermore, according to this embodiment, even when the degrees of freedom are limited due to contact with the environment (for example, when in contact with or close to a wall), it is possible to continue moving with the remaining degrees of freedom, thereby expanding the range of applications.

[0076] In real-world situations, humans manipulate objects using various parts of their body, such as their fingers, wrists, or elbows. Therefore, an end effector fixed to a specific part is not necessarily appropriate. Furthermore, in situations where continuous movement with limited degrees of freedom is required, such as when performing a task using the degrees of freedom from the elbow while keeping the elbow in contact with the environment, conventional techniques have found it difficult to achieve this by keeping the manipulator's kinetic chain fixed. In contrast, the above-described embodiment achieves the advantage of easier implementation by treating the kinetic chain as a task and its weight rather than directly changing its definition. Furthermore, the above-described embodiment provides multiple parts that can serve as end effectors, enabling a wider variety of operations by dynamically changing the definition.

[0077] A program for implementing all or part of the functions of the robot control device 6 (or 6A) of the present invention may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform all or part of the processing performed by the robot control device 6 (or 6A). The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a web page provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSIs (Large Scale Integration) such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), GPUs (Graphics Processing Units), and SOCs (System On Chips), or by hardware (including circuitry), or may be realized by a combination of software and hardware.

[0078] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0079] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0080] REFERENCE SIGNS LIST 1...robot control system, 2...robot, 3...environment sensor, 4...HMD, 5...operation unit, 6...robot control device, 21...manipulator, 21L...first manipulator, 21R...second manipulator, 211, 211L, 211R...hand, 212, 212L, 212R...arm, 213, 213L, 213R...actuator, 214, 214, 214L, 214R...sensor Sa, 22...body, 23...communication unit, 31...sensor, 32...communication unit, 41...image display unit, 42...gaze detection unit, 43...communication unit, 51...sensor, 52...communication unit, 61...acquisition unit, 62...region designation unit, 63...degree of freedom setting unit, 64...control unit, 65...drive circuit, 66...image generation unit, 67...output unit, 68...storage unit, 69...operation unit, NW...network, 641...determiner, 642...inverse kinematics calculation unit

Claims

1. A robot control system comprising: a robot having a plurality of joints; a sensor for acquiring state quantities of the robot; a part designation unit for designating a part of the robot to be operated; a degree of freedom determination unit for determining the degrees of freedom to be used of the robot; and a control unit that receives as input the state quantities of the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part of the robot to a target position in three-dimensional space.

2. The robot control system of claim 1, wherein the control unit calculates joint angle targets using inverse kinematics, uses mathematical optimization to solve the inverse kinematics, adds costs related to target positions of parts on the robot to an objective function, and changes the weight of the costs related to corresponding joint angles depending on the degrees of freedom used.

3. A robot control system as described in claim 1 or claim 2, wherein the part designation unit determines the parts on the robot using information necessary for determining the parts on the robot, and the information necessary for determining the parts on the robot is at least one of objects around the robot, work content, objects around each of the candidate parts, and position information or posture information for each of the candidate parts.

4. A robot control system as described in claim 1 or claim 2, wherein the degree of freedom determination unit determines the available degrees of freedom using information necessary for determining the available degrees of freedom, and the information necessary for determining the available degrees of freedom is at least one of the degrees of freedom of each candidate part, objects around the robot, work content, objects around each of the candidate parts, and position information or posture information of each of the candidate parts.

5. A robot control system according to claim 1 or claim 2, wherein the part on the robot is at least one of the fingertips, finger pads, palm, wrist, and part on an arm of a hand provided on the robot.

6. A robot control device comprising: a part designation unit that designates a part of a robot having a plurality of joints to be operated; a degree of freedom determination unit that determines the degrees of freedom to be used of the robot; and a control unit that receives as inputs the state quantities of the robot acquired by a sensor possessed by the robot, the part on the robot, a target position of the part on the robot, and the degrees of freedom to be used, and outputs joint angle targets that move the part on the robot to a target position in three-dimensional space.

7. A robot control method, wherein a part designation unit designates a part of a robot having a plurality of joints to be operated, a degree of freedom determination unit determines the degrees of freedom to be used of the robot, and a control unit receives as input the state quantities of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part of the robot to a target position in three-dimensional space.

8. A program that causes a computer of a robot control device to specify a part of a robot having multiple joints to be operated, causes a degree of freedom determination unit to determine the degrees of freedom to be used of the robot, and causes a control unit to input the state quantities of the robot acquired by a sensor possessed by the robot, the part on the robot, the target position of the part on the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part on the robot to a target position in three-dimensional space.

Images