Robot control system, robot control device, robot control method, and program

Abstract

To provide a robot control system, a robot control device, a robot control method, and a program that enable operations suitable for tasks and scenes.SOLUTION: A robot control system comprises a robot having a plurality of joints, a sensor that acquires a state quantity of the robot, a part designation unit that designates a part on the robot to be operated in the robot, a degree-of-freedom determination unit that determines a degree of freedom to be used in the robot, and a control unit that receives, as inputs, the state quantity of the robot, the part on the robot, a target position of the part on the robot, and the degree of freedom to be used, and outputs a joint angle target for moving the part on the robot to a target position in a three-dimensional space.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a robot control system, a robot control device, a robot control method, and a program. [Background technology]

[0002] For example, a method for operating a robot with a manipulator has been proposed. In all such operations, the position and orientation of a specific part (end effector) on the manipulator in three-dimensional space and the force applied from that part to an object are controlled. In a typical manipulator, the end effector is usually located near the tip of the manipulator (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-20259 Summary of the Invention [Problem to be solved by the invention]

[0004] In real-world situations, people manipulate objects using various parts of their body, such as their wrists or elbows. Therefore, controlling an end effector that is fixed to a specific part, as in conventional technology, is often inappropriate. This makes it difficult for conventional technology to achieve operations appropriate for a given task or scene.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a robot control system, a robot control device, a robot control method, and a program that enable operations appropriate for tasks and scenes. [Means for solving the problem]

[0006] (1) In order to achieve the above object, a robot control system according to one aspect of the present invention is a robot control system including: a robot having a plurality of joints; a sensor for acquiring state quantities of the robot; a part designation unit for designating a part of the robot to be operated; a degree-of-freedom determination unit for determining degrees of freedom to be used of the robot; and a control unit that receives as input the state quantities of the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part of the robot to a target position in three-dimensional space.

[0007] (2) In one aspect of the robot control system according to (1) above, the control unit may calculate joint angle targets using inverse kinematics, use mathematical optimization to solve the inverse kinematics, add costs related to the target positions of parts on the robot to an objective function, and change the weight of the costs related to the corresponding joint angles depending on the degrees of freedom used.

[0008] (3) In a robot control system according to one aspect of (1) or (2) above, the part designation unit may determine the parts on the robot using information necessary for determining the parts on the robot, and the information necessary for determining the parts on the robot may be at least one of objects around the robot, work content, objects around each of the candidate parts, and position information or posture information for each of the candidate parts.

[0009] (4) In a robot control system according to any one of the above (1) to (3), the degree of freedom determination unit determines the available degrees of freedom using information necessary for determining the available degrees of freedom, and the information necessary for determining the available degrees of freedom may be at least one of the degrees of freedom of each candidate part, objects around the robot, work content, objects around each candidate part, and position information or posture information of each candidate part.

[0010] (5) In a robot control system according to any one of the above aspects (1) to (4), the part on the robot may be at least one of the fingertips, finger pads, palm, wrist, and part on the arm of a hand provided on the robot.

[0011] (6) In order to achieve the above object, a robot control device according to one aspect of the present invention is a robot control device that includes: a part designation unit that designates a part of a robot having a plurality of joints to be operated; a degree of freedom determination unit that determines the degrees of freedom to be used of the robot; and a control unit that receives as inputs a state quantity of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs a joint angle target that moves the part of the robot to a target position in three-dimensional space.

[0012] (7) In order to achieve the above object, one aspect of the present invention provides a robot control method in which a part designation unit designates a part of a robot having a plurality of joints to be operated, a degree of freedom determination unit determines the degrees of freedom to be used of the robot, and a control unit receives as inputs the state quantities of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part of the robot to a target position in three-dimensional space.

[0013] (8) In order to achieve the above object, a program according to one aspect of the present invention causes a computer of a robot control device to specify a part of a robot having a plurality of joints to be operated, causes a degree of freedom determination unit to determine the degrees of freedom to be used of the robot, and causes a control unit to input the state quantities of the robot acquired by a sensor possessed by the robot, the part of the robot, a target position of the part of the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part of the robot to a target position in three-dimensional space. [Effects of the Invention]

[0014] According to the above (1) to (8), it is possible to perform operations suited to the task or scene. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram illustrating an example of a robot workspace. [Figure 2] 1 is a diagram illustrating an example of the configuration of a robot control system according to a first embodiment. [Figure 3] 10A and 10B are diagrams illustrating examples of parts and examples of behavior when parts are changed in the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a process for generating a joint angle target value according to the first embodiment. [Figure 5] 3 is a flowchart of processing performed by the robot control device according to the first embodiment. [Figure 6] This is an example of changing the degree of freedom depending on the contact point, and is a diagram showing the robot as seen from the front. [Figure 7] FIG. 1 is a diagram for explaining an overview of remote control of a robot. [Figure 8] FIG. 10 is a diagram illustrating an example of the configuration of a robot control system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component can be recognized. In all the drawings for explaining the embodiments, the same reference numerals are used for components having the same functions, and repeated explanations will be omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0017] [overview] In this embodiment, for example, in a robot with a hand equipped with multiple fingers, the tip or vicinity of the finger is not fixed as an end effector as in the prior art, but the end effector is switched depending on the operation content and the scene. Thus, in this embodiment, an "end effector" is a part that can be used in a task. The usable part is, for example, at least one of the fingertips, finger pads, palm, wrist, and part on the arm of the end effector (hand) equipped in the robot.

[0018] First Embodiment In this embodiment, an example will be described in which an operator controls a robot by inputting work instructions and the like. FIG. 1 is a diagram illustrating an example of a robot workspace. As shown in FIG. 1, an environmental sensor 3 and a robot control device 6 are installed in the robot workspace. The environmental sensor 3 may be attached to the robot 2. The robot 2 also includes the robot control device 6 and a manipulator 21 (a first manipulator 21L and a second manipulator 21R). The manipulator 21 includes, for example, hands 211 (211L, 211R). The operator operates the operation unit 69 (FIG. 2) provided in the robot control device 6 to cause the robot 2 to operate the target object obj. The robot 2 performs work in the robot workspace under the control of the robot control device 6. The robot 2 does not have to be a humanoid robot, and may have one arm.

[0019] [Robot control system configuration] Next, a description will be given of an example of the configuration of the robot control system 1. Fig. 2 is a diagram showing an example of the configuration of the robot control system according to this embodiment. As shown in FIG. 2, the robot control system 1 includes, for example, a robot 2, an environmental sensor 3, and a robot control device 6.

[0020] The robot 2 includes, for example, a first manipulator 21L, a second manipulator 21R, a body 22, and a communication unit 23. The first manipulator 21L includes, for example, a hand 211L, an arm 212L, an actuator 213L, and a sensor 214L. The second manipulator 21R includes, for example, a hand 211R, an arm 212R, an actuator 213R, and a sensor 214R. The robot 2 may also include a power supply unit, legs, a head, etc., which are not shown.

[0021] In the following description, when there is no need to distinguish between the first manipulator 21L and the second manipulator 21R, they will also be referred to as manipulators 21. Similarly, when there is no need to distinguish between the hands 211L and 211R, they will also be referred to as hands 211, and when there is no need to distinguish between the arms 212L and 212R, they will also be referred to as arms 212. When there is no need to distinguish between the actuators 213L and 213R, they will also be referred to as actuators 213, and when there is no need to distinguish between the sensors 214L and 214R, they will also be referred to as sensors 214. The robot 2 transmits and receives various information to and from the robot control device 6 via a wired or wireless network NW.

[0022] The environmental sensor 3 includes, for example, a sensor 31 and a communication unit 32. The environmental sensor 3 also includes a power supply unit (not shown) and the like. The environmental sensor 3 transmits and receives various information to and from the robot control device 6 via a wired or wireless network NW.

[0023] The robot control device 6 includes, for example, an acquisition unit 61, a part designation unit 62, a degree of freedom determination unit 63, a control unit 64, a drive circuit 65, an output unit 67, a storage unit 68, and an operation unit 69. The robot control device 6 also includes a power supply unit and the like (not shown). The robot control device 6 transmits and receives various information to and from the robot 2 and HDM 4 via a wired or wireless network NW. The robot control device 6 receives various information from the environmental sensor 3 via the wired or wireless network NW.

[0024] The configuration example shown in FIG. 2 is just an example, and the present invention is not limited to this.

[0025] [Functions of each device in the robot control system] Next, the functions of each device in the robot control system 1 will be described with reference to FIG. (Robot 2) The hand 211 includes, for example, a plurality of fingers. Each finger includes a joint. The hand 211 may be a gripper or the like. One end of the arm 212 is attached to the hand 211 via a joint, and the other end is attached to the body 22 via a joint. An actuator 213 is attached to each joint. The sensor 214 is, for example, a six-axis sensor attached to a joint, a tactile sensor attached to a finger, a force sensor, etc. The six-axis sensor detects forces along three axes (x, y, z) and moments along three axes (α, β, γ).

[0026] The communication unit 23 transmits the detection value detected by the sensor 214 to the robot control device 6. The communication unit 23 receives the control signal or control instruction output by the robot control device 6. The data output by the robot 2 includes identification information that can identify the robot 2. The data acquired by the robot 2 also includes identification information that can identify that the data is addressed to the robot 2. The robot 2 may include a drive circuit for driving the actuator 213.

[0027] (Environmental Sensor 3) The environmental sensor 3 is installed, for example, in the robot workspace, as shown in FIG. The sensor 31 is, for example, an RGB-D camera that acquires RGB (red, green, blue) information and depth information. Note that the information is acquired at predetermined time intervals, for example. The communication unit 32 outputs the detection value detected by the sensor 31 to the robot control device 6. The data output by the environmental sensor 3 includes identification information that allows the environmental sensor 3 to be identified.

[0028] (Robot control device 6) The acquisition unit 61 acquires a first sensor value detected by the sensor 214 from the robot 2. The acquisition unit 61 acquires a first sensor value detected by the environment sensor 3. The acquisition unit 61 acquires an operation result of the operator detected by the operation unit 69.

[0029] The part designation unit 62 designates the parts and number of parts on the robot 2 to be operated. The part designation unit 62 may designate the parts by acquiring information indicating the parts input by the operator operating the operation unit 69. Alternatively, the part designation unit 62 may designate the parts by estimating the relationship between the operator's operation intention (see, for example, Japanese Patent Application No. 2022-006498) and the operation target object based on information acquired from the environment sensor 3. The parts to be set are not limited to one, and may be multiple (for example, the thumb and index finger).

[0030] The degree of freedom determination unit 63 determines the degrees of freedom to be used for each designated part, for example, depending on the task or the environment. Note that the degree of freedom determination unit 63 may determine the degrees of freedom based on, for example, the detection results of the environmental sensor 3 or the estimated operation content, or may determine the degrees of freedom based on the degrees of freedom input by the operator operating the operation unit 69. For example, when the arm 212 of the robot 2 performs a task while in contact with a wall, the contact with the wall may be detected based on the detection results of the force sensor or the environmental sensor 3 provided in the robot 2, or may be input by the operator operating the operation unit 69.

[0031] The control unit 64 receives as input the robot state quantities, the part on the robot 2 to be operated, the target position of the part on the robot 2 to be operated, and the degrees of freedom to be used, and calculates the target joint angles of the robot 2 to move the part on the robot 2 to the target position (target position in three-dimensional space). The control unit 64 generates a robot control command including the target joint angles, and outputs the generated robot control command to the drive circuit 65.

[0032] The drive circuit 65 generates a drive signal for controlling the robot 2 based on the robot control command generated by the control unit 64. Note that if the robot 2 is equipped with the drive circuit 65, the robot control device 6 does not need to be equipped with the drive circuit 65. Alternatively, the robot control device 6 and the robot 2 may each be equipped with a part of the drive circuit 65.

[0033] The output unit 67 outputs the drive signal output by the drive circuit 65 or the operation command generated by the control unit 64 to the robot 2.

[0034] The storage unit 68 stores, for example, programs, mathematical expressions, thresholds, identification information of the robot 2, identification information of the environmental sensor 3, and the like used by each unit of the robot control device 6.

[0035] [Examples of parts and behavior when changing parts] Next, examples of parts and examples of behavior when the parts are changed will be described. Fig. 3 is a diagram showing examples of parts in this embodiment and examples of behavior when the parts are changed. In FIG. 3, reference symbol g11 denotes a fingertip, reference symbol g12 denotes a hand 211, reference symbol g13 denotes a wrist, and reference symbol g14 denotes an arm 212.

[0036] Reference symbol g20 is an example of behavior when the tip of the index finger g11 is set as the part. In this case, the robot control device 6 issues a joint angle command centered on the tip of the index finger g11, as shown by the circle g21. In this case, since the specified part is the fingertip, control is performed using the respective degrees of freedom of the arm and finger, for example, to achieve this.

[0037] Reference symbol g30 is an example of behavior when the wrist g13 is set as the part. In this case, the robot control device 6 issues a joint angle command centered on the wrist g13, as shown in circle g31. In this case, since the specified part is the wrist, to achieve this, for example, control is performed using the degrees of freedom of each of the arm and hand.

[0038] [Joint angle target value] Next, an example of a method for generating target joint angle values ​​performed by the part designation unit 62, degree of freedom determination unit 63, and control unit 64 will be described. FIG. 4 is a diagram showing an example of a process for generating target joint angle values ​​according to this embodiment. The control unit 64 includes, for example, an inverse kinematics calculation unit 642. A determiner 641 corresponds to the part designation unit 62 and degree of freedom determination unit 63.

[0039] The decision unit 641 is, for example, a trained model. The first decision unit 641-1 corresponds to the part designation unit 62, and the second decision unit 641-2 corresponds to the degree of freedom determination unit 63. Note that the decision unit 641 outputs the end effector command value as it is, since the end effector command value is a command value for where to move the designated end effector part.

[0040] During learning, the first decision device 641-1 receives "information necessary to determine the part of the end effector (part on the robot)" and training data that is the correct answer to output, and outputs "the part of the end effector." When in use, the first determiner 641-1 receives "information necessary to determine the part of the end effector (part on the robot)" and outputs "part of the end effector." The "information necessary to determine the end effector's location" refers to at least one of the following: objects around the robot, the work content, objects around each of the candidate locations, and the position information or posture information of each of the candidate locations.

[0041] During learning, the second decision device 641-2 receives "information necessary to determine usable degrees of freedom" and teacher data that is the correct answer to output, and outputs "usable degrees of freedom." When in use, the second decision device 641-2 receives "information necessary to determine the usable degrees of freedom" and outputs "usable degrees of freedom." Furthermore, the "information necessary for determining the available degrees of freedom" is, for example, the information obtained by the second determiner 641-2 determining contact with a part of the robot 2 using the detection results of sensors (force sensors, torque sensors) provided in the robot 2. Note that the determination may be made using the sensors provided in the robot 2 in this way, and the second determiner 641-2 may make the determination using other methods, not limited to the learned model.

[0042] The information required to determine the available degrees of freedom may be setting information set by the operator, or may be information based on the detection values ​​of the sensor 214 and the environmental sensor 3. Here, the available degrees of freedom refer to the degrees of freedom of the joints of the specified part, excluding, for example, directions in which the joints cannot move due to the environment. For example, if there is a wall or other object around the specified part, moving the joint in that direction may result in a collision with the wall or other object. For this reason, the degree of freedom determination unit 63 sets the degrees of freedom based on, for example, the detection results detected by the environmental sensor 3.

[0043] The inverse kinematics calculation unit 642 calculates the objective function C of the following equation (1) using the input "end effector part, available degrees of freedom, end effector command value": all is solved to calculate the joint angle target value.

[0044]

number

[0045] In equation (1), W1 and W2 are weights for the joint angular acceleration task, and are expressed as vectors for the degrees of freedom, so the magnitude of the weights represents the degrees of freedom that can be used. Also, in equation (1), W1 and W2 are parameters to be updated. As shown in equation (1), the objective function is composed of, for example, C1 related to the end effector position and C2 related to the joint angle. Each cost is expressed, for example, by a weighted norm, and the dimension of the weight vector corresponds to the dimension of the cost. For example, the dimension of the weight vector of C2 corresponds to the joint degrees of freedom, and by changing this for each joint, the degrees of freedom that are actively used are controlled. As shown in equation (1), the objective function is composed of, for example, C1 related to the end effector position and C2 related to the joint angle. Each cost is expressed, for example, by a weighted norm, and the dimension of the weight vector corresponds to the dimension of the cost. For example, the dimension of the weight vector of C2 corresponds to the joint degrees of freedom, and by changing this for each joint, the degrees of freedom that are actively used are controlled.

[0046] As shown in equation (1), the end effector part is the objective function of the quadratic programming problem, and the available degrees of freedom are reflected in the weights of the quadratic programming problem. That is, in this embodiment, mathematical optimization is used to solve the inverse kinematics, an objective function related to the end effector target is added according to the set part, and the weights of the objective function related to the joint angle are changed according to the determined degrees of freedom. Minimizing the norm of the first term in equation (1) means placing (moving) the specified part to a certain position in three-dimensional space.

[0047] In the control, the coordinate system used is, for example, a robot coordinate system. Positions in different coordinate systems are converted into the robot coordinate system using a well-known method.

[0048] [Example of processing procedure] Next, a description will be given of an example of the processing procedure performed by the robot control device 6. Fig. 5 is a flowchart of the processing performed by the robot control device according to this embodiment.

[0049] (Step S1) The acquisition unit 61 acquires the operation result detected by the operation unit 69.

[0050] (Step S2) The acquisition unit 61 acquires the first sensor value (robot state quantity) detected by the sensor 214 from the robot 2. The acquisition unit 61 acquires the first sensor value detected by the environment sensor 3.

[0051] (Step S3) The part designation unit 62 estimates the relationship between the operator's operation intention and the operation target object based on information acquired from the environment sensor 3, the operation unit 69, etc., for example.

[0052] (Step S4) The part designation unit 62 sets the parts of the robot 2 to be operated and the number of parts on the robot 2 based on the estimated operation intention, the detection results of the environmental sensor 3, etc. Note that the part designation unit 62 may also designate parts by acquiring information indicating the parts designated by the operator by operating the operation unit 69, for example.

[0053] (Step S5) The degree of freedom determining unit 63 determines the degree of freedom to be used among the joints of the specified part based on the specified part and the detection results of the environmental sensor 3, for example.

[0054] (Step S6) The inverse kinematics calculation unit 642 of the control unit 64 calculates the joint angle target value using the "part of the end effector, the available degrees of freedom, and the end effector command value" and equation (1).

[0055] (Step S7) The control unit 64 transmits the joint angle target value or the drive signal generated by inputting the joint angle target value to the drive circuit 65 to the robot 2 via the output unit 67.

[0056] [Example of changing the degrees of freedom depending on the contact point] Next, an example of changing the degree of freedom depending on the contact point will be described with reference to FIG. Figure 6 shows an example of changing the degree of freedom depending on the contact point, and is a front view of the robot. Note that Figure 6 is an example of the results of a simulation.

[0057] Image g50 in Fig. 6 is a diagram showing the degrees of freedom etc. before the contact point is changed, and image g60 in Fig. 6 is a diagram showing the degrees of freedom etc. after the contact point is changed. Symbol g71 is the fingertip joint, symbol g72 is the wrist joint, symbol g73 is the arm joint, symbol g74 is the first shoulder joint, and symbol g75 is the second shoulder joint. Symbol g76 represents the contact point. Symbol g77 represents the direction of each degree of freedom at each joint.

[0058] Before contact, as shown by the dashed rectangle g51 in the diagram of reference symbol g50, for example, the arm, hand, and fingers are controlled without any restrictions so that all degrees of freedom can be used. In contrast, after contact, as shown by the dashed-line rectangle g61 of symbol g60, the area below the contact point g71 is used without moving the area above the contact point g71, so the available degrees of freedom are restricted (applied to). Note that such switching of the available degrees of freedom is determined by the degree-of-freedom determination unit 63, which determines whether or not an object has come into contact with the contact point based on the detection results of, for example, a sensor provided in the robot 2 or the environmental sensor 3.

[0059] Without the control of this embodiment, the shoulder, arm, and hand can move without any restrictions on the degree of freedom because contact points other than the fingertips are not taken into consideration, which causes a discrepancy between the actual measured values ​​obtained by the sensors equipped in the robot and the command values.

[0060] In contrast, when the control of this embodiment is performed, the control limits the degrees of freedom of the shoulders and the like in consideration of the contact points, and the joint angles can be controlled within a range in three-dimensional space. In this way, according to this embodiment, the difference between the command values ​​and the actual measured values ​​acquired by the sensors equipped in the robot 2 can be reduced.

[0061] In the above example, an example of controlling one manipulator 21 has been described, but when controlling two manipulators 21, each manipulator 21 is controlled by the above-mentioned control method. The number of manipulators 21 may be three or more, and in that case, each manipulator 21 may be controlled by the above-mentioned control method.

[0062] As described above, in this embodiment, the position and number of end effectors are determined based on the judgment of a person or the robot control device 6. Then, in this embodiment, the degrees of freedom that can be used are actively determined based on the judgment of a person or the robot control device 6. Furthermore, in this embodiment, mathematical optimization is used to solve the inverse kinematics, an objective function related to the end effector target is added according to the determined position and number, and the weight of the objective function related to the joint angle is changed according to the determined degrees of freedom.

[0063] As a result, according to this embodiment, by changing the end effector part depending on the situation, it becomes possible to perform operations suited to the task or scene. Furthermore, according to this embodiment, even when the degrees of freedom are limited due to contact with the environment (for example, when contacting or being close to a wall), it is possible to continue moving with the remaining degrees of freedom, thereby expanding the range of applications.

[0064] Second Embodiment Next, an example of remotely controlling the robot 2 will be described. FIG. 7 is a diagram for explaining an overview of remote control of a robot. As shown in FIG. 7, in the remote control space, an operator Us wears, for example, an HMD (head mounted display) 4 on his head and operation units 5 (5L, 5R) such as data gloves on his hands. An environment sensor 3 and a robot control device 6A are installed in the robot working space. Note that the environment sensor 3 may be attached to the robot 2. The robot 2 also includes a robot control device 6 and a manipulator 21 (a first manipulator 21L and a second manipulator 21R). The manipulator 21 includes, for example, hands 211 (211L, 211R). The operator Us remotely controls the robot 2 and manipulates the target object obj, for example, by moving the hand or fingers wearing the manipulation unit 5 while viewing an image displayed on the HMD 4. The robot 2 performs work in the robot workspace in accordance with remote control. Note that the robot 2 does not have to be a humanoid robot and may have one arm.

[0065] [Robot control system configuration] Next, a configuration example of the robot control system 1A will be described. Fig. 8 is a diagram showing a configuration example of the robot control system according to this embodiment. As shown in FIG. 8, a robot control system 1A includes, for example, a robot 2, an environment sensor 3, an HMD 4, an operation unit 5, and a robot control device 6A.

[0066] The HMD 4 includes, for example, an image display unit 41, a gaze detection unit 42, and a communication unit 43. The HMD 4 also includes a power supply unit (not shown) and the like. The HMD 4 transmits and receives various information to and from the robot control device 6A via a wired or wireless network NW.

[0067] The operation unit 5 includes, for example, a sensor 51 and a communication unit 52. The operation unit 5 transmits information to the robot control device 6A via a wired or wireless network NW.

[0068] The robot control device 6A includes, for example, an acquisition unit 61A, a part designation unit 62A, a degree of freedom determination unit 63A, a control unit 64, a drive circuit 65, an image generation unit 66, an output unit 67A, and a storage unit 68A. The robot control device 6 also includes a power supply unit (not shown) and the like. The robot control device 6A transmits and receives various information to and from the robot 2 and HDM 4 via a wired or wireless network NW. The robot control device 6A receives various information from the environmental sensor 3 and operation unit 5 via a wired or wireless network NW.

[0069] The acquisition unit 61A acquires the second sensor value detected by the sensor 51 of the operation unit 5 and the second sensor value detected by the line-of-sight detection unit 42 of the HMD 4 in addition to the information acquired by the acquisition unit 61.

[0070] The part designation unit 62A designates the part and number of parts on the robot 2 to be operated. The part designation unit 62A may designate a part by acquiring information indicating a part designated by the operator operating the operation unit 5. Alternatively, the part designation unit 62A may designate a part by estimating the relationship between the operator's operation intention and the operation target object based on information acquired from the environmental sensor 3, the operation unit 5, etc.

[0071] The degree of freedom determination unit 63A determines the degree of freedom to be used for each designated part, for example, according to the task or the environment. Note that the degree of freedom determination unit 63A may determine the degree of freedom based on, for example, the detection result of the environmental sensor 3 or the estimated operation content, or may determine the degree of freedom based on the degree of freedom input by the operator, for example, by operating the operation unit 5.

[0072] The image generation unit 66 generates an image to be provided to the HMD 4 based on the image captured by the environmental sensor 3. Note that, as for a method of generating an image to be displayed on the HMD 4 and examples of the image, for example, the method described in Japanese Patent Application No. 2022-156322 may also be used.

[0073] The output unit 67A outputs the image data generated by the image generation unit 66 to the HMD 4 in addition to the information output by the output unit 67.

[0074] The storage unit 68A stores the information stored in the storage unit 68, as well as the identification information of the HMD 4, the identification information of the operation unit 5, and the like.

[0075] With the above configuration, according to this embodiment, in remote operation, the end effector part can be changed depending on the situation, making it possible to perform operations suited to the task or scene, just like in the first embodiment. Furthermore, according to this embodiment, even when the degrees of freedom are limited by contact with the environment (for example, when in contact with or close to a wall), it is possible to continue moving with the remaining degrees of freedom, thereby expanding the range of applications.

[0076] In real-world situations, humans manipulate objects using various parts of their body, such as their fingers, wrists, or elbows, so an end effector fixed to a specific part is not necessarily appropriate. Furthermore, in situations where continuous movement with limited degrees of freedom is required, such as when performing a task using the degrees of freedom from the elbow down while keeping the elbow in contact with the environment, it has been difficult with conventional technology to achieve this while keeping the manipulator's kinetic chain fixed. In contrast to this, according to the above-described embodiment, the definition of the kinematic chain is not changed directly, but is handled as a task and its weight, which has the effect of making it easier to handle in implementation. Also, according to the above-described embodiment, since there are multiple parts that can become end effectors, dynamically changing these makes it possible to perform a wider variety of operations.

[0077] A program for implementing all or part of the functions of the robot control device 6 (or 6A) of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be loaded into a computer system and executed to perform all or part of the processing performed by the robot control device 6 (or 6A). The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a web page provision environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" also includes devices that retain a program for a certain period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line. Alternatively, some or all of these components may be realized by LSI (Large Scale Integration) such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or SOC (System On Chip) or hardware (including circuitry), or may be realized by a combination of software and hardware.

[0078] The program may also be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be a program that realizes part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0079] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0080] 1...robot control system, 2...robot, 3...environment sensor, 4...HMD, 5...operation unit, 6...robot control device, 21...manipulator, 21L...first manipulator, 21R...second manipulator, 211, 211L, 211R...hand, 212, 212L, 212R...arm, 213, 213L, 213R...actuator, 214, 214, 214L, 214R...sensor Sa, 22...body, 23...communication unit, 31...sensor, 32...communication unit, 41...image display unit, 42...gaze detection unit, 43...communication unit, 51...sensor, 52...communication unit, 61...acquisition unit, 62...region designation unit, 63...degree of freedom setting unit, 64...control unit, 65...drive circuit, 66...image generation unit, 67...output unit, 68...storage unit, 69...operation unit, NW...network, 641...determiner, 642...inverse kinematics calculation unit

Claims

1. a robot having multiple joints; a sensor for acquiring a state quantity of the robot; a part designation unit that designates a part of the robot to be operated; a degree of freedom determination unit that determines the degrees of freedom to be used in the robot; a control unit that receives as input the state quantities of the robot, a part on the robot, a target position of the part on the robot, and the degrees of freedom to be used, and outputs joint angle targets that move the part on the robot to the target position in three-dimensional space; A robot control system comprising:

2. The control unit Calculate the joint angle targets using inverse kinematics, The inverse kinematics is solved by using mathematical optimization, adding a cost related to a target position of the part on the robot to an objective function, and changing a weight of the cost related to the corresponding joint angle depending on the degree of freedom used. The robot control system of claim 1 .

3. The part designation unit determining a part on the robot using information necessary to determine the part on the robot; The information necessary to determine the part on the robot is: At least one of objects around the robot, work content, objects around each of the candidate parts, and position information or posture information of each of the candidate parts. The robot control system according to claim 1 or 2.

4. The degree of freedom determination unit determining the available degrees of freedom using information necessary to determine said available degrees of freedom; The information necessary to determine the available degrees of freedom is: At least one of the degrees of freedom of each candidate part, objects around the robot, work content, objects around each candidate part, and position information or posture information of each candidate part. The robot control system according to claim 1 or 2.

5. The part on the robot is: At least one of the fingertips, finger pads, palm, wrist, and arm of the hand of the robot. The robot control system according to claim 1 or 2.

6. a part designation unit that designates a part of a robot having a plurality of joints to be operated; a degree of freedom determination unit that determines the degrees of freedom to be used in the robot; a control unit that receives as inputs a state quantity of the robot acquired by a sensor included in the robot, a part on the robot, a target position of the part on the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part on the robot to a target position in three-dimensional space; A robot control device comprising:

7. the part designation unit designates a part of the robot to be operated among the robots having a plurality of joints; a degree of freedom determination unit that determines degrees of freedom to be used in the robot; a control unit receives as inputs a state quantity of the robot acquired by a sensor included in the robot, a part on the robot, a target position of the part on the robot, and the degrees of freedom to be used, and outputs a joint angle target for moving the part on the robot to a target position in three-dimensional space; Robot control method.

8. The robot control computer A part of a robot having a plurality of joints to be operated is designated; a degree of freedom determination unit that determines degrees of freedom to be used in the robot; a control unit receives as inputs the state quantities of the robot acquired by a sensor included in the robot, a part on the robot, a target position of the part on the robot, and the degrees of freedom to be used, and outputs joint angle targets for moving the part on the robot to the target position in three-dimensional space; program.

Images