Robot remote control device, robot remote control system, robot remote control method, and storage medium

Abstract

In the first invention, a robot remote operation control device is provided. When an operator remotely operates the robot, the device infers the operator's intended action to be performed by the robot based on the operator's state information, and determines the object-holding method based on the inferred action intention. In the second invention, a robot remote operation control device is provided. When an operator remotely operates the robot, the device infers the operator's action based on robot environmental sensor values ​​and operator sensor values. Based on the inferred operator's action, appropriate control commands are used to replace commands related to a portion of the operator's degrees of freedom, reducing the degrees of freedom of the operator's action and generating control commands. In the third invention, a control device for remote robot operation is provided. When an operator remotely operates the robot, the device infers the robot's motion status based on information about the robot's motion environment and operational status. Based on the operational status information, control commands are generated to drive and control the robot by causing the robot's actuators to move. In this case, the control commands are determined based on characteristic parameters related to the control characteristics corresponding to the motion status. In the fourth invention, a control device for remote operation of a robot is provided. When an operator remotely operates the robot, the device determines the predicted trajectory of the robot's actuators from the current time to a predicted time after a predetermined prediction time based on the robot's motion information and operation status information, and generates control commands based on the predicted trajectory.

Description

Technical Field

[0001] This invention relates to a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a storage medium.

[0002] This application claims priority based on Japanese Patent Application No. 2021-058952, No. 2021-061137, No. 2021-060914, and No. 2021-060904, all filed on March 31, 2021, the contents of which are incorporated herein by reference. Background Technology

[0003] A control device that enables a user to assist in operating a robot is proposed. As such a control device, for example, a control device is proposed comprising: a first information acquisition unit that acquires first user posture information representing the posture of a first user operating the robot; a second information acquisition unit that acquires pre-change posture information representing the posture before the change, the posture before the change being the robot's posture before the change is based on the first user posture information; and a determination unit that, based on the pre-change posture information and the first user posture information acquired by the first information acquisition unit at the time point in time when the robot performs the pre-change posture as represented by the pre-change posture information, determines a target posture different from the first user's posture as the robot's posture (see Patent Document 1).

[0004] In the system described in Patent Document 1, the robot's posture is changed to a posture corresponding to the posture detected by a device worn by the operator.

[0005] It should be noted that in order to perform various tasks using a remotely operated robot, it is necessary to accurately instruct and control the six degrees of freedom of the fingertip target (position, posture) in three-dimensional space. For example, in order to open the cap of a plastic bottle with the vertical direction of the plastic bottle not perpendicular to the table, the operator performing the remote operation instructs: align the grip with the vertical direction of the plastic bottle, and rotate the cap simultaneously.

[0006] Furthermore, robot systems that receive operator actions and control the robot according to the received actions have been proposed. For example, the robot system described in Patent Document 1 obtains first user posture information representing the posture of a first user operating the robot, and obtains pre-change posture information representing the pre-change posture of the robot before the posture changes based on the first user posture information. Based on the pre-change posture information and the first user posture information obtained at the time point when the robot performs the pre-change posture represented by the pre-change posture information, this robot system determines a target posture different from the first user's posture as the robot's posture.

[0007] Furthermore, the robot described in Patent Document 1 has a robotic arm composed of multiple joints. Sometimes, it is assumed that the operator wants the robotic arm to move according to a target posture in three-dimensional space. In this case, the robot system described in Patent Document 1 performs inverse kinematics calculations to determine the target angles, torques, and other individual control values ​​of each joint constituting the robotic arm, and controls its movements.

[0008] In addition, the robot system described in Patent Document 1 obtains user information from the user device via a network, which serves as the basis for the first user's posture information.

[0009] Prior art literature

[0010] Patent documents

[0011] Patent Document 1: Japanese Patent No. 6476358 Summary of the Invention

[0012] The problem the invention aims to solve

[0013] However, in the existing technology, when a robot is tasked with a specific operation, a user who is not skilled in operating the robot sometimes finds it difficult to make the robot perform the task with high precision compared to a user who is skilled in operating the robot.

[0014] Furthermore, in existing technologies, in order to perform various tasks using remotely operated robots, operators are required to have a high level of operational skills while accurately grasping the status of the robot and objects at remote locations. As a result, the number of tasks that can be performed is limited, and its efficiency is reduced.

[0015] Therefore, in the existing technology, when operating remotely, it is difficult for the operator to determine and accurately control the target values ​​of the six degrees of freedom (position, posture) in space and to perform the operation.

[0016] Furthermore, there is an inverse (compromise) relationship between improving the tracking ability towards the control target and the flexibility or stability of the solution calculated by inverse kinematics. According to inverse kinematics calculations, for the control target as a whole robot, there may not be a single control value that becomes a solution, and there may be multiple solutions. Moreover, even if a solution exists, it may not be stable. On the other hand, various tasks are instructed to the robot through manipulation. Generally, the balance between motion characteristics such as flexibility, smoothness, and tracking ability regarding the expected action according to the task may vary depending on its relationship with the stability of the solution. For example, if a balance between tracking ability and other motion characteristics such as flexibility and smoothness is predetermined, tracking ability can be obtained even if it is not needed. Instead, even if a stable solution is obtained, motion characteristics such as flexibility and smoothness may sometimes be sacrificed. Conversely, sometimes the flexibility of the action is obtained at the expense of motion characteristics such as tracking ability. This means that it is difficult to make settings suitable for all tasks. Thus, the inability to obtain the expected action through manipulation may become a major reason for the reduced work efficiency of robots.

[0017] Furthermore, in robotic systems, it is crucial for the robot to follow the operator's movements without delay in order to achieve work efficiency and reduce workload. However, in robot control and motion information communication based on operator actions, delays cannot be completely eliminated. On the other hand, the delay in robot movements (following delay) causes discomfort to the operator, becomes a major cause of decreased work speed and work failures, and ultimately leads to a decrease in work efficiency.

[0018] The present invention was made in view of the above-mentioned problems, and one of its objectives is to provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a storage medium that enable even users who are not skilled in operating robots to perform tasks with high precision.

[0019] The present invention was made in view of the above-mentioned problems, and one of its objectives is to provide a robot remote operation control device, a robot remote operation control system, a robot remote operation control method, and a storage medium that are easy for operators to perform operations on.

[0020] The present invention was made in view of the above points, and one of its objectives is to provide a control device, robot system, control method, and storage medium that can improve work efficiency.

[0021] The present invention was made in view of the above points, and one of its objectives is to provide a control device, robot system, control method, and storage medium that can improve the sense of operation.

[0022] Solution for solving the problem

[0023] To address the aforementioned problems, the present invention employs the following solution.

[0024] (1) A robot remote operation control device according to one aspect of the present invention includes an information acquisition unit, an intention inference unit, and a gripping method determination unit. In robot remote operation control in which an operator remotely operates a robot capable of gripping an object, the information acquisition unit acquires operator state information of the operator operating the robot, the intention inference unit infers the operator's intention to perform an action on the robot based on the operator state information, and the gripping method determination unit determines the gripping method of the object based on the inferred intention of the operator's action.

[0025] (2) In the above scheme (1), the intention inference unit can also classify the operator's posture based on the operator's state information, thereby determining the classification of the robot's posture and inferring the operator's action intention.

[0026] (3) In the above scheme (1), the intention inference unit can infer the holding method of the object to be held and at least one of the objects to be held based on the operator's state information, thereby inferring the operator's action intention.

[0027] (4) In the above scheme (1), the intention inference unit can infer the method of holding the object to be held based on the operator's state information, and infer the object to be held based on the inferred method of holding the object to be held, thereby inferring the operator's intention to act.

[0028] (5) In the above scheme (1), the intention inference unit can infer the method of holding the object to be held based on the operator's state information, and infer the object to be held based on the inferred method of holding the object to be held, thereby inferring the operator's intention to act.

[0029] (6) In any of the above schemes (1) to (5), the operator status information may be at least one of the operator's line of sight information, the operator's arm movement information, and the operator's head movement information.

[0030] (7) In any of the above schemes (1) to (6), the information acquisition unit acquires the position information of the object, and the holding method determination unit also uses the acquired position information of the object to infer the object to be held.

[0031] (8) In any of the above schemes (1) to (7), the gripping method determination unit obtains the position information of the gripping part of the robot and corrects the position information of the gripping part based on the operator's state information.

[0032] (9) In the above scheme (8), it is also possible to further include a robot state image production unit, wherein the intention inference unit obtains information related to the object based on the image captured by the shooting device, and the robot state image production unit generates an image to be provided to the operator based on the information related to the object, the position information of the gripping part, the operator state information, and the corrected position information of the gripping part.

[0033] (10) A robot remote operation control system according to one aspect of the present invention comprises: a robot having a gripping part for gripping the object and a detection part for detecting position information of the gripping part; the robot remote operation control device of any of the above aspects (1) to (9); an environmental sensor for detecting position information of the object; and a sensor for detecting operator state information of the operator operating the robot.

[0034] (11) In a robot remote operation control method of the present invention, in which an operator remotely operates a robot capable of holding an object, an information acquisition unit acquires operator state information of the operator operating the robot, an intent inference unit infers at least one of the object to be held and the holding method based on the operator state information, and a holding method determination unit determines the holding method of the object based on the inference result.

[0035] (12) In a remote operation control of a robot in which an operator remotely operates a robot capable of holding an object, the program of one aspect of the present invention causes the computer to perform the following processing: obtain operator state information of the state of the operator operating the robot, infer at least one of the object to be held and the holding method based on the operator state information, and determine the holding method of the object based on the inference result.

[0036] (13) A robot remote operation control device according to one aspect of the present invention includes an intent inference unit and a control command generation unit. In robot remote operation, the intent inference unit infers the operator's movement based on robot environment sensor values ​​obtained from environmental sensors installed on the robot or the robot's surrounding environment and operator sensor values ​​obtained from operator sensors as the operator's movement. The control command generation unit generates appropriate control commands for a portion of the operator's degrees of freedom based on the inferred operator's movement, thereby reducing the degrees of freedom of the operator's movement and generating control commands.

[0037] (14) In the above scheme (13), the control instruction generation unit may also restrict the degree of freedom that the operator should control and the range that can be controlled, and provide action assistance for the restricted degree of freedom in the operator's action instructions to the robot.

[0038] (15) In the above scheme (13) or (14), the control command generation unit does not reduce the degree of freedom of the operator's actions when the distance between the gripping part of the robot and the object based on the operator's operation is outside the specified range, and the control command generation unit reduces the degree of freedom of the operator's actions when the distance between the gripping part of the robot and the object based on the operator's operation is within the specified range, and the robot environment sensor value has captured image information and depth information.

[0039] (16) In any of the above schemes (13) to (15), the intention inference unit may input the robot environment sensor values ​​and the operator sensor values ​​into the learned intention inference model to infer the operator's actions.

[0040] (17) In any of the above schemes (13) to (16), the operator sensor value may be at least one of the operator's gaze information and information related to the posture and position of the operator's arm, i.e., operator arm information.

[0041] (18) In any of the above schemes (13) to (17), the robot environment sensor values ​​may have captured image information and depth information.

[0042] (19) A robot remote operation control system according to one aspect of the present invention includes a gripping part for gripping an object, the robot remote operation control device described in any of the above (13) to (18), an environmental sensor, and an operator sensor. In robot remote operation, the environmental sensor is set on the robot or the robot's surrounding environment to detect robot environmental sensor values, and the operator sensor detects the operator's movement as an operator sensor value.

[0043] (20) In a robot remote operation control method according to one aspect of the present invention, in a robot remote operation in which the operator's movement is identified and the operator's movement is transmitted to the robot to operate the robot, the intention inference unit infers the operator's movement based on the robot environment sensor value obtained by the environment sensor installed on the robot or the robot's surrounding environment and the operator sensor value as the operator's movement obtained by the operator sensor, and the control command generation unit generates appropriate control commands for a portion of the operator's movement based on the inferred operator's movement, thereby reducing the degree of freedom of the operator's movement and generating control commands.

[0044] (21) In a robot remote operation in which the operator's movement is identified and transmitted to the robot to operate the robot, the program of one aspect of the present invention causes the computer to perform the following processing: inferring the operator's movement based on robot environment sensor values ​​obtained by environmental sensors located on the robot or the robot's surrounding environment and operator sensor values ​​obtained by operator sensors as the operator's movement; and generating appropriate control instructions for a portion of the operator's degrees of freedom based on the inferred operator's movement, thereby reducing the degrees of freedom of the operator's movement and generating control instructions.

[0045] (22) A control device according to one aspect of the present invention comprises: an action state inference unit that infers the action state of the robot based at least on environmental information representing the robot's action environment and operation information representing the operation state; a control command generation unit that generates control commands for causing the robot's actuator to perform actions based on the operation information; and a drive control unit that controls the robot's actions based on the control commands, wherein the control command generation unit determines the control commands based on characteristic parameters related to control characteristics corresponding to the action state.

[0046] (23) In the above scheme (22), the action state inference unit may also infer the action state based on operator information, which represents the status of the operator operating the robot.

[0047] (24) In the above scheme (22) or (23), there may also be a target position inference unit, which infers the target position of the actuator based at least on the operation information and the environment information.

[0048] (25) In the above scheme (24), the control command generation unit may determine the amount of operation to drive the actuator toward the target position based on the characteristic parameters. The characteristic parameters include convergence determination parameters, which represent the convergence determination conditions for convergence toward the target position.

[0049] (26) In the above scheme (25), the control command generation unit may determine the operation quantity based on the objective function, which represents the load for causing the actuator to move toward the target position. The objective function is a function obtained by synthesizing multiple factors, and the characteristic parameters include the weight of each factor.

[0050] (27) In the above scheme (25) or (26), the drive control unit may determine the operation amount based on the characteristic parameters in order to reduce the deviation between the target value and the output value, wherein the target value is the target value based on the control command, the output value is the output value output from the action mechanism that drives the actuator, and the characteristic parameters include the gain of the deviation relative to the operation amount.

[0051] (28) The program of one aspect of the present invention enables the computer to function as a control device in any of the above-described aspects (22) to (27).

[0052] (29) A robot system according to one aspect of the present invention includes a control device and a robot according to any of the above-described aspects (22) to (27).

[0053] (30) A control method of one aspect of the present invention is a control method in a control device, wherein the control device performs: a first step, in which the robot's motion state is inferred based at least on environmental information representing the robot's motion environment and operation information representing the operation state; a second step, in which control instructions for causing the robot's actuators to move are generated based on the operation information; and a third step, in which the robot's motion is controlled based on the control instructions, wherein the second step determines the control instructions based on characteristic parameters, the characteristic parameters being related to control characteristics corresponding to the motion state.

[0054] (31) A control device according to one aspect of the present invention comprises: a trajectory prediction unit, which determines the predicted trajectory of the robot's actuator from the current time to a predicted time after a predetermined prediction time based at least on motion information representing the robot's action and operation information representing the operation status; and a control command generation unit, which generates control commands based on the predicted trajectory.

[0055] (32) In the above scheme (31), there may also be an action state inference unit, which infers the action state of the robot based at least on the environmental information representing the robot's action environment and the operation information, and the trajectory prediction unit determines the prediction time based on the action state.

[0056] (33) In the above scheme (32), there may also be a drive control unit that determines the amount of operation of the robot's motion mechanism based on a target value, wherein the target value is a target value that gives the robot's motion mechanism a displacement at the target position of the actuator forming the predicted trajectory at each time, and the motion state inference unit determines the gain of the target value based on the motion state.

[0057] (34) In the above scheme (33), the drive control unit can also synthesize the first component and the second component to determine the operation amount. The first component is obtained based on the deviation between the output value of the displacement given to the current position of the actuator and the target value and the first gain. The second component is obtained based on the target value and the second gain. The action state inference unit determines the first gain and the second gain based on the action state.

[0058] (35) In any of the above schemes (32) to (34), the action state inference unit may also infer the action state based on operator information, which indicates the status of the operator operating the robot.

[0059] (36) The program of one aspect of the present invention enables the computer to function as a control device in any of the above-described aspects (31) to (34).

[0060] (37) A robot system according to one aspect of the present invention includes a control device and the robot of any of the above-described aspects (31) to (34).

[0061] (38) One aspect of the present invention is a control method in a control device, wherein the control method comprises: a first step, in which a predicted trajectory of the robot's actuator is determined from the current time to a predicted time after a predetermined predicted time, based at least on motion information representing the robot's action and operation information representing the operation status; and a second step, in which a control command is generated based on the predicted trajectory.

[0062] The effects of the invention

[0063] According to the above schemes (1) to (12), even if the operator does not perform accurate alignment, the object can be picked up. Therefore, even users who are not familiar with the operation of the robot can make the robot perform tasks with high precision.

[0064] According to the above schemes (2) to (5), the operator's intention is inferred by the movement of the operator's arm, including the hand and fingers, thereby enabling the operator's intention to be inferred with high accuracy.

[0065] According to the above scheme (8), the position information of the gripping part is corrected based on the actual position of the robot's gripping part and the operator's state. Therefore, even if the operator does not perform accurate alignment, the object can still be picked up.

[0066] According to the above scheme (9), an image generated based on the corrected position information of the gripping part can be provided to the operator, so the operator can easily operate the robot remotely.

[0067] According to the above schemes (13) to (21), control target values ​​are generated by replacing a portion of the six degrees of freedom based on the situation, thereby limiting the degrees of freedom that the operator should control, thus making it easier for the operator to perform the work.

[0068] According to the above schemes (22), (28), (29), and (30), the actuator's actions are controlled using operational information based on the action status inferred from the action environment and operational conditions. Since the actuator is operated according to the action status, the robot's work efficiency is improved.

[0069] According to the above scheme (23), the action status is also accurately inferred by referring to the operator's condition. Therefore, the robot's work efficiency is further improved.

[0070] According to the above scheme (24), the robot's motion is controlled to move the actuator toward a target position determined based on the motion environment and operating conditions. The operator does not need to perform operations to accurately indicate the target position, thus further improving the robot's operating efficiency.

[0071] According to the above scheme (25), the convergence criterion for determining whether the actuator's position converges to the target position is determined based on the action status. Therefore, the accuracy of the required or expected position or the stability of the solution can be achieved based on the action status.

[0072] According to the above scheme (26), the weight of the factors related to the load of the actuator's action is determined based on the action status. Therefore, the action characteristics can be adjusted according to the action status, thereby reducing the factors of the required or expected types.

[0073] According to the above scheme (27), the gain of the deviation between the target value and the output value relative to the operation quantity is adjusted according to the action status. Since the speed at which the actuator moves to the target position can be adjusted according to the action status, the operation using the robot becomes more efficient.

[0074] According to the above schemes (31), (36), (37), and (38), the robot's actuators are driven according to control commands generated based on the actuators' predicted trajectories up to a predicted time later than the current time. Therefore, the delay before the operation is reflected in the robot's actions is reduced or eliminated. For the operator, the sense of operation is improved, thus simultaneously achieving increased work efficiency and reduced workload.

[0075] According to the above scheme (32), the prediction time is determined based on the robot's motion status inferred from the robot's motion environment and operating conditions. Therefore, a balance is struck between improving the sense of control and balancing the accuracy of the controlled actuator's position, based on the motion status.

[0076] According to the above scheme (33), the contribution of the target value to the operation amount is adjusted according to the action status, and the operation amount is the operation amount for the motion mechanism. Therefore, the sensitivity of the actuator's action to the operator's operation is adjusted according to the action status.

[0077] According to the above scheme (34), the balance between feedback and feedforward terms is adjusted based on the action status. Therefore, the balance between the sensitivity and accuracy of the actuator's action to the operator's operation is adjusted based on the action status.

[0078] According to the above scheme (35), the operator's condition is also taken into account to accurately infer the action status. Therefore, the robot's work efficiency and workload are further improved. Attached Figure Description

[0079] Figure 1 This is a diagram illustrating an outline of the robot remote operation control system according to an embodiment and an overview of its operation.

[0080] Figure 2 This is a block diagram illustrating a structural example of a robot remote operation control system according to an embodiment.

[0081] Figure 3 This is a diagram showing an example of an operator wearing an HMD and controller on their body.

[0082] Figure 4 This is a diagram illustrating an example of the processing steps of a robot and a robot remote operation control device according to an embodiment.

[0083] Figure 5 This is a diagram illustrating a state where three objects are placed on a table and the operator attempts to make the robot hold object obj3 with its left hand.

[0084] Figure 6 This is a flowchart illustrating a processing example of a robot remote operation control device according to an implementation method.

[0085] Figure 7 This is a diagram illustrating an example of a robot status image displayed by an HMD in an embodiment.

[0086] Figure 8 This is a diagram illustrating an outline of the robot remote operation control system according to an embodiment and an overview of its operation.

[0087] Figure 9 This is a block diagram illustrating a structural example of a robot remote operation control system according to an embodiment.

[0088] Figure 10 This is a diagram showing an example of an operator wearing an HMD and controller on their body.

[0089] Figure 11 This is a diagram illustrating an outline of the intent deduction and control command generation process of the implementation method.

[0090] Figure 12 This diagram illustrates a situation where the operator intends to open the cap of a plastic bottle.

[0091] Figure 13 This diagram illustrates a situation where the operator's intention is to attempt to grab the box.

[0092] Figure 14 This is a diagram illustrating an example of information stored in the storage unit of an embodiment.

[0093] Figure 15 This is a diagram illustrating an example of correction of the fingertip target in the gripping position based on the assistance of a robot remote operation control device.

[0094] Figure 16 This is a flowchart illustrating the processing steps of a robot and a robot remote operation control device according to an implementation method.

[0095] Figure 17This is a schematic block diagram illustrating a structural example of a robot system according to the third embodiment.

[0096] Figure 18 This is a block diagram illustrating a functional structure example of a part of the control device according to the third embodiment.

[0097] Figure 19 This is a schematic block diagram illustrating an example of the hardware structure of the control device according to the third embodiment.

[0098] Figure 20 This is a flowchart illustrating an example of motion control processing according to the third embodiment.

[0099] Figure 21 This is a schematic block diagram illustrating a structural example of a robot system according to the fourth embodiment.

[0100] Figure 22 This is a block diagram illustrating a functional structure example of a part of the control device according to the fourth embodiment.

[0101] Figure 23 This is a flowchart illustrating an example of motion control processing according to the fourth embodiment. Detailed Implementation

[0102] <First Implementation>

[0103] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that, in the drawings used in the following description, the scale of each component has been appropriately altered in order to identify the size of each component.

[0104] [summary]

[0105] First, an overview of the tasks and processes performed by the robot's remote operation control system is provided.

[0106] Figure 1 This is a diagram illustrating an overview of the robot remote operation control system 1 of this embodiment and an overview of its operation. (See diagram for example.) Figure 1 In this way, the operator Us wears, for example, an HMD (Head-Mounted Display) 5 and controllers 6 (6a, 6b). Environmental sensors 7a and 7b are installed within the workspace. It should be noted that the environmental sensors 7 can also be mounted on the robot 2. Furthermore, the robot 2 has gripping units 222 (222a, 222b). The environmental sensors 7 (7a, 7b), as described later, include, for example, an RGB camera and a depth sensor. The operator Us remotely operates the robot 2 by moving their hand and fingers while observing the image displayed on the HMD 5 and wearing the controllers 6. Figure 1In the example, operator Us remotely operates robot 2, causing it to hold a plastic bottle obj located on table Tb. It should be noted that during remote operation, operator Us cannot directly visually confirm the actions of robot 2, but can indirectly visually confirm the state of the image on the side of robot 2 through HMD5. In this embodiment, the robot remote operation control device 3 of robot 2 obtains information about the state of the operator operating robot 2 (operator state information), infers the object to be held and the holding method based on the obtained operator state information, and determines the holding method based on the inferred object.

[0107] [Example of the structure of a robot remote operation control system]

[0108] Next, an example of the structure of the robot remote operation control system 1 will be described.

[0109] Figure 2 This is a block diagram illustrating a structural example of the robot remote operation control system 1 according to this embodiment. Figure 2 Thus, the robot remote operation control system 1 includes a robot 2, a robot remote operation control device 3, an HMD 5, a controller 6, and an environmental sensor 7.

[0110] Robot 2 includes, for example, a control unit 21, a drive unit 22, a sound collection unit 23, a storage unit 25, a power supply 26, and a sensor 27.

[0111] The robot remote operation control device 3 includes, for example, an information acquisition unit 31, an intention inference unit 33, a gripping method determination unit 34, a robot status image generation unit 35, a transmission unit 36, and a storage unit 37.

[0112] The HMD5 includes, for example, an image display unit 51, a gaze detection unit 52, a sensor 53, a control unit 54, and a communication unit 55.

[0113] The controller 6 includes, for example, a sensor 61, a control unit 62, a communication unit 63, and a feedback unit 64.

[0114] The environmental sensor 7 includes, for example, an imaging device 71, a sensor 72, an object position detection unit 73, and a communication unit 74.

[0115] It should be noted that the robot remote control device 3 is connected to the HMD5, for example, via a wireless or wired network. The robot remote control device 3 is also connected to the controller 6, for example, via a wireless or wired network. The robot remote control device 3 is connected to the environmental sensor 7, for example, via a wireless or wired network. The robot remote control device 3 is also connected to the robot 2, for example, via a wireless or wired network. It should also be noted that the robot remote control device 3 and the HMD5 can also be connected directly without a network. The robot remote control device 3 and the controller 6 can also be connected directly without a network. The robot remote control device 3 and the environmental sensor 7 can also be connected directly without a network. The robot remote control device 3 and the robot 2 can also be connected directly without a network.

[0116] [Functional Example of a Robot Remote Operation and Control System]

[0117] Next, refer to Figure 1 The functions of the robot remote operation control system are explained.

[0118] The HMD5 displays the robot's status image received from the robot remote control device 3. The HMD5 detects the operator's eye movement, head movement, etc., and sends the detected operator status information to the robot remote control device 3.

[0119] The image display unit 51 displays the robot status image received from the robot remote operation control device 3 under the control of the control unit 54.

[0120] The line-of-sight detection unit 52 detects the operator's line of sight and outputs the detected line-of-sight information (operator sensor value) to the control unit 54.

[0121] Sensor 53, such as an accelerometer or gyroscope, detects the movement and tilt of the operator's head and outputs the detected head movement information (operator sensor value) to the control unit 54.

[0122] The control unit 54 transmits the gaze information detected by the gaze detection unit 52 and the head movement information detected by the sensor 53 to the robot remote operation control device 3 via the communication unit 55. Additionally, the control unit 54 displays the robot's status image transmitted by the robot remote operation control device 3 on the image display unit 51.

[0123] The communication unit 55 receives the robot's status image sent by the robot remote operation control device 3, and outputs the received robot status image to the control unit 54. Under the control of the control unit 54, the communication unit 55 sends gaze information and head movement information to the robot remote operation control device 3.

[0124] The controller 6 is, for example, a tactile data glove worn on the operator's hand. The controller 6 uses sensors 61 to detect orientation, finger movements, and hand movements, and sends the detected operator status information to the robot remote operation control device 3.

[0125] Sensor 61 is, for example, an accelerometer, a gyroscope, or a magnetometer. It should be noted that sensor 61 may have multiple sensors; for example, sensor 61 may use two sensors to track the movement of each finger. Sensor 61 detects information related to the operator's arm posture and position, such as position, finger movement, and hand movement—that is, operator arm information (operator sensor values, operator state information)—and outputs the detected operator arm information to control unit 62. It should be noted that the operator arm information includes fingertip position / posture information, finger angle information, elbow position / posture information, and information obtained by tracking the movement of each part, encompassing information about the entire arm.

[0126] The control unit 62 transmits operator arm information to the robot remote operation control device 3 via the communication unit 63. The control unit 62 controls the feedback unit 64 based on the feedback information.

[0127] The communication unit 63, under the control of the control unit 62, sends line-of-sight information and operator arm information to the robot remote operation control device 3. The communication unit 63 receives feedback information sent by the robot remote operation control device 3 and outputs the received feedback information to the control unit 62.

[0128] Feedback unit 64 provides feedback information to the operator under the control of control unit 62. Based on the feedback information, feedback unit 64 provides sensory feedback to the operator through units such as vibration-inducing units (not shown), air pressure-inducing units (not shown), hand-restraining units (not shown), temperature-sensing units (not shown), and hardness / softness-sensing units (not shown) installed on grip unit 222 of robot 2.

[0129] Environmental sensor 7 is, for example, positioned at a location capable of capturing and detecting the operation of robot 2. It should be noted that environmental sensor 7 can be possessed by robot 2 or installed on robot 2. Alternatively, there can be multiple environmental sensors 7, or... Figure 1This is set up in the working environment and also installed on robot 2. Environmental sensor 7 sends the object's position information (environmental sensor value), the captured image (environmental sensor value), and the detected sensor value (environmental sensor value) to the robot remote operation control device 3. It should be noted that environmental sensor 7 can be a motion capture device, or it can detect the object's position information through motion capture. Alternatively, a GPS receiver (not shown) with a position information transmitter can be installed on the object. In this case, the GPS receiver can also send position information to the robot remote operation control device 3.

[0130] The imaging device 71 is, for example, an RGB camera. It should be noted that the positional relationship between the imaging device 71 and the sensor 72 in the environmental sensor 7 is known.

[0131] Sensor 72 is, for example, a depth sensor. It should be noted that the imaging device 71 and sensor 72 can also be distance sensors.

[0132] The object position detection unit 73 detects the three-dimensional position, size, and shape of objects in the captured image using well-known methods, based on the captured image and the detection results from the sensors. Referring to pattern matching models stored in the object position detection unit 73, the object position detection unit 73 performs image processing (edge ​​detection, binarization, feature extraction, image emphasis processing, image extraction, pattern matching processing, etc.) on the image captured by the imaging device 71 to infer the object's position. It should be noted that when multiple objects are detected from the captured image, the object position detection unit 73 detects the position of each object individually. The object position detection unit 73 transmits the detected object position information (environment sensor values), the captured image (environment sensor values), and the sensor values ​​(environment sensor values) to the robot remote operation control device 3 via the communication unit 74.

[0133] The communication unit 74 sends the object's position information to the robot remote operation control device 3. The communication unit 74 sends the object's position information (environmental sensor value), the captured image (environmental sensor value), and sensor values ​​(environmental sensor value) to the robot remote operation control device 3.

[0134] Without remote operation, the robot 2's actions are controlled by the control unit 21. With remote operation, the robot 2's actions are controlled based on the gripping plan information generated by the robot remote operation control device 3.

[0135] The control unit 21 controls the drive unit 22 based on the gripping method information output by the robot remote operation control device 3. The control unit 21 performs sound recognition processing (sound range detection, sound source separation, sound source localization, noise suppression, sound source confirmation, etc.) on the sound signals collected by the sound collection unit 23. If the sound recognition result includes action instructions for the robot, the control unit 21 can also control the robot 2's actions based on these action instructions in the sound. The control unit 21 performs image processing (edge ​​detection, binarization, feature extraction, image emphasis processing, image extraction, pattern matching, etc.) on the images captured by the environment sensor 7 based on the information stored in the storage unit 25. It should be noted that the data sent by the environment sensor 7 can be, for example, a point cloud with location information. The control unit 21 extracts object-related information (object information) from the captured images through image processing. This object information includes, for example, the object's name and location. The control unit 21 controls the drive unit 22 based on the program stored in the storage unit 25, the sound recognition results, and the image processing results. The control unit 21 outputs the motion status information of the robot 2 to the robot status image generation unit 35. The control unit 21 generates feedback information and sends the generated feedback information to the controller 6 via the robot remote operation control device 3.

[0136] The drive unit 22 drives various parts of the robot 2 (arms, fingers, feet, head, torso, waist, etc.) according to the control of the control unit 21. The drive unit 22 may include actuators, gears, artificial muscles, etc.

[0137] The sound collection unit 23 is, for example, a microphone array equipped with multiple microphones. The sound collection unit 23 outputs the collected sound signals to the control unit 21. The sound collection unit 23 may also have a sound recognition processing function. In this case, the sound collection unit 23 outputs the sound recognition result to the control unit 21.

[0138] Storage unit 25 stores, for example, the control unit 21, the programs, thresholds, etc. used for control. It should be noted that storage unit 25 can also serve as storage unit 37. Alternatively, storage unit 37 can also serve as storage unit 25.

[0139] Power source 26 supplies power to various parts of robot 2. Power source 26 may also include, for example, a rechargeable battery or charging circuit.

[0140] Sensor 27 may be, for example, an accelerometer, a gyroscope, a magnetometer, or encoders for each joint. It should be noted that sensor 27 is installed at each joint and in the head of robot 2. Sensor 27 outputs the detected results to the control unit 21, the intent inference unit 33, the gripping method determination unit 34, and the robot state image generation unit 35.

[0141] The information acquisition unit 31 acquires gaze information and head movement information from HMD5, acquires operator arm information from controller 6, and acquires environmental sensor values ​​(object position information, sensor values, and images) from environmental sensor 7. It then outputs the acquired operator state information to the intention inference unit 33 and the robot state image creation unit 35.

[0142] The intention inference unit 33 infers the operator's action intention based on the information acquired by the information acquisition unit 31. It should be noted that the intention inference unit 33 uses at least one of gaze information, operator arm information, and head movement information to infer the operator's action intention. It should also be noted that the intention inference unit 33 may use environmental sensor values ​​for intention inference. The operator's action intention will be described below.

[0143] The gripping method determination unit 34 determines the gripping method of the object based on the action intention inferred by the intention inference unit 33 and the detection results detected by the sensor 27. It also determines the gripping method by processing the image captured by the imaging device 71. The gripping method determination unit 34 outputs the determined gripping method information to the control unit 21.

[0144] The robot state image generation unit 35 performs image processing (edge ​​detection, binarization, feature extraction, image enhancement, image extraction, clustering, etc.) on the images captured by the imaging device 71. Based on the gripping method information inferred by the gripping method determination unit 34, the image processing results, and the robot 2's motion state information output by the control unit 21, the robot state image generation unit 35 infers the position and movement of the robot 2's hand, infers the operator's hand movement, and generates the robot state image displayed on the HMD5 based on the inferred results. It should be noted that the robot state image may also include information related to the processing attempted by the robot remote operation control device 3, error information, and system state information indicating the system's state.

[0145] The sending unit 36 ​​sends the robot status image generated by the robot status image generation unit 35 to the HMD5. The sending unit 36 ​​obtains the feedback information output by the robot 2 and sends the obtained feedback information to the controller 6.

[0146] Storage unit 37 stores the template used by intention inference unit 33 during inference, the learned model used for inference, etc. Additionally, storage unit 37 temporarily stores sound recognition results, image processing results, and handling method information. Storage unit 37 also stores model images used for comparison in pattern matching processing during image processing.

[0147] [Example of an operator wearing the HMD5 and controller 6]

[0148] Next, an example of the operator wearing the HMD5 and controller 6 on their body will be described.

[0149] Figure 3 This diagram illustrates an example of an operator wearing the HMD5 and controller 6. Figure 3 In this example, the operator Us wears controller 6a on his left hand, controller 6b on his right hand, and HMD5 on his head. It should be noted that... Figure 3 The HMD5 and controller 6 shown are examples, but the wearing method, shape, etc. are not limited to these.

[0150] [Operator Status Information]

[0151] Next, the operator status information acquired by the information acquisition unit 31 will be further explained.

[0152] Operator status information indicates the operator's state. This includes information about the operator's gaze, finger movements and positions, and hand movements and positions.

[0153] HMD5 detects the operator's line of sight information.

[0154] The controller 6 detects information about the movement and position of the operator's fingers and the movement and position of the operator's hand.

[0155] [Example of information inferred from the intention inference unit 33]

[0156] Next, an example of information inferred by the intention inference unit 33 will be explained.

[0157] The intent inference unit 33 infers the operator's action intent based on the acquired operator state information. The operator's action intent includes, for example, the desired task to be performed by the robot 2, the desired task content, and the movement of the hand and fingers at each moment. The intent inference unit 33 classifies the posture of the operator's arm based on the operator sensor values ​​from the controller 6, thereby classifying the posture of the robot 2's arm, including the gripping part 222. Based on the classification results, the intent inference unit 33 infers the operator's desired action intent for the robot. For example, the intent inference unit 33 infers the method of holding an object and the object to be held as the operator's action intent. The task objective includes, for example, holding an object or moving an object. The task content includes, for example, lifting an object while holding it, or moving an object while holding it.

[0158] The intention inference unit 33 infers the operator's intention for action, for example, by using the grasp classification (GRASP Taxonomy) method (see, for example, Reference 1).

[0159] In this embodiment, for example, the operator's state is classified by classifying the posture of the operator or robot 2, i.e., the gripping posture, using a gripping classification method, and the operator's action intention is inferred. The intention inference unit 33 inputs the operator's state information into the learned model stored in the storage unit 37, for example, to infer the operator's action intention. In this embodiment, intention inference is performed by classifying the gripping posture, thereby enabling high-precision inference of the operator's action intention. It should be noted that other methods can also be used for classifying the gripping posture.

[0160] Reference 1; Thomas Feix, Javier Romero, et al., "The GRASP Taxonomy of HumanGrasp Types" IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb. 2016), IEEE, p66-77

[0161] Alternatively, the intention inference unit 33 can also make inferences by comprehensively using gaze and arm movements. In this case, the intention inference unit 33 can also input gaze information, hand movement information, and the position information of objects on the table into the learned model to infer the operator's action intention.

[0162] The intention inference unit 33, for example, infers the object to be held based on the operator's state information. The intention inference unit 33 infers the object to be held based on, for example, gaze information. Next, the intention inference unit 33 infers the operator's hand posture based on the inferred object to be held.

[0163] Alternatively, the intention inference unit 33 may, for example, infer the operator's hand posture based on the operator's state information. Then, the intention inference unit 33 infers the object to be grasped based on the inferred hand posture. For example, if there are three objects on the table, the intention inference unit 33 infers which of the three objects is the candidate to be grasped based on the hand posture.

[0164] In addition, the intention inference unit 33 can also infer the future trajectory of the operator's fingertip based on the operator's state information and the robot 2's state information.

[0165] It should be noted that the intention inference unit 33 can also use the detection results detected by the sensor 27, the results obtained by image processing of the images captured by the environmental sensor 7, etc., to infer the object to be grasped and the position of the object.

[0166] In addition, since the coordinate systems are different in the operator's operating environment and the robot's motion environment, calibration of the operator's operating environment and the robot's motion environment can be performed, for example, when the robot 2 starts.

[0167] In addition, during gripping, the robot remote operation control device 3 can also determine the gripping position based on the gripping force of the robot 2, the friction between the object and the gripping part, and the error of the gripping position during gripping.

[0168] [Example of processing robot 2 and robot remote operation control device 3]

[0169] Next, an example of the processing of robot 2 and robot remote operation control device 3 will be described.

[0170] Figure 4 This is a diagram illustrating an example of the processing steps of the robot 2 and the robot remote operation control device 3 in this embodiment.

[0171] (Step S1) The information acquisition unit 31 acquires line-of-sight information (operator sensor value) and head movement information (operator sensor value) from the HMD5, and acquires operator arm information (operator sensor value) from the controller 6.

[0172] (Step S2) The information acquisition unit 31 acquires the environmental sensor value from the environmental sensor 7.

[0173] (Step S3) The intent inference unit 33 infers the operator's action intent based on the acquired operator sensor values, such as the task content and the object to be grasped. It should be noted that the intent inference unit 33 uses at least one of gaze information, operator arm information, and head movement information to infer the operator's intent. It should also be noted that the intent inference unit 33 may use environmental sensor values ​​to infer the operator's action intent. Next, the grasping method determination unit 34 calculates remote motion commands for the robot 2 based on the inference results.

[0174] (Step S4) The control unit 21 calculates the drive command value for stable grip based on the remote action command value calculated by the robot remote operation control device 3.

[0175] (Step S5) The control unit 21 uses drive command values ​​to control the drive unit 22 to drive the gripping part of the robot 2, etc. After processing, the control unit 21 returns to the processing of step S1.

[0176] It should be noted that, Figure 4 The processing steps shown are one example; robot 2 and robot remote operation control device 3 can also perform the above processing in parallel.

[0177] [Inference results, assignment information]

[0178] Next, use Figures 5-7 An example of the inference results and task information will be explained.

[0179] Figure 5 This diagram illustrates an example of a situation where three objects, obj1 to obj3, are placed on a table, and the operator attempts to make robot 2 hold object obj3 with its left hand.

[0180] In this situation, the robot remote control device 3 needs to determine which of the objects obj1 to obj3 the operator wants the robot 2 to hold. It should be noted that the robot remote control device 3 needs to determine whether the operator is attempting to hold the object with their right hand or left hand.

[0181] Here, we will explain in advance why it is necessary to infer the operator's intentions.

[0182] In remote operation, the world seen by the operator through the HMD5 differs from the actual world seen with their own eyes. Furthermore, even when the operator gives instructions via the controller 6, they are not actually holding the object, thus the perception still differs from the actual situation in the real world. Additionally, a delay occurs between the operator's instructions and the robot 2's actions due to communication and computation time. Moreover, due to the differences in physical structure between the operator and the robot (primarily the hand), even if the operator instructs the movement of their own fingers and the robot accurately tracks it, the robot may not actually be able to hold the object. To eliminate this situation, in this embodiment, the operator's intention is inferred, and the operator's actions are converted into appropriate actions for the robot.

[0183] Thus, in the remote operation of existing systems, it is difficult to identify the actual situation in the real world and to successfully pick up objects. In the remote operation of existing systems, for example, the operator needs to gradually bring the gripping part 222a of robot 2 close to the object to accurately align and grip it.

[0184] In contrast, in this embodiment, the operator's intention is inferred, and the robot 2's actions are controlled based on the inferred result. Thus, even if the operator does not perform accurate alignment, the object can still be picked up.

[0185] Next, examples of processing such as intent inference processing and correction will be explained.

[0186] Figure 6 This is a flowchart of a processing example of the robot remote operation control device 3 in this embodiment.

[0187] (Step S101) The intent inference unit 33 uses the acquired environmental sensor values ​​to perform environmental recognition, for example, to identify three objects obj1 to obj3 placed on a table.

[0188] When an operator remotely manipulates robot 2 to attempt to make it grasp object obj3, the operator typically directs their gaze towards the object they wish to grasp. Therefore, the intent inference unit 33 infers that the target object is object obj3 based on the gaze information included in the operator's state information obtained from HMD5. It should be noted that the intent inference unit 33 can also use head direction and tilt information included in the operator's state information for inference. It should be noted that when multiple objects are involved, the intent inference unit 33 calculates the probability of each object being the target object (probability of target attainment). It should be noted that the intent inference unit 33 calculates the probability, for example, based on gaze information, the distance between the inferred target object and the gripping part of robot 2, and the position and motion (trajectory) of controller 6.

[0189] (Step S102) The intention inference unit 33 compares the acquired operator state information, including arm position and movement (hand position, hand movement (trajectory), finger position, finger movement (trajectory), arm position, arm movement (trajectory)), and head position and movement, with the template stored in the storage unit 37 to classify the actions. Therefore, when attempting to hold object obj3 with the left hand, the operator's action intention and holding method (holding method) are inferred. For example, the holding method determination unit 34 determines the holding method by referring to the template stored in the storage unit 37. It should be noted that the holding method determination unit 34 may also input the above information into the learned model stored in the storage unit 37 to select the holding method. It should be noted that the intention inference unit 33 uses at least one of gaze information, operator arm information, and head movement information to infer the operator's action intention. It should be noted that the intention inference unit 33 may also use environmental sensor values ​​for intention inference.

[0190] (Step S103) The holding method determination unit 34 determines the holding method of the robot 2 based on the inferred operator's intention to move.

[0191] (Step S104) The gripping method determination unit 34 calculates the offset between the position of the operator's hand and fingers and the position of the robot's gripping part. The storage unit 37, for example, stores a delay time in advance, which is obtained by pre-measuring the time required from issuing an instruction to the movement of the drive unit 22. The gripping method determination unit 34 calculates the offset, for example, using the delay time stored in the storage unit 37. Next, the gripping method determination unit 34 corrects the offset between the position of the operator's hand and fingers and the position of the robot's gripping part. The gripping method determination unit 34 calculates the target value of the current action based on the sampling time of the robot control.

[0192] (Step S106) The robot state image generation unit 35 generates the robot state image displayed on the HMD5 based on the results identified and inferred by the intention inference unit 33 and the results calculated by the gripping method determination unit 34. It should be noted that the robot state image also includes information related to the processing attempted by the robot remote operation control device 3, system state information, etc.

[0193] Figure 7 This is a diagram showing an example of a robot status image displayed by the HMD5 in this embodiment.

[0194] Images g11 to g13 correspond to objects obj1 to obj3 placed on the table. The probability of reaching the target in this case is 0.077 for image g11, 0.230 for image g12, and 0.693 for image g13.

[0195] Image g21 shows the actual position of the gripping part of robot 2.

[0196] Image g22 represents the position input by the operator's controller 6.

[0197] Image g23 represents the command position of the gripping part of robot 2 after correction.

[0198] It should be noted that, in Figure 7 In the storage unit 37, shape data (such as CAD (Computer Aided Design) data) of the gripping part of the robot 2 is stored. The robot state image generation unit 35 uses this shape data to generate an image of the gripping part of the robot 2. Furthermore, the robot state image generation unit 35 uses methods such as SLAM (Simultaneous Localization and Mapping) to create the image. Figure 7 Such a robot state image.

[0199] In this way, the operator can visually observe the actual position of the gripping part of the robot 2 (image g21), the position they input (image g22), and the corrected position of the gripping part of the robot 2 (image g23), thus assisting the operator. Furthermore, in this embodiment, robot motion correction based on the operator's intention is performed; for example, the processing attempted by the robot remote operation control device 3 is visually indicated to the operator, thus enabling smooth remote operation. Therefore, according to this embodiment, the position information of the gripping part is corrected based on the actual position of the robot's gripping part and the operator's state, thereby enabling object pickup even if the operator does not perform precise alignment.

[0200] Thus, in this embodiment, the following I to V steps are performed for remote operation.

[0201] I. Object Recognition

[0202] II. Intention inference (e.g., inferring the object being held and its classification based on line of sight and the operator's fingertip trajectory)

[0203] III. Motion correction (e.g., correcting the robot's fingertip trajectory to a gripping position, and selecting a gripping method).

[0204] IV. Stable grip (control of the grip section used to stably grip the object through the selected gripping method)

[0205] V. The HMD is used to display information about the robot model, recognition results, information related to the processing attempted by the robot's remote control device 3, and information related to the system's state.

[0206] Here, we will further explain the motion correction.

[0207] The gripping method determination unit 34 determines, for example, the contact point between the robot 2's fingers and the object, based on the selected action category, object shape, inferred physical parameters such as object friction and weight, and constraints such as the torque that the robot 2 can output. Then, the gripping method determination unit 34 uses the joint angle calculated based on this information as a target value to correct the action.

[0208] Next, the concept of maintaining stability will be explained.

[0209] When the gripping method determination unit 34 performs an action according to the target value, it controls, for example, the joint angle and torque of the fingers in real time, thereby eliminating the error between the target value / parameter estimation value and the value observed from the sensor 27 of the robot 2. Therefore, according to this embodiment, a stable and continuous grip can be achieved without the gripping object falling.

[0210] Therefore, according to this embodiment, object pickup can be achieved even if the operator does not perform precise alignment. As a result, according to this embodiment, even users who are not skilled in operating the robot 2 can make the robot perform tasks with high precision.

[0211] It should be noted that the above example illustrates an instance where robot 2 is equipped with a remote operation control device 3, but it is not limited to this. The remote operation control device 3 may not be equipped with robot 2, or it may be an external device of robot 2. In this case, robot 2 and the remote operation control device 3 can also send and receive various types of information. Alternatively, robot 2 may have some of the functional components of the remote operation control device 3, while the external device has other functional components.

[0212] In addition, the robot 2 mentioned above can be, for example, a two-legged walking robot, a stationary receiving robot, or a work robot.

[0213] Furthermore, the above example illustrates an example of controlling robot 2 remotely, but it is not limited to this. For example, if robot 2 is a bipedal walking robot, the operator can also remotely control robot 2's walking by wearing a controller on their foot. In this case, robot 2 could also detect object information such as obstacles through image processing, and the operator could remotely control it by walking to avoid the obstacles.

[0214] Furthermore, the examples above illustrate the use of HMD5 worn by the operator, but are not limited to this. The detection of gaze information and the provision of a robot status image to the operator could also be achieved through a combination of sensors and image display devices.

[0215] It should be noted that programs for implementing all or part of the functions of robot 2 and robot remote operation control device 3 in this invention can also be recorded on a computer-readable recording medium, allowing the computer system to read and execute the program recorded on the recording medium, thereby performing all or part of the processing performed by robot 2 and robot remote operation control device 3. It should be noted that the term "computer system" here includes hardware such as operating system and external devices. Furthermore, "computer system" also includes systems built on local networks and systems built in the cloud. Additionally, "computer-readable recording medium" refers to removable media such as floppy disks, optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording medium" also includes recording media that retain programs for a fixed period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0216] Alternatively, the program can be transmitted from a computer system storing the aforementioned program in a storage device or the like to another computer system via a transmission medium or through transmission waves in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium capable of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the aforementioned program can also be used to implement a portion of the aforementioned functions. Additionally, it can be a program capable of implementing the aforementioned functions through combination with a program already recorded in a computer system, i.e., a so-called differential file (differential program).

[0217] The present invention has been described above using embodiments, but the present invention is not limited to such embodiments and can be modified and replaced in various ways without departing from the spirit of the present invention.

[0218] <Second Implementation>

[0219] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the scale of each component has been appropriately altered in the drawings used in the following description in order to identify the size of each component.

[0220] [summary]

[0221] First, an overview of the tasks and processes performed by the robot's remote operation control system is provided.

[0222] Figure 8 This is a diagram illustrating the outline of the robot remote operation control system 1001 according to this embodiment and its operation overview. Figure 8In this way, the operator Us wears, for example, an HMD (Head-Mounted Display) 1005 and a controller 1006. Environmental sensors 1007 (1007a, 1007b) are installed in the working environment. It should be noted that environmental sensor 1007c can also be installed on the robot 1002. Environmental sensors 1007 (1007a, 1007b), as described later, include, for example, an RGB camera and a depth sensor. The operator Us remotely operates the robot 1002 by moving their hand and fingers while observing the image displayed on the HMD 1005 and wearing the controller 1006. Figure 8 In the example, operator Us remotely operates robot 1002, causing it to grasp a plastic bottle obj located on table Tb, for example, opening the bottle cap. It should be noted that during remote operation, operator Us cannot directly visually confirm the actions of robot 1002, but can indirectly visually confirm the state of the image on the robot 1002 side through HMD 1005. In this embodiment, based on information obtained from the controller 1006 worn by the operator and information obtained from the environmental sensors 1007, the operator's intention (information related to the object, task content) is inferred. Based on the inferred result, the degrees of freedom that the operator should control are limited to support the operation.

[0223] [Example of the structure of a robot remote operation control system]

[0224] Next, an example of the structure of the robot remote operation control system 1001 will be described.

[0225] Figure 9 This is a block diagram illustrating a structural example of the robot remote operation control system 1001 according to this embodiment. Figure 9 Thus, the robot remote operation control system 1001 includes a robot 1002, a robot remote operation control device 1003, an HMD 1005, a controller 1006, and an environmental sensor 1007.

[0226] The robot 1002 includes, for example, a control unit 1021, a drive unit 1022, a sound collection unit 1023, a storage unit 1025, a power supply 1026, and a sensor 1027.

[0227] The robot remote operation control device 1003 includes, for example, an information acquisition unit 1031, an intent inference unit 1033, a control command generation unit 1034, a robot status image creation unit 1035, a transmission unit 1036, and a storage unit 1037.

[0228] The HMD 1005 includes, for example, an image display unit 1051, a gaze detection unit 1052 (operator sensor), a control unit 1054, and a communication unit 1055. It should be noted that the HMD 1005 may also include a sensor for detecting the tilt of the operator's head, etc.

[0229] The controller 1006 includes, for example, a sensor 1061 (operator sensor), a control unit 1062, a communication unit 1063, and a feedback unit 1064.

[0230] The environmental sensor 1007 may include, for example, a camera 1071, a sensor 1072, and a communication unit 1073.

[0231] It should be noted that the robot remote control device 1003 and HMD 1005 are connected, for example, via a wireless or wired network. The robot remote control device 1003 and controller 1006 are connected, for example, via a wireless or wired network. The robot remote control device 1003 and environmental sensor 1007 are connected, for example, via a wireless or wired network. The robot remote control device 1003 and robot 1002 are connected, for example, via a wireless or wired network. It should also be noted that the robot remote control device 1003 and HMD 1005 can also be directly connected without a network. The robot remote control device 1003 and controller 1006 can also be directly connected without a network. The robot remote control device 1003 and environmental sensor 1007 can also be directly connected without a network. The robot remote control device 1003 and robot 1002 can also be directly connected without a network.

[0232] [Functional Example of a Robot Remote Operation and Control System]

[0233] Next, refer to Figure 8 The functions of the robot remote operation control system are explained.

[0234] HMD1005 displays the robot's status image received from the robot remote operation control device 1003. HMD1005 detects the operator's eye movement, etc., and sends the detected eye movement information (operator sensor values) to the robot remote operation control device 1003.

[0235] The image display unit 1051 displays the robot status image received from the robot remote operation control device 1003 under the control of the control unit 1054.

[0236] The line-of-sight detection unit 1052 detects the operator's line of sight and outputs the detected line-of-sight information to the control unit 1054.

[0237] The control unit 1054 transmits the gaze information based on the gaze information detected by the gaze detection unit 1052 to the robot remote operation control device 1003 via the communication unit 1055. The control unit 1054 displays the robot status image transmitted by the robot remote operation control device 1003 on the image display unit 1051.

[0238] The communication unit 1055 receives the robot's status image sent by the robot remote operation control device 1003, and outputs the received robot status image to the control unit 1054. Under the control of the control unit 1054, the communication unit 1055 sends operator status information to the robot remote operation control device 1003.

[0239] The controller 1006 is, for example, a haptic data glove worn on the operator's hand. The controller 1006 uses sensors 1061 to detect information related to the operator's arm posture and position, such as orientation, finger movements, and hand movements—that is, operator arm information. The detected operator arm information (operator sensor values) is then sent to the robot remote control device 1003. It should be noted that the operator arm information (operator sensor values) includes information about the entire arm, such as fingertip position / posture information, finger angle information, elbow position / posture information, and information obtained by tracking the movements of various parts.

[0240] Sensor 1061 may be, for example, an accelerometer, a gyroscope, or a magnetometer. It should be noted that sensor 1061 may have multiple sensors; for example, sensor 1061 may use two sensors to track the movement of each finger. Sensor 1061 detects information about the operator's arm and outputs this information to the control unit 1062.

[0241] The control unit 1062 transmits operator arm information, based on the detection results detected by the sensor 1061, to the robot remote operation control device 1003 via the communication unit 1063. The control unit 1062 controls the feedback unit 1064 based on the feedback information.

[0242] The communication unit 1063, under the control of the control unit 1062, sends operator arm information to the robot remote operation control device 1003. The communication unit 1063 receives feedback information sent by the robot remote operation control device 1003 and outputs the received feedback information to the control unit 1062.

[0243] The feedback unit 1064 provides feedback information to the operator under the control of the control unit 1062. Based on the feedback information, the feedback unit 1064 provides sensory feedback to the operator through units such as vibration-generating units (not shown), air pressure-generating units (not shown), hand-restraining units (not shown), temperature-sensing units (not shown), and hardness / softness-sensing units (not shown) installed on the gripping part 1222 of the robot 1002.

[0244] Environmental sensor 1007 is, for example, positioned at a location capable of capturing and detecting the operation of robot 1002. It should be noted that environmental sensor 1007 can be possessed by robot 1002 or installed on robot 1002. Alternatively, there can be multiple environmental sensors 1007, or they can be configured as follows: Figure 8 It is set up in the working environment and also installed on the robot 1002. The environmental sensor 1007 sends the captured images and the detected results (environmental sensor values) to the robot remote operation control device 1003.

[0245] The imaging device 1071 is, for example, an RGB camera. The imaging device 1071 transmits the captured image to the robot remote operation control device 1003 via the communication unit 1073. It should be noted that the positional relationship between the imaging device 1071 and the sensor 1072 in the environmental sensor 1007 is known.

[0246] Sensor 1072 is, for example, a depth sensor. Sensor 1072 transmits the detected sensor values ​​to the robot remote operation control device 1003 via communication unit 1073. It should be noted that the imaging device 1071 and sensor 1072 can also be distance sensors.

[0247] The communication unit 1073 sends the captured image and the sensor values ​​detected by the sensor 1072 as environmental sensor information to the robot remote operation control device 1003. It should be noted that the environmental sensor 1007 can also use the captured image and sensor values ​​to detect the position information of objects and send the detection result as environmental sensor information to the robot remote operation control device 1003. It should also be noted that the data sent by the environmental sensor 1007 can be, for example, a point cloud containing position information.

[0248] Without remote operation, the robot 1002's movements are controlled by the control unit 1021. With remote operation, the robot 1002's movements are controlled by the gripping plan information generated by the robot remote operation control device 1003.

[0249] The control unit 1021 controls the drive unit 1022 based on control commands output by the robot remote operation control device 1003. Based on information stored in the storage unit 1025, the control unit 1021 uses images captured by the environmental sensor 1007 and detection results to obtain the three-dimensional position, size, and shape of objects in the captured images using well-known methods. It should be noted that the control unit 1021 can also perform sound recognition processing (sound range detection, sound source separation, sound source localization, noise suppression, sound source confirmation, etc.) on the sound signals collected by the sound collection unit 1023. If the sound recognition results include action instructions for the robot, the control unit 1021 can also control the robot 1002's movements based on these action instructions in the sound. The control unit 1021 generates feedback information and sends it to the controller 1006 via the robot remote operation control device 1003.

[0250] The drive unit 1022 drives various parts (arms, fingers, feet, head, torso, waist, etc.) of the robot 1002 under the control of the control unit 1021. The drive unit 1022 may include actuators, gears, artificial muscles, etc.

[0251] The sound collection unit 1023 is, for example, a microphone array equipped with multiple microphones. The sound collection unit 1023 outputs the collected sound signals to the control unit 1021. The sound collection unit 1023 may also have a sound recognition processing function. In this case, the sound collection unit 1023 outputs the sound recognition result to the control unit 1021.

[0252] Storage unit 1025, for example, stores the control unit 1021's control program, thresholds, etc., and temporarily stores sound recognition results, image processing results, control commands, etc. It should be noted that storage unit 1025 can also serve as storage unit 1037. Alternatively, storage unit 1037 can also serve as storage unit 1025.

[0253] The power supply 1026 supplies power to various parts of the robot 1002. The power supply 1026 may also include, for example, a rechargeable battery or a charging circuit.

[0254] Sensor 1027 may be, for example, an accelerometer, a gyroscope, a magnetometer, or encoders for each joint. It should be noted that sensor 1027 is mounted on each joint and the head of robot 1002. Sensor 1027 outputs the detected results to control unit 1021, intent inference unit 1033, control command generation unit 1034, and robot state image creation unit 1035.

[0255] The robot remote operation control device 1003 infers the operator's intention to move based on the operator detection value detected by the operator sensor (the line of sight detection unit 1052 of HMD1005 and the sensor 1061 of controller 1006) and the environmental sensor value detected by the environmental sensor 1007, and generates control commands for the robot 1002.

[0256] The information acquisition unit 1031 acquires gaze information from the HMD 1005 and operator arm information from the controller 1006, and outputs the acquired gaze information and operator arm information to the intent inference unit 1033 and the robot state image creation unit 1035. The information acquisition unit 1031 acquires environmental sensor information from the environmental sensor 1007, and outputs the acquired environmental sensor information to the intent inference unit 1033 and the robot state image creation unit 1035. The information acquisition unit 1031 acquires the detection results detected by the sensor 1027, and outputs the acquired detection results to the intent inference unit 1033 and the robot state image creation unit 1035.

[0257] The intent inference unit 1033 infers information related to the target object (robot environmental sensor values) (name of the target object, position of the target object, vertical tilt of the target object, etc.) based on environmental sensor information acquired by the information acquisition unit 1031. It should be noted that the intent inference unit 1033, for example, performs image processing (edge ​​detection, binarization, feature extraction, image emphasis processing, image extraction, pattern matching, etc.) on the image captured by the imaging device 1071 of the environmental sensor 1007 based on information stored in the storage unit 1037 to infer the name of the target object. The intent inference unit 1033 infers the position of the target object, vertical tilt angle, etc., based on the image processing results and the detection results of the depth sensor. It should be noted that the intent inference unit 1033 also uses, for example, the operator's gaze information detected by the HMD 1005 to infer the target object. The intent inference unit 1033 infers the operator's action intent based on the gaze information and operator arm information acquired by the information acquisition unit 1031, as well as information related to the target object. It should be noted that the operator's intentions and inference methods will be described later.

[0258] The control command generation unit 1034 generates control commands for holding an object based on the results inferred by the intention inference unit 1033, the detection results detected by the sensor 1027, and the images and detection results captured by the environmental sensor 1007. As described later, the control command generation unit 1034 generates control commands by limiting the degrees of freedom and controllable range that the operator should control, i.e., reducing the operator's degrees of freedom of movement. The control command generation unit 1034 outputs the determined control command information to the control unit 1021.

[0259] The robot status image generation unit 1035 generates the robot status image displayed on the HMD 1005 based on the control command information generated by the control command generation unit 1034.

[0260] The sending unit 1036 sends the robot status image generated by the robot status image generation unit 1035 to the HMD 1005. The sending unit 1036 obtains the feedback information output by the robot 1002 and sends the obtained feedback information to the controller 1006.

[0261] Storage unit 1037 stores the positional relationship between the imaging device 1071 and sensor 1072 of environmental sensor 1007. Storage unit 1037 stores information that limits the degrees of freedom and controllable range of the object to be assisted, i.e., the operator, according to each task. Storage unit 1037 stores model images for comparison in pattern matching processing during image processing. Storage unit 1037 stores the control program for the robot remote operation control device 1003. It should be noted that the program can also reside in the cloud or on a network.

[0262] [Example of an operator wearing the HMD1005 and controller 1006]

[0263] Next, an example of the operator wearing the HMD1005 and controller 1006 will be described.

[0264] Figure 10 This diagram illustrates an example of an operator wearing the HMD1005 and controller 1006. Figure 10 In this example, the operator Us wears controller 1006a on his left hand, controller 1006b on his right hand, and HMD 1005 on his head. It should be noted that... Figure 10 The HMD1005 and controller 1006 shown are examples, and the wearing method, shape, etc. are not limited to these.

[0265] [Example of intent inference and control command generation]

[0266] Next, an overview of intent inference and control command generation processing will be provided.

[0267] Figure 11 This diagram illustrates an outline of the intent deduction and control command generation process in this embodiment.

[0268] like Figure 11 In this way, the robot remote operation control device 1003 obtains line-of-sight information from HMD 1005, operator arm information from controller 1006, environmental sensor information from environmental sensor 1007, and detection results detected by sensor 1027.

[0269] The information obtained from robot 1002 includes the detection values ​​of the motor encoders set in each joint of robot 1002.

[0270] The information obtained from the environmental sensor 1007 includes images captured by the RGB camera and detection values ​​detected by the depth sensor.

[0271] The intent inference unit 1033 infers the operator's intention based on the acquired information and the information stored in the storage unit 1037. The operator's intention may be, for example, the object to be operated on or the content of the operation. The content of the operation may be, for example, opening the cap of a plastic bottle.

[0272] Based on the results inferred by the intent inference unit 1033 and the information obtained, the control command generation unit 1034 restricts the degree of freedom and controllable range that the operator should control, according to the object and the work content. Then, the control command generation unit 1034 generates control commands based on the results inferred by the intent inference unit 1033.

[0273] [Explanation of the degrees of freedom that the operator should control and the limitations on the range that can be controlled]

[0274] Next, use Figure 12 , Figure 13 The limitations on the operator's controllable degrees of freedom and controllable range are explained. It should be noted that... Figure 12 , Figure 13 The xyz axes in this embodiment are the xyz axes in the robot world. The robot remote operation control device 1003 of this embodiment uses the xyz axis in the robot world as a reference to perform the following degree of freedom restriction and fingertip target correction.

[0275] (Example of the first operation)

[0276] Figure 12 This diagram illustrates a situation where the operator intends to open the cap of a plastic bottle.

[0277] The intent inference unit 1033 infers the object Obj1001 based on the operator's fingertip tracking results, the operator's gaze information, the images captured by the environmental sensor 1007, and the detection results. As a result, the intent inference unit 1033 infers that the object Obj1001 is a "plastic bottle" based on the acquired information. Furthermore, the intent inference unit 1033 detects, for example, the vertical direction of the plastic bottle and the tilt of the plastic bottle's vertical direction relative to, for example, the z-axis direction, based on the images captured by the environmental sensor 1007 and the detection results.

[0278] Next, the intent inference unit 1033 infers the operation content based on the tracking results of the operator's fingertips, the operator's gaze information, the images captured by the environmental sensor 1007, and the detection results. As a result, the intent inference unit 1033 infers, based on the acquired information, that the operation content is the action of "opening the cap of a plastic bottle".

[0279] exist Figure 12 In such a situation, in the existing method, the operator needs to remotely operate the gripping part 1221 of the robot 1002 to align it vertically with the plastic bottle and open the cap. Thus, the existing method has the problem of requiring a high level of operational skill from the operator.

[0280] In contrast, in this embodiment, the robot remote operation control device 1003 generates control target values ​​for a portion of the six degrees of freedom based on the situation, thereby defining the degrees of freedom that the operator should control. Figure 12 In this example, the robot remote control device 1003 controls the gripping part 1221 to assist in positioning the gripping part 1221 of the robot 1002 vertically. Therefore, the operator does not need to worry about the tilt of the plastic bottle and can focus on the instruction for the rotation of the cap.

[0281] (Example of the second operation)

[0282] Figure 13 The diagram illustrates a situation where the operator's intention is to attempt to grab the box.

[0283] Based on the information obtained, the intention inference unit 1033 infers that the operation object Obj1002 is a "box".

[0284] Next, based on the information obtained, the intention inference unit 1033 infers that the operation content is the action of "grabbing a box".

[0285] exist Figure 13 In such a case, the robot remote control device 1003, for example, controls the gripping part 1221 to assist in the positioning of the manipulated object Obj 1002 in the xy plane and its rotation about the xyz axes. Thus, the operator can focus on giving instructions regarding the translational movement along the z-axis.

[0286] It should be noted that, Figure 12 , Figure 13 The objects, tasks, and auxiliary content shown are just one example; the objects, tasks, and auxiliary content are not limited to this. The auxiliary content can be anything that corresponds to the objects and tasks.

[0287] Here, an example of the information stored in the storage unit 1037 used for auxiliary operations will be described.

[0288] Figure 14 This diagram illustrates an example of the information stored in the storage unit 1037 of this embodiment. (See diagram below.) Figure 14 In this way, the storage unit 1037 establishes and stores the object, the work content, the degrees of freedom restricting the operator (the enhanced degrees of freedom of robot 1002), and the degrees of freedom that the operator can manipulate. It should be noted that... Figure 14 The example shown is one example; other information can also be linked and stored.

[0289] In addition, since the coordinate systems are different in the operator's operating environment and the robot's motion environment, the operator's operating environment and the robot's motion environment can be calibrated, for example, when the robot 1002 starts.

[0290] In addition, when holding, the robot remote operation control device 1003 can also determine the holding position based on the holding force of the robot 1002 and the friction between the object and the holding part, taking into account the error of the holding position during holding.

[0291] [Adjustment of the fingertip target for holding the position]

[0292] Next, an example of correcting the fingertip target in the gripping position with the assistance of the robot remote operation control device 1003 will be described.

[0293] Figure 15 This diagram illustrates an example of correcting the fingertip target's gripping position with the assistance of a robot remote control device 1003. Figure 15 In the example, the object g1021 being attempted to be grasped is roughly rectangular. When the robot remote control device 1003 assists in making the grasp easier by restricting the degrees of freedom of the operator's input g1011, it changes the fingertip target to a more easily graspable position (corrected position g1012). In changing the fingertip target, for example in... Figure 12When the vertical direction of the object is tilted relative to the z-axis, the angle of the z-axis direction input by the operator is corrected to align with the vertical direction of the plastic bottle, generating control commands for the gripping unit 1221. More specifically, firstly, the control command generation unit 1034 infers the pattern to be assisted for each object. The inference method is performed, for example, by matching with the database stored in the storage unit 1037 or by machine learning. After inferring the assisting pattern, the control command generation unit 1034 performs vector operations (exterior product / inner product) between the coordinate system of the object and the coordinate system of the robot's fingertip, thereby correcting the direction. Alternatively, the control command generation unit 1034 may also include direction correction based on vector operations, for example, by using an end-to-end machine learning method to directly generate correction command values ​​based on sensor information for correction and command value generation. Thus, according to this embodiment, the robot remote operation control device 1003 infers the object and the task content, and then corrects the fingertip target to an easy-to-grip position, making it easy for the operator to give instructions.

[0294] It should be noted that the intention inference unit 1033 or the storage unit 1037 may, for example, include an intention inference model. This model takes into account gaze information, operator arm information, environmental sensor information, and detection results detected by the sensors of the robot 1002, and learns by using the object and the operation content as teaching data. The intention inference model may also be located in the cloud. The intention inference unit 1033 may also input the acquired information into the intention inference model to infer the operator's intention. Alternatively, the intention inference unit 1033 may also input the acquired information into the intention inference model to infer at least one of the object and the operation content.

[0295] Alternatively, the intention inference unit 1033 may use the acquired information to infer the operator's intention through probabilistic reasoning. Alternatively, the intention inference unit 1033 may use the acquired information to infer at least one of the object and the task content through probabilistic reasoning.

[0296] [Example of processing steps for robot 1002 and robot remote operation control device 1003]

[0297] Next, the processing steps of robot 1002 and robot remote operation control device 1003 will be explained.

[0298] Figure 16 This is a flowchart illustrating the processing steps of the robot 1002 and the robot remote operation control device 1003 in this embodiment.

[0299] (Step S1001) The information acquisition unit 1031 acquires line-of-sight information from the HMD 1005 and acquires operator arm information from the controller 1006.

[0300] (Step S1002) The information acquisition unit 1031 acquires environmental sensor information from the environmental sensor 1007.

[0301] (Step S1003) The intent inference unit 1033 infers the operator's intent, including the object being held and the work content, based on the obtained line of sight information, operator arm information, and environmental sensor information.

[0302] (Step S1004) The control command generation unit 1034 determines the restrictions on the degrees of freedom assisted by the robot remote operation control device 1003, i.e., the degrees of freedom of the operator's motion instructions, based on the inferred operator's intention and the information stored in the storage unit 1037. It should be noted that the control command generation unit 1034 generates appropriate control commands for a portion of the operator's motion degrees of freedom, thereby reducing the operator's motion degrees of freedom and generating control commands.

[0303] (Step S1005) The control command generation unit 1034 corrects the fingertip position of the target to be held based on the arm motion information contained in the acquired operator arm information and the restricted degrees of freedom determined in step S1004.

[0304] (Step S1006) The robot status image generation unit 1035 generates the robot status image displayed on the HMD 1005 based on the operator arm information obtained by the information acquisition unit 1031, the result inferred by the intention inference unit 1033, and the gripping position corrected by the control command generation unit 1034.

[0305] (Step S1007) The control command generation unit 1034 generates control commands based on the corrected fingertip position of the target to be held.

[0306] (Step S1008) Based on the control commands generated by the control command generation unit 1034, the control unit 1021 controls the drive unit 1022 to drive the gripping part of the robot 1002. After processing, the control unit 1021 returns to the processing in step S1001.

[0307] It should be noted that, Figure 16 The processing steps shown are one example; the robot remote operation control device 1003 can also perform the above processing in parallel.

[0308] In addition, the image displayed by the image display unit 1051 of the HMD1005 is, for example, Figure 15Such an image. It should be noted that the robot status image may not include the image of the operator's input g1011. It should also be noted that the robot status image generation unit 1035 can, for example, use text prompts or auxiliary patterns, or display rectangles, arrows, etc., on the HMD 1005 to indicate to the operator the content related to the restricted degrees of freedom assisted by the robot. Therefore, when the operator observes the robot status image displayed on the HMD 1005 and gives work instructions, such as when opening a plastic bottle cap, they do not need to worry about aligning the grip with the vertical direction of the plastic bottle, and can focus on the work instructions of opening the cap.

[0309] As described above, in this embodiment, the robot remote operation control device 1003 uses sensor information installed on the robot 1002, the environment, and the operator to determine the task content and the target object. Based on the situation, it generates control target values ​​for a portion of the six degrees of freedom, thereby limiting the degrees of freedom that the operator should control. That is, in this embodiment, by automatically generating appropriate control commands for a portion of the degrees of freedom, the number of degrees of freedom that the operator should control is reduced.

[0310] Therefore, according to this embodiment, in a remotely operated robot, it is possible to simultaneously reduce the operator's burden and diversify and increase the efficiency of the operation.

[0311] It should be noted that the above example of a limitation on degrees of freedom is just one example, and is not limited to this. A limitation on a degree of freedom does not mean that the operator has no control over that degree of freedom at all, and also includes situations where the area that can be operated is restricted.

[0312] For example, the robot remote control device 1003 may not restrict the degrees of freedom when the distance to the object is outside a specified range, but may restrict the degrees of freedom when the distance to the object is within a specified range. As an example, the specified range is a range of ±1001m for x-axis translation.

[0313] It should be noted that the examples above illustrate remote manipulation of objects, but are not limited to this. Even for other tasks, the robot remote control device 1003 restricts the degrees of freedom according to the task content.

[0314] Furthermore, when there are multiple tasks, the robot remote control device 1003 can also change the restrictions on the degrees of freedom each time the tasks change. For example, when multiple objects are placed on a table and a plastic bottle is held and then the cap is opened, the robot remote control device 1003 sets a restriction on the first degree of freedom during the holding phase and a restriction on the second degree of freedom when the cap is opened.

[0315] In addition, when performing tasks using the gripping parts of both parties, the robot remote operation control device 1003 restricts the degrees of freedom and corrects the fingertip target according to each gripping part.

[0316] In addition, the robot 1002 mentioned above can be, for example, a two-legged walking robot, a fixed receiving robot, or a work robot.

[0317] Furthermore, the above example illustrates how robot 1002 can be controlled remotely, but it is not limited to this example.

[0318] Furthermore, the example above illustrates an operator wearing an HMD1005, but it is not limited to this. The detection of gaze information and the provision of a robot status image to the operator could also be achieved through a combination of sensors and image display devices.

[0319] It should be noted that programs for implementing all or part of the functions of the robot 1002 and the robot remote operation control device 1003 in this invention can also be recorded on a computer-readable recording medium, allowing the computer system to read and execute the program recorded on the recording medium, thereby performing all or part of the processing performed by the robot 1002 and the robot remote operation control device 1003. It should be noted that the term "computer system" here includes hardware such as an operating system and external devices. Furthermore, "computer system" also includes systems built on local networks and systems built in the cloud. Additionally, "computer-readable recording medium" refers to removable media such as floppy disks, optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Moreover, "computer-readable recording medium" also includes recording media that retain programs for a fixed period of time, such as volatile memory (RAM) within a computer system that acts as a server or client when the program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0320] Alternatively, the program can be transmitted from a computer system storing the aforementioned program in a storage device or the like to another computer system via a transmission medium or through transmission waves in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium capable of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line. Furthermore, the aforementioned program can also be used to implement a portion of the aforementioned functions. Additionally, it can be a program capable of implementing the aforementioned functions through combination with a program already recorded in a computer system, i.e., a so-called differential file (differential program).

[0321] The present invention has been described above using embodiments, but the present invention is not limited to such embodiments and can be modified and replaced in various ways without departing from the spirit of the present invention.

[0322] <Third Implementation Method>

[0323] Hereinafter, the third embodiment of the present invention will be described with reference to the accompanying drawings.

[0324] Figure 17 This is a schematic block diagram showing a structural example of the robot system S2001 of this embodiment.

[0325] Robot system S2001 is a control system capable of controlling the movements of robot 2020 according to the actions of the user. Furthermore, robot system S2001 is also a manipulation system that manipulates the movements of robot 2020 based on user input. Robot 2020 is equipped with an actuator (also called an end-effector, robotic hand, fingertip actuator, etc.). An actuator is a component that mechanically acts on other objects to produce an effect. The actuator has a mechanism with multiple fingers capable of grasping or releasing other objects. Each finger can move according to the movement of the corresponding finger of the operator. Thus, various tasks can be performed.

[0326] In this embodiment, the robot system S2001 infers the motion state of the robot 2020 based on environmental information representing the robot 2020's motion environment and operational information representing the operational state. As described later, when inferring the motion state, the robot system S2001 may also use drive state information obtained from the drive sensor 2026, operator gaze information, etc. Based on characteristic parameters representing the control characteristics corresponding to the motion state, the robot system S2001 generates control commands to move the actuators of the robot 2020 to a target position. The robot system S2001 controls the motion of the robot 2020 based on the generated control commands.

[0327] In this application, "motion status" primarily refers to the manner in which the robot 2020 performs actions, i.e., the task or motion pattern. Motion status can include elements such as the type of task being attempted, the positional relationship between the robot and the objects involved in the task, and the type or characteristics of the objects. Motion status can also be defined as including any elements or any combination of elements. Even if the operational information is common, the target position to which the actuators of the robot 2020 move may differ depending on the motion status.

[0328] Robot system S2001 includes robot 2020, display device 2050, operating device 2070, and environmental information acquisition unit 2080. Robot 2020 has one or more manipulators. A manipulator is also called a robotic arm. Each manipulator has multiple segments that are interconnected. Each segment includes an actuator. An actuator is a component that is connected to one end of the manipulator and contacts an object to perform a function. The motion mechanism of robot 2020 has joints for each pair of segments, including two segments, and an actuator for each joint. By changing the angle between each pair of segments using the actuator, the position of the actuator can be moved. In the following description, the case with one manipulator (single-arm type) is emphasized, but there can also be two or more. Furthermore, robot 2020 can be a fixed, placed type or a movable type capable of moving its own position. In the following description, the case of robot 2020 being a placed type is emphasized.

[0329] The robot 2020's control device 2030 is connected to the display device 2050, the operating device 2070, and the environmental information acquisition unit 2080 via wireless or wired means, enabling the input and output of various information.

[0330] The display device 2050 and the operating device 2070 may also be located in a space physically separate from the robot 2020 and the environmental information acquisition unit 2080. In this case, the display device 2050 and the operating device 2070 may also be configured as a remote operating system connected to the control device 2030 via a communication network.

[0331] The robot 2020 includes a drive unit 2024, a drive sensor 2026, a power supply 2028, and a control device 2030.

[0332] The drive unit 2024 has an actuator for each joint, and operates according to control commands input from the drive control unit 2040 of the control device 2030. Each actuator is equivalent to a motor, which changes the angle between the two segments connected to it according to the target value of the drive amount indicated by the control command.

[0333] The drive sensor 2026 detects the drive state of the drive unit 2024 driving the robot 2020, and outputs drive state information representing the detected drive state to the control device 2030. The drive sensor 2026 is configured, for example, to include a rotary encoder that detects the angle between two segments for each joint.

[0334] The power supply 2028 supplies power to the various components of the robot 2020. The power supply 2028 includes, for example, power terminals, a secondary battery, and a voltage converter. The power terminals can be connected to power lines to receive power from an external source, supplying the power to the secondary battery or voltage converter.

[0335] The secondary battery stores power supplied by the power supply terminal. The secondary battery can supply power to each component via a voltage converter. The voltage converter converts the voltage of the power supplied from the power supply terminal or the secondary battery into the specified voltage required by each component, and then supplies the converted power to each component.

[0336] Next, refer to Figure 17 and Figure 18 The functional structure of the control device 2030 in this embodiment will be explained. Figure 18 This is a block diagram illustrating a functional structure example of a portion of the control device 2030 according to this embodiment. The control device 2030 includes an information acquisition unit 2032, an operation status inference unit 2034, a target position inference unit 2036, a control command generation unit 2038, a drive control unit 2040, a communication unit 2042, and a storage unit 2044.

[0337] The information acquisition unit 2032 acquires various information related to the operator's condition and the robot's condition 2020. For example, the information acquisition unit 2032 acquires gaze information and first operator action information from the display device 2050. The gaze information and the first operator action information constitute operator information representing the operator's condition. The gaze information is information indicating the direction of the gaze of at least one eye of the operator at a certain moment. The first operator action information is information indicating the movement of the operator's head. The head movement is represented by the direction and position of the head at each moment. In this application, the gaze information and the first operator action information are sometimes collectively referred to as operator information.

[0338] The information acquisition unit 2032 acquires second operator motion information from the operation device 2070. The second operator motion information represents body movements related to the operator's actions. This second operator motion information constitutes operation information, which indicates the operating status of the robot 2020. The second operator motion information includes information representing the movements of the hand, a primary body part of the operator. Hand movements include movements of at least two fingers. In addition to hand movements, the second operator motion information may also include information representing wrist movements. Wrist movements can be represented using a representative position of the wrist, or they may include information about its posture. Body part movements are represented using information about the position of the body part at each moment. In this application, the second operator motion information is sometimes referred to as operation information.

[0339] The information acquisition unit 2032 acquires environmental information from the environmental information acquisition unit 2080. The environmental information represents the operating environment of the robot 2020. The environmental information includes image data and distance data, wherein the image data represents an image showing the operating environment of the robot 2020, and the distance data represents the distance to objects distributed in the operating environment of the robot 2020.

[0340] The information acquisition unit 2032 acquires drive state information from the drive sensor 2026. The drive state information can also be regarded as posture information representing the posture of the robot 2020. The time sequence of the drive state information at each moment is equivalent to robot motion information representing the actions of the robot 2020. The information acquisition unit 2032 stores the acquired information in the storage unit 2044.

[0341] The motion state inference unit 2034 reads the latest environmental and operational information from the storage unit 2044 at the current time, and infers the motion state of the robot 2020 based on the read environmental and operational information. To infer the motion state of the robot 2020, the motion state inference unit 2034 uses, for example, a prescribed machine learning model. In the motion state inference unit 2034, a parameter set learned using training data is pre-set so that when the input environmental and operational information is used as input information, the output information obtained sets the reliability of the known motion state corresponding to the input information to 1 and sets the reliability of other candidate motion states to 0. The training data consists of a group including a large amount of input information and known output information. The parameter set is learned in such a way that, as a whole, the difference between the inferred value of the output information inferred from the input information and the known output information is minimized. In this case, the reliability for each motion state obtained from the machine learning model can be determined as a real value between 0 and 1. The motion state inference unit 2034 can determine the motion state of the robot 2020 as the candidate motion state whose reliability calculated according to each motion state candidate is higher than the specified reliability and becomes the largest. As a motion state, for example, it can determine: approach to an object (target object) among multiple objects in the robot 2020's motion environment that is in the direction closest to the hand's movement direction represented by the operation information at that time, and the manner in which the object in contact with the actuator at that time acts (e.g., grasping, releasing, rubbing, etc.).

[0342] The motion state inference unit 2034 can also read the latest operator information at that point in time from the storage unit 2044. Based on the operator information, in addition to environmental and operational information, it infers the motion state of the robot 2020. The operator information also considers the direction of the operator's head movement. Therefore, when inferring the motion state, the direction of the actions that the operator is interested in is further considered based on the operator information. In the motion state inference unit 2034, a parameter set learned using training data as described above is pre-set so that when operator information is input as input to the machine learning model in addition to environmental and operational information, the output is output information where the reliability of the known motion state corresponding to the input information is 1, and the reliability of the candidates for other motion states is 0.

[0343] It should be noted that the motion state inference unit 2034 can also read the latest drive state information at that point in time from the storage unit 2044, and further infer the motion state of the robot 2020 based on the drive state information. The drive state information represents the posture of the robot 2020 and its temporal changes. Therefore, the posture of the robot 2020 is also considered when inferring the motion state. In the motion state inference unit 2034, a parameter set learned by using training data as described above is preset, so that when drive state information is input as input to the machine learning model in addition to input environment information and operation information, the output is output information in which the reliability of the known motion state corresponding to the input information is 1 and the reliability of the candidate motion state for other motion states is 0.

[0344] The motion state inference unit 2034 determines the characteristic parameters corresponding to the inferred motion state. The characteristic parameters are parameters related to the control characteristics of the actuator's motion. These characteristic parameters may include, for example, any one of the convergence determination parameters, the weight coefficients of each factor of the objective function, or a combination thereof. The storage unit 2044 pre-stores a characteristic parameter table representing the characteristic parameters according to each motion state. The motion state inference unit 2034 refers to the characteristic parameter table, determines the characteristic parameters corresponding to the inferred motion state, and outputs the determined characteristic parameters to the control command generation unit 2038.

[0345] The convergence determination parameter is a parameter representing the convergence determination condition, which is used to determine whether the position of the actuator converges to the target position. The strength of the convergence determination condition is equivalent to the degree of constraint of the converged position (sometimes referred to as the "convergence position" in this application) relative to the target position; in other words, the accuracy relative to the target position. As a convergence determination parameter, a threshold of distance from the target position can be used, for example. The smaller the threshold of distance from the target position, the stronger the convergence determination condition. The motion state inference unit 2034 can also set a convergence determination parameter that indicates a stronger convergence determination condition for motion states where the followership of the actuator's position as the control object is more important. Followership refers to the characteristic that the actuator's position can be accurately controlled relative to the target position, or the characteristic that requires accurate control, i.e., accuracy. Such motion states include, for example, situations where the actuator is placed on a delicate object and is very close to the target position, or situations where the motion indicated by the operation information is subtle.

[0346] Inverse kinematics calculation is an analytical method that determines the displacement of each joint, i.e., one or both of angle and angular acceleration, based on the target position; it is also called inverse kinematics analysis. It should be noted that inverse dynamics calculation is an analytical method that determines the driving force of the joints used to achieve the motion of each joint (i.e., the time change of one or both of angle and angular acceleration) in a linkage mechanism composed of multiple connected segments; it is also called inverse dynamics analysis. Therefore, inverse kinematics calculation is closely related to inverse dynamics calculation. In this embodiment, characteristic parameters related to inverse kinematics calculation sometimes include characteristic parameters related to inverse dynamics calculation.

[0347] On the other hand, the stronger the convergence criteria, the higher the probability that the solution calculated by inverse kinematics does not exist, or even if a solution exists, the probability that the solution is unstable. Therefore, the motion state inference unit 2034 can also set convergence criteria parameters that indicate a weaker convergence condition (e.g., separation from the target position) for motion states where the greater the flexibility of the actuator's position as the control target, the weaker the convergence criteria. Flexibility refers to the characteristic that allows the actuator's position to deviate from the target position. Flexibility refers to the characteristic that allows reliable control, i.e., safety. Motion states that allow flexibility include, for example, situations where multiple objects are distributed around the path to the target position.

[0348] The objective function is used to numerically find the optimal solution (optimization problem) for the robot 2020's motion using known methods in inverse kinematics calculations. That is, the objective function is a quantitative representation of the magnitude of the load required to move the actuator towards the target position, and it comprises multiple factors synthesized from these factors. Weighting coefficients represent the contribution of each factor to the objective function. Load represents the cost associated with control. Among the multiple factors, the magnitude of the control error at each time step is included as a factor. The magnitude of the control error is represented, for example, by the magnitude (distance) between the final target position and the convergence error. The larger the weighting coefficients assigned to the control error, the better the actuator's tracking of the target position. The objective function may also include any one or a combination of the following as factors: the time of arrival from the actuator's current position to the convergence position; the magnitude of the jerk from the current position to the convergence position; the electric force from the current position to the convergence position; the amount of motion of the robot 2020 until reaching the convergence position; and the magnitude of the difference between the current pose of the robot 2020 and the pose of the robot 2020 at the time the actuator is at the convergence position. In this application, optimization includes exploring solutions that minimize the objective function value, and is not limited to minimizing the function value absolutely. Therefore, the function value may also temporarily increase.

[0349] Therefore, the motion state inference unit 2034 can also be configured such that the more a motion state requires the actuator to follow the position of the controlled object, the larger the weighting coefficient for control error. The motion state inference unit 2034 can also be configured such that the more a motion state allows for flexibility in the position of the controlled actuator, the smaller the weighting coefficient for control error. The motion state inference unit 2034 can also be configured such that the more a motion state requires continuity in the position of the controlled actuator, the larger the weighting coefficient for the magnitude of jerk.

[0350] The motion state inference unit 2034 determines the drive control parameters corresponding to the inferred motion state. These drive control parameters are parameters related to the control characteristics of the drive for each segment constituting the robot 2020. The drive control parameters are used by the controller forming the drive control unit 2040 when determining the operating amount of the actuators driving each joint. The storage unit 2044 stores a drive control parameter table representing the drive control parameters according to each motion state. The motion state inference unit 2034 refers to the drive control parameter table, determines the drive control parameters corresponding to the inferred motion state, and outputs the determined drive control parameters to the drive control unit 2040.

[0351] For example, when the controller is a PID (Proportional-Integral-Differential) controller, the drive control parameters include proportional gain, integral gain, and differential gain. Proportional gain is the gain calculated by multiplying the proportional term (one component of the operational variable) by the deviation between the target value and the output value at that point in time. Integral gain is the gain calculated by multiplying the integral term (another component of the operational variable) by the integral value of the deviation up to that point in time. Differential gain is the gain calculated by multiplying the differential term (another component of the operational variable) by the differential value of the deviation at that point in time. The operation condition inference unit 2034 may, for example, determine that the differential gain is relatively larger than other types of gains for operation conditions requiring greater following ability. The operation condition inference unit 2034 may also determine that the integral gain is relatively larger than other types of gains for operation conditions requiring greater flexibility.

[0352] The target position inference unit 2036 reads the latest environmental and operational information at that point in time, and infers the target position of the actuators of the robot 2020 based on the read environmental and operational information. The target position inference unit 2036 outputs the target position information representing the inferred target position to the control command generation unit 2038.

[0353] In order to infer the target position, the target position inference unit 2036 uses, for example, a machine learning model. To distinguish it from the machine learning model in the motion state inference unit 2034, the machine learning model in the motion state inference unit 2034 and the machine learning model in the target position inference unit 2036 are sometimes referred to as the first machine learning model and the second machine learning model, respectively.

[0354] In the target position inference unit 2036, a learned parameter set is preset so that when environmental information and operational information are input to the machine learning model, output information related to the target position corresponding to the input information is obtained.

[0355] The target position is set as a location on the surface of the object that, empirically, is most likely to be acted upon by the actuator. The target position can depend on the shape of the object. For a long cylinder with a height greater than its diameter, the position closest to the actuator among the cross-sections of the center of the base, the center of the surface, and the midpoint in the height direction is more likely to be the target position. The output information from the machine learning model can also be information representing the coordinates of the target position, and is not limited to this. The output information may, for example, include information such as the object for which the target position is set, the type, shape, and orientation of the object, and the position of the target position within that object. The target position inference unit 2036 can determine the coordinates of the target position based on this output information.

[0356] Furthermore, the target location inference unit 2036 can also read the latest operator information at that time point from the storage unit 2044, and infer the target location based on the operator information in addition to the environmental information and operation information. Therefore, it infers the target location based on the operator information and also considering the direction in which the operator's interest was generated. However, in the target location inference unit 2036, a learned parameter set is pre-set so that when the machine learning model is fed with operator information as input in addition to environmental information and operation information, the coordinates of the target location corresponding to that input information are obtained as output information.

[0357] However, the target position inference unit 2036 may not necessarily use the second machine learning model to infer the target position. For example, in the case where the motion state notified by the motion state inference unit 2034 is a motion state without continuous translational movement of the actuator, the target position inference unit 2036 does not use the second machine learning model to infer the target position. Such motion states include, for example, stationary, rotating, grasping or releasing an object, and friction on the surface of an object. Even without using the second machine learning model to infer the target position, the target position inference unit 2036 may use the position of the operator's hand shown in the operation information as the target position.

[0358] It should be noted that, as the first machine learning model and the second machine learning model, the target position inference unit 2036 can utilize mathematical models such as neural networks and random forests.

[0359] The control command generation unit 2038 generates motion commands to move the actuator from its current position toward the target position indicated by the target position information input from the target position inference unit 2036. "Towards the target position" or "towards the target position" implies movement toward the target position as an objective, but does not guarantee reaching the target position. The control command generation unit 2038 can read the drive state information stored in the storage unit 2044 and calculate the actuator's current position based on the angles of each joint and the dimensions of each segment indicated by the drive state information. In addition to the angles of each joint and the dimensions of each segment indicated by the drive state information, the control command generation unit 2038 can also calculate the actuator's speed based on the angular velocity of each joint. Based on the characteristic parameters input from the motion state inference unit 2034 and the drive state information (robot motion information), the control command generation unit 2038 performs known inverse kinematics calculations to determine the angles of each joint, causing the actuator to move toward a convergent position. The control command generation unit 2038 outputs motion commands (joint commands) representing the determined angles (joint angles) of each joint to the drive control unit 2040.

[0360] When the characteristic parameters include convergence determination parameters, the control command generation unit 2038 can also determine any position within the region shown by the convergence determination parameters from the target position as the convergence position during inverse kinematics calculation, and stop the movement of the actuator 2038. When determining whether the actuator's current position has converged, the control command generation unit 2038 determines, for example, whether the actuator's current position is within a threshold value of the distance specified by the characteristic parameters from the target position. When it is determined that the actuator's position has converged, the control command generation unit 2038 generates a control command indicating that the movement has stopped and outputs it to the drive control unit 2040. When the drive control unit 2040 receives a control command indicating that the movement has stopped from the control command generation unit 2038, it stops updating the target values ​​for each joint and fixes them. At the convergence position, the control command generation unit 2038 can also apply a constraint condition that makes the actuator's velocity zero, or the drive control unit 2040 can apply a constraint condition that makes the angular velocity of each joint zero.

[0361] When the convergence characteristic parameters include weight coefficients for each factor of the objective function, the control command generation unit 2038 performs optimization calculations in the inverse kinematics calculation based on the objective function synthesized using the factors corresponding to their weight coefficients. The objective function is calculated, for example, by summing (weighted sum) the factors whose products are their weight coefficients. In the optimization calculation, the control command generation unit 2038 may also recursively calculate the angles of each joint to minimize the objective function representing the load generated until the actuator reaches the target position or convergence position.

[0362] It should be noted that if the target position inference unit 2036 does not obtain target position information, the control command generation unit 2038 may also use the hand position shown in the operation information as the target position. The control command generation unit 2038 performs inverse kinematics calculations to generate motion commands that use the angles of each joint used to move the actuator from the current position toward the adopted target position as the target values.

[0363] The drive control unit 2040 determines the drive command based on the control command input from the control command generation unit 2038, the drive control parameters input from the motion state inference unit 2034, and the drive state information (robot motion information) read from the storage unit 2044. More specifically, the drive control unit 2040 is configured to include a controller that calculates the operation quantity for controlling the angle of each joint indicated by the control command to a target value, and determines the control command representing the calculated operation quantity. The drive control unit 2040 outputs the determined control command to the drive unit 2024. The controller uses the angle represented by the drive state information as the output value, and calculates the operation quantity by applying drive control parameters to the deviation between the target value and the output value according to the control mode set in the controller. The PID controller calculates the operation quantity by multiplying the deviation, the time integral value of the deviation, and the time derivative value of the deviation by the proportional gain, integral gain, and derivative gain, respectively.

[0364] The communication unit 2042 transmits and receives various information wirelessly or via wired means between the various components of the robot 2020 and other components (i.e., the display device 2050, the operating device 2070, and the environmental information acquisition unit 2080). The communication unit 2042 is configured, for example, to include an input / output interface, a communication interface, etc.

[0365] The storage unit 2044 temporarily or permanently stores data used for various processes in the control device 2030, as well as various data acquired by the control device 2030. The storage unit 2044 is configured, for example, to include storage media such as RAM (Random Access Memory) or ROM (Read Only Memory).

[0366] The display device 2050 is configured to include a display unit 2052, a gaze detection unit 2054, a motion detection unit 2056, a control unit 2058, and a communication unit 2060. The display device 2050 may also include a support member that can be worn on the head of the user, thus being configured as a head-mounted display (HMD).

[0367] Display unit 2052 receives image data from environmental information acquisition unit 2080 via control device 2030 using communication unit 2060. Based on the received image data, display unit 2052 displays an image representing the operating environment of robot 2020. When worn on operator's head, the display screen of display unit 2052 can also be positioned facing directly at the eyes.

[0368] The gaze detection unit 2054 is configured to include a gaze sensor that detects the gaze direction of one or both of the operator's eyes. The gaze detection unit 2054 uses a communication unit 2060 to transmit gaze information indicating the gaze direction detected at each moment to a control device 2030. When worn on the operator's head, the gaze detection unit 2054 can also be positioned so that at least one eye is exposed.

[0369] The motion detection unit 2056 detects the operator's head movements and, using the communication unit 2060, sends first operator motion information indicating the detected movements to the control device 2030. The motion detection unit 2056 is configured, for example, to include an accelerometer for detecting the operator's movements.

[0370] The control unit 2058 controls the functions of the display device 2050. The control unit 2058 is configured to include a processor such as a CPU (Central Processing Unit).

[0371] The communication unit 2060 and the control device 2030 transmit and receive various information. The communication unit 2060 is configured to include a communication interface.

[0372] The operating device 2070 includes a motion detection unit 2072, a control unit 2074, and a communication unit 2076. The operating device 2070 may also include a support member that can be worn on the hand of the operator, thus forming a data glove.

[0373] The motion detection unit 2072 detects the operator's hand movements and, using the communication unit 2076, sends second operator motion information, representing the detected movements, to the control device 2030. The motion detection unit 2072 is configured, for example, to include an accelerometer for detecting the operator's hand movements. A wrist tracker may also be further connected to the operating device 2070. The wrist tracker is configured to include a support member that can be worn on the operator's wrist and an accelerometer for detecting wrist movements. The accelerometer for detecting wrist movements is part of the motion detection unit 2072. The second operator motion information, including information representing wrist movements, is sent to the control device 2030.

[0374] The control unit 2074 controls the functions of the operating device 2070. The control unit 2074 is configured to include a processor such as a CPU.

[0375] The communication unit 2076 and the control device 2030 transmit and receive various information. The communication unit 2076 is configured to include a communication interface.

[0376] It should be noted that the motion detection unit 2072 can be connected to the communication unit 2060 of the display device 2050. The second operator's motion information can also be sent to the control device via the communication unit 2060. In this case, the control unit 2074 and the communication unit 2076 can be omitted.

[0377] The environmental information acquisition unit 2080 includes a camera unit 2082, a ranging unit 2084, and a communication unit 2086. The environmental information acquisition unit 2080 can be installed on the shell of the robot 2020 or in a location separate from the robot 2020.

[0378] The imaging unit 2082 captures images of the operating environment within a predetermined range from the robot 2020. The operating environment includes the range accessible to the actuators of the robot 2020. The captured images may not necessarily include the entire image of the robot 2020. The imaging unit 2082 is a digital camera that captures images at predetermined intervals. The imaging unit 2082 transmits image data representing the captured images to the control device 2030 via the communication unit 2086.

[0379] The ranging unit 2084 measures the distance to the surface of an object within a specified range of motion environment from the robot 2020. The ranging unit 2084 includes, for example, a wave transmitter, a wave receiver, and a distance detection unit. The wave transmitter emits waves such as infrared radiation. The wave transmitter is configured, for example, to include a light-emitting diode (LED). The wave receiver receives reflected waves generated by reflection from the surface of the object. The wave receiver is configured, for example, to include a photodiode. A reflected wave is generated corresponding to the wave emitted from the wave transmitter, which is incident as an incident wave onto the surface of the object. The distance detection unit detects the phase difference between the emitted wave and the reflected wave, and can determine the distance to the surface of the object based on the detected phase difference. The distance detection unit transmits distance data to the control device 2030 via the communication unit 2086. This distance data represents the distance determined according to each direction corresponding to each pixel of the image. It should be noted that the wave used for distance measurement is not limited to infrared radiation, but can also be millimeter waves, ultrasonic waves, etc.

[0380] The communication unit 2086 and the control device 2030 transmit and receive various information. The communication unit 2086 is configured to include a communication interface.

[0381] It should be noted that if the environmental information acquisition unit 2080 is installed in the shell of the robot 2020 and can send and receive various information with other functional units of the robot 2020, the communication unit 2086 may be omitted.

[0382] (Hardware structure)

[0383] Next, an example of the hardware structure of the control device 2030 in this embodiment will be described.

[0384] Figure 19 This is a schematic block diagram illustrating an example of the hardware structure of the control device 2030 according to this embodiment. The control device 2030 functions as a computer comprising a processor 2102, a ROM 2104, a RAM 2106, an auxiliary storage unit 2108, and an input / output unit 2110. The processor 2102, ROM 2104, RAM 2106, auxiliary storage unit 2108, and input / output unit 2110 are connected in a manner that allows them to input and output various types of data.

[0385] The processor 2102 reads, for example, the program and various data stored in the ROM 2104, and executes the program to perform processing for implementing the functions of each part of the control device 2030 and processing for controlling those functions. The processor 2102 is, for example, a CPU.

[0386] It should be noted that, in this application, the process indicated by various commands described in the program is sometimes referred to as "program execution" or "program execution".

[0387] ROM 2104, for example, stores programs for execution by processor 2102.

[0388] RAM 2106 functions, for example, as a working area for temporarily storing various data and programs used by the processor 2102.

[0389] The auxiliary storage unit 2108 permanently stores various types of data. Data acquired by the control device 2030 is stored in the auxiliary storage unit 2108. The auxiliary storage unit 2108 may include storage media such as HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0390] The input / output unit 2110 can, for example, input and output various data to other devices wirelessly or via wired connection. The input / output unit 2110 can also be connected to other devices via a network. The input / output unit may have one or both of an input / output interface and a communication interface.

[0391] It should be noted that the display device 2050, the operating device 2070, and the environmental information acquisition unit 2080 can also be configured to have the same... Figure 19 The illustrated hardware structure is the same as that of a computer that implements the functions of each device.

[0392] (Motion control processing)

[0393] Next, an example of motion control processing in this embodiment will be described. Figure 20 This is a flowchart illustrating an example of the motion control processing of this embodiment.

[0394] (Step S2102) The motion status inference unit 2034 obtains the latest environmental information, operation information and line of sight information at that point in time, and infers the motion status of the robot 2020 based on this information.

[0395] (Step S2104) The motion state inference unit 2034 refers to the characteristic parameter table and the drive control parameter table and determines the characteristic parameters and drive control parameters corresponding to the inferred motion state.

[0396] (Step S2106) The target position inference unit 2036 obtains the latest environmental information, operation information and line of sight information at that time point, and infers the target position of the actuator of the robot 2020 based on this information.

[0397] (Step S2108) The control command generation unit 2038 generates a control command for moving the actuator from the current position toward the target position based on the determined characteristic parameters.

[0398] (Step S2110) The drive control unit 2040 uses the drive control parameters to generate a drive command that controls the angle of each joint to the target value shown in the control command.

[0399] (Step S2112) The control command generation unit 2038 determines whether the actuator's position has converged. The control command generation unit 2038 can determine convergence, for example, by whether the actuator's current position is within a threshold value of a specified distance indicated by a characteristic parameter from the target position. When it is determined that the actuator's position has converged (step S2112 "Yes"), a control command indicating a stop to the action is generated and output to the drive control unit 2040. This stops the robot 2020's movement. Then, the process ends. Figure 20 The processing is as follows: If it is determined that the position of the actuator has not converged (step S2112, "No"), the process returns to step S2102.

[0400] <Fourth Implementation>

[0401] Next, the fourth embodiment of the present invention will be described. In the following description, the differences from the third embodiment will be the primary focus; regarding functions and structures common to the third embodiment, unless otherwise stated, the description of the third embodiment will be used. Except... Figure 21 In addition, refer to Figure 22 This will explain the functional structure of the robot system in this embodiment. Figure 21 This is a schematic block diagram showing a structural example of the robot system S2001 of this embodiment. Figure 22 This is a block diagram illustrating a functional structure example of a part of the control device 2030 of this embodiment.

[0402] The robot system S2001 of this embodiment determines the predicted trajectory of the actuators of the robot 2020 from the current time to a predicted time after a predetermined prediction time based on position information indicating the position of the robot 2020 and operation information indicating the operation status. The robot system S2001 generates control commands based on the determined predicted trajectory. The robot system S2001 of this embodiment further includes a trajectory prediction unit 2046 in the control device 2030.

[0403] The trajectory prediction unit 2046 uses robot motion information (drive state information) and operation information, which at least represent the actions of the robot 2020 at that point in time, to predict the trajectory of the actuators of the robot 2020 from the present to a predicted moment after a predetermined prediction time. Operation information is the primary factor in combination with acceleration, deceleration, or direction changes of the actuators. The trajectory prediction unit 2046 performs linear prediction, for example, based on the drive state information and operation information over a predetermined period up to the present, calculating the position and velocity of the actuators up to the prediction moment. In the trajectory prediction unit 2046, the time sequence of the actuators' positions at each moment up to the prediction moment constitutes the predicted trajectory. The prediction time represents the elapsed time from the present to the prediction moment, not the time consumed in the prediction.

[0404] The trajectory prediction unit 2046 may, for example, use a Kalman filter to predict the actuator's position sequentially. The Kalman filter is a method that has a prediction step and an update step, and repeats these steps sequentially. In the prediction step (a), the trajectory prediction unit 2046 adds the time evolution (weighted sum) of the actuator's state variables up to the previous moment to the external force, calculating the inferred value of the current actuator's state variables. However, the state variables are vectors with position and velocity as elements. The external force is a vector with velocity and acceleration as elements. The velocity is calculated based on the angles and angular velocities of each joint and the segment dimensions shown in the drive state information. The acceleration is calculated based on the angles, velocities, and angular accelerations of each joint and the segment dimensions shown in the drive state information. Additionally, the trajectory prediction unit 2046 adds the time evolution of the prediction error of the state variables up to the previous moment to the time evolution of the error, calculating the current prediction error. In the update step (b), the trajectory prediction unit 2046 calculates the difference between the current position (observation value) of the actuator and the inferred value of the current actuator position as the observation residual. The orbit prediction unit 2046 calculates the covariance of the current observation residuals based on the current prediction error. The orbit prediction unit 2046 calculates the Kalman gain based on the covariance of the current prediction error and the observation residuals. The orbit prediction unit 2046 updates the current actuator state variables to the value obtained by adding the inferred value of the current actuator state variables to the product of the observation residuals and the Kalman gain. The orbit prediction unit 2046 updates the current prediction error to the value obtained by multiplying the residuals of the mapping to the observation space by a Kalman gain starting from 1 with the prediction error.

[0405] The prediction time can be fixed to a specified value (e.g., 0.2s-2s) or it can be variable. The prediction time is preset to be equal to or less than the delay time until the operator's action is represented as the robot's action (also known as the "follow-up delay"). A longer prediction time results in a greater prediction error for the actuator's position, which may impair operability.

[0406] When the prediction time is set to be variable, a prediction schedule representing the prediction time for each action condition can be pre-stored in the storage unit 2044. Especially when the prediction time exceeds the delay time, the robot 2020's actions precede the operator's actions, which may cause significant prediction errors. Therefore, the delay time can also be set as an upper limit for the prediction time. The action condition inference unit 2034 can use the above method to infer the action condition of the robot 2020 and determine the prediction time corresponding to the inferred action condition by referring to the prediction schedule.

[0407] Alternatively, the more precise the position of the actuator being controlled is required in a given action, the smaller the prediction time set by the action state inference unit 2034. This type of action replaces situations where prediction errors by the actuator are not allowed and a delay is permitted. For example, this applies to situations where the object being operated on is a small object, or where the operator's operation is delicate. Conversely, the more responsive the actuator's position is required in a given action, the larger the prediction time set by the action state inference unit 2034. This type of action is suitable for situations where prediction errors are allowed and the responsiveness of the actuator for the operation is expected. For example, this applies to situations where the actuator moves globally toward the target object, or where the actuator retreats to a space where no other objects are present.

[0408] The motion state inference unit 2034 outputs prediction time information, representing the determined prediction time, to the trajectory prediction unit 2046. The trajectory prediction unit 2046 calculates the position and velocity of the actuator up to the prediction time, which is determined using the prediction time shown in the prediction time information input from the motion state inference unit 2034. The trajectory prediction unit 2046 outputs prediction trajectory information, representing the predicted trajectory, to the control command generation unit 2038. This predicted trajectory represents the position and velocity of the actuator at least from the current time to the prediction time.

[0409] When generating motion commands to move the actuator from its current position toward a target position, the control command generation unit 2038 performs known inverse kinematics calculations based on a predicted trajectory input from the trajectory prediction unit 2046 and characteristic parameters input from the motion state inference unit 2034, instead of using drive state information, to determine the angles of each joint, causing the actuator to move toward a convergent position. Conversely, the drive state information up to the present can also be considered as the actual trajectory representing the actuator's position at each moment. In this embodiment, by using a predicted trajectory instead of the actual trajectory, the delay in the actuator's motion can be compensated for.

[0410] Furthermore, the controller included in the drive control unit 2040 of this embodiment can also synthesize the first component calculated in the feedback term and the second component calculated in the feedforward term to calculate the operating quantity. The feedback term refers to a component of the operating quantity calculated based on the deviation between the target value and the output value, namely the first component. The target value is the angle of each joint corresponding to the target position. The output value is the angle of each joint shown in the drive state information. This output value corresponds to the current position of the actuator at that point in time. The PI control and PID control described above are equivalent to methods that determine the operating quantity based on the first component without considering the second component. That is, the proportional term, integral term, and derivative term described above are each included in the first component. The feedforward term is a component that calculates the operating quantity based on the target value without considering the output value, namely the second component. The second component includes a second proportional term that is proportional to the target value. The second proportional term is obtained by multiplying the target value by a second proportional gain.

[0411] Therefore, in the drive control parameter table, the drive control parameters corresponding to each operating condition are preset as follows: in addition to including a first gain related to the first component (corresponding to any one or a combination of the proportional gain, integral gain, and derivative gain mentioned above), a second gain related to the second component is also included. The second gain includes a second proportional gain. The second component may also include one or both of a second integral term proportional to the time integral value of the target value and a second derivative term proportional to the time derivative value of the target value. The second integral term is obtained by multiplying the target value by the second integral gain. The second derivative term is obtained by multiplying the target value by the second derivative gain. The second gain also includes one or both of the second integral gain and the second derivative gain. Furthermore, the ratio of the first gain to the second gain can vary depending on the operating condition. For example, the first and second gains can be determined such that the more precise the position of the actuator being controlled is required, the higher the ratio of the first factor to the second factor can be increased. In such operating conditions, the feedback term is relatively emphasized. Furthermore, the first and second gains are determined such that the ratio of the second factor to the first factor is relatively increased in operating conditions that require greater compliance with the actuator's position. In such operating conditions, the feedforward term is given relative importance.

[0412] The motion state inference unit 2034 refers to the drive control parameter table, determines the drive control parameters, including a first gain and a second gain, corresponding to the inferred motion state, and outputs the determined drive control parameters to the drive control unit 2040. Based on the first gain and second gain shown in the drive control parameters input from the motion state inference unit 2034, the drive control unit 2040 calculates a first factor and a second factor for each joint, and synthesizes the calculated first factor and second factor to determine the operating quantity. The drive control unit 2040 outputs a drive command representing the operating quantity determined for each joint to the drive unit 2024.

[0413] It should be noted that the controller in this embodiment may also omit the first component and determine the second component as the operating quantity. In this case, the setting of the first gain may also be omitted.

[0414] (Motion control processing)

[0415] Next, an example of motion control processing in this embodiment will be described. Figure 23 This is a flowchart illustrating an example of the motion control processing of this embodiment. Figure 23 The processing shown is relative to Figure 20 The process shown also includes steps S2122, S2124, and S2126.

[0416] After the target position inference unit 2036 finishes processing step S2106, it proceeds to processing step S2122.

[0417] (Step S2122) The motion state inference unit 2034 refers to the prediction time schedule and determines the prediction time corresponding to the motion state it has determined.

[0418] (Step S2124) The trajectory prediction unit 2046 uses robot motion information and operation information to infer the predicted trajectory of the actuator from the present to the predicted moment after the determined prediction time.

[0419] (Step S2126) The control command generation unit 2038 uses the predicted trajectory and characteristic parameters to generate control commands, expressed as target values, for each joint angle that will move the actuator toward the target position. Then, the process proceeds to step S2110.

[0420] It should be noted that in step S2122, the action state inference unit 2034 may also refer to the drive control parameter table to determine the drive control parameters corresponding to the action state.

[0421] In step S2110, the drive control unit 2040 may also use the determined drive control parameters to calculate a first factor and a second factor based on the target value and the deviation between the target value and the output value, and synthesize the calculated first factor and the second factor to determine the operation quantity. The drive control unit 2040 generates a drive command representing the determined operation quantity and outputs the generated drive command to the drive unit 2024.

[0422] As explained above, the control device 2030 of this embodiment includes an action state inference unit 2034, which infers the action state of the robot 2020 based at least on environmental information representing the action environment of the robot 2020 and operation information representing the operation state. The control device 2030 includes a control command generation unit 2038, which generates control commands for causing the actuators of the robot 2020 to move based on the operation information. The control device 2030 includes a drive control unit 2040, which controls the movement of the robot 2020 based on the control commands. The control command generation unit 2038 determines the control commands based on characteristic parameters related to control characteristics corresponding to the action state.

[0423] According to this structure, the actuator's movements are controlled using operational information inferred from the motion environment and operational conditions. Because the actuator is operated based on the motion conditions, the robot's operational efficiency is improved.

[0424] In addition, the motion status inference unit 2034 can also infer the motion status based on operator information indicating the status of the operator operating the robot.

[0425] Based on this structure, and taking into account the operator's condition, the robot's actions can be accurately inferred. Therefore, the robot's operational efficiency is further improved in 2020.

[0426] In addition, the control device 2030 may also include a target position inference unit 2036, which infers the target position of the actuator based at least on operation information and environmental information.

[0427] According to this structure, the robot 2020's movements are controlled so that the actuators move towards a target position determined based on the action environment and operating conditions. The operator does not need to perform operations to accurately indicate the target position, thus further improving the robot 2020's operational efficiency.

[0428] Alternatively, the control command generation unit 2038 can also determine the amount of operation to drive the actuator toward the target position based on characteristic parameters. These characteristic parameters may also include convergence determination parameters that represent convergence criteria for convergence toward the target position.

[0429] According to this structure, the convergence criterion is determined based on the action conditions, and this criterion determines that the actuator's position converges to the target position. Therefore, the required or expected position accuracy or solution stability can be achieved based on the action conditions.

[0430] The control command generation unit 2038 may also determine the operating quantity based on a target function representing the load used to move the actuator toward the target position. Alternatively, the target function may be a function synthesized from multiple factors, with characteristic parameters including the weight of each factor.

[0431] According to this structure, the weights of factors related to the load on the actuator's action are determined based on the operating conditions. Therefore, the operating characteristics can be adjusted according to the operating conditions, thereby reducing factors related to the type of requirement or expectation.

[0432] The drive control unit 2040 can also determine the operating amount based on characteristic parameters, thereby reducing the deviation between the target value based on the control command and the output value output from the actuator's motion mechanism. The characteristic parameters may also include the gain of the deviation relative to the operating amount.

[0433] According to this structure, the gain relative to the operational quantity is adjusted based on the deviation between the target value and the output value according to the motion status. Because the speed at which the actuator moves towards the target position can be adjusted according to the motion status, the operator's work using the robot becomes more efficient.

[0434] In addition, the control device 2030 includes: a trajectory prediction unit 2046, which determines the predicted trajectory of the actuators of the robot 2020 from the current time to a predicted time after a predetermined prediction time, based at least on motion information representing the actions of the robot 2020 and operation information representing the operation status; and a control command generation unit 2038, which generates control commands based on the predicted trajectory.

[0435] According to this structure, the actuators of robot 2020 are driven by control commands generated based on the predicted trajectory of the actuators up to a predicted time later than the current time. Therefore, the delay in the time it takes for the operation to be reflected in the robot 2020's actions (follow-up delay) is reduced or eliminated. For the operator, the sense of operation is improved, thus enabling both increased work efficiency and reduced workload.

[0436] Additionally, the control device 2030 may also include a motion state inference unit 2034, which infers the motion state of the robot 2020 based at least on environmental information and operational information representing the robot 2020's motion environment. The trajectory prediction unit 2046 may also determine the prediction time based on the motion state.

[0437] According to this structure, the prediction time is determined based on the robot's motion state inferred from its motion environment and operational status. Therefore, a balance is struck between improving the sense of control and balancing the accuracy of the controlled actuator's position, based on the motion state.

[0438] Alternatively, the control device 2030 may also include a drive control unit 2040, which determines the amount of operation on the motion mechanism based on the target value of the displacement of the motion mechanism of the robot 2020 given a target position, wherein the target position is the target position of the actuator forming the predicted trajectory at each moment. The motion state inference unit 2034 may also determine the gain of the target value based on the motion state.

[0439] According to this structure, the contribution of the target value to the operational quantity is adjusted based on the action status, where the operational quantity is the operational quantity for the motion mechanism. Therefore, the sensitivity of the actuator's action to the operator's operation is adjusted according to the action status.

[0440] Additionally, the drive control unit 2040 can synthesize the first component and the second component to determine the operating quantity. The first component is obtained based on the deviation between the output value of the displacement assigned to the current position of the actuator and the target value, and a first gain. The second component is obtained based on the target value and the second gain. The operation status inference unit 2034 can also determine the first gain and the second gain based on the operation status.

[0441] Based on this structure, the balance between feedback and feedforward terms is adjusted according to the motion status. Therefore, the sensitivity and accuracy of the actuator's actions to the operator's input are adjusted according to the motion status.

[0442] In addition, the motion status inference unit 2034 can also infer the motion status based on operator information indicating the status of the operator operating the robot.

[0443] Based on this structure, and taking into account the operator's condition, the robot's actions are accurately inferred. Therefore, the robot's operational efficiency is further improved, further reducing the workload.

[0444] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the specific structure is not limited to the structure described above, and various design changes can be made without departing from the spirit of the present invention.

[0445] The above-described implementation methods and their variations can be combined with each other without creating contradictions, and some of them can be omitted.

[0446] For example, in the above embodiments, the target position inference unit 2036 may be omitted, as may the processing based on the target position inferred by the target position inference unit 2036.

[0447] In the fourth embodiment, when the prediction time is fixed, the action state inference unit 2034 may be omitted, and the processing based on the action state may also be omitted.

[0448] In the environmental information acquisition unit 2080, the ranging unit 2084 can also be integrated with the imaging unit 2082 to form a three-dimensional image imaging unit. Alternatively, the environmental information acquisition unit 2080 can also include a multi-view imaging unit to replace the imaging unit 2082 and the ranging unit 2084. A multi-view imaging unit is an imaging unit that has multiple viewpoints and is capable of capturing images from each viewpoint. The multi-view imaging unit includes a stereo camera. The images captured by the multi-view imaging unit from each viewpoint (multi-view images) represent the robot 2020's operational environment in three dimensions. The target position inference unit 2036 determines the area occupied by an object from the robot 2020's operational environment by analyzing the multi-view images.

[0449] Alternatively, part or all of the display device 2050 may be omitted. The description primarily focuses on the case where gaze information represents the gaze direction of one of the operator's eyes (e.g., the left eye), but is not limited to this. The gaze information may also represent the gaze directions of both the left and right eyes individually. If the gaze direction of one eye cannot be used at a certain moment (e.g., during blinking), the target position inference unit 2036 may also use the gaze direction of the other eye when determining the fixation point. When the gaze directions of both eyes can be used, the target position inference unit 2036 may determine the fixation point as the intersection point of the line segments representing the gaze directions of both eyes, or the midpoint of the nearest points of these line segments. Furthermore, the target position inference unit 2036 may determine the gaze direction representing the gaze directions of both eyes from the direction from the head to the fixation point, and determine the intersection point, which is the intersection of the determined gaze direction and the surface of the object. The determined intersection point is used for setting the target position.

[0450] Robot 2020 can also have multiple robotic arms. These robotic arms can correspond to the forearms and hands of multiple operators. Each robotic arm can move according to the movements of the forearms and hands of its respective operator. Control device 2030 can also use the above method to determine a target position for each robotic arm and move the robotic arm toward the determined target position. Robot 2020 can also be a dual-arm robot with two robotic arms. For a dual-arm robot, for example, one robotic arm and the other can correspond to the left and right hands of an operator, respectively.

[0451] Explanation of reference numerals in the attached figures

[0452] 1…Robot remote operation control system, 2…Robot, 3…Robot remote operation control device, 5…HMD, 6…Controller, 7…Environmental sensor, 21…Control unit, 22…Drive unit, 23…Sound collection unit, 25…Storage unit, 26…Power supply, 27…Sensor, 31…Information acquisition unit, 33…Intent inference unit, 34…Holding method determination unit, 35…Robot status image generation unit, 36…Transmission unit, 37…Storage unit, 51…Image display unit, 52…Gaze detection unit, 53…Sensor, 54…Control unit, 55…Communication unit, 61…Sensor, 62…Control unit, 63…Communication unit 64…Feedback unit, 71…Imaging device, 72…Sensor, 73…Object position detection unit, 74…Communication unit, 1001…Robot remote operation control system, 1002…Robot, 1003…Robot remote operation control device, 1005…HMD, 1006…Controller, 1007…Environmental sensor, 1021…Control unit, 1022…Drive unit, 1023…Sound collection unit, 1025…Storage unit, 1026…Power supply, 1027…Sensor, 1221…Holding unit, 1031…Information acquisition unit, 1033…Intent inference unit, 1034…Control command generation unit. 1035…Robot Status Image Generation Unit, 1036…Transmission Unit, 1037…Storage Unit, 1051…Image Display Unit, 1052…Gaze Detection Unit, 1054…Control Unit, 1055…Communication Unit, 1061…Sensor, 1062…Control Unit, 1063…Communication Unit, 1064…Feedback Unit, 1071…Image Capture Device, 1072…Sensor, 1073…Communication Unit, S2001…Robot System, 2020…Robot, 2024…Drive Unit, 2026…Drive Sensor, 2028…Power Supply, 2030…Control Device, 2032…Information Acquisition Unit, 2034… …Motion Status Inference Unit, 2036…Target Position Inference Unit, 2038…Control Command Generation Unit, 2040…Drive Control Unit, 2042…Communication Unit, 2044…Storage Unit, 2046…Track Prediction Unit, 2050…Display Device, 2052…Display Unit, 2054…Line of Sight Detection Unit, 2056…Motion Detection Unit, 2058…Control Unit, 2060…Communication Unit, 2070…Operating Device, 2072…Motion Detection Unit, 2074…Control Unit, 2076…Communication Unit, 2080…Environmental Information Acquisition Unit, 2082…Photography Unit, 2084…Distance Measuring Unit, 2086…Communication Unit.

Claims

1. A remote operation and control device for a robot, wherein, The robot remote operation control device includes an information acquisition unit, an intent inference unit, and a gripping method determination unit. In remote operation and control of robots, where an operator remotely manipulates the robot capable of grasping objects. The information acquisition unit acquires operator status information about the operator performing the operation on the robot. The intent inference unit infers the operator's intention to make the robot perform an action based on the operator's state information. The gripping method determination unit determines the gripping method of the object based on the inferred intention of the operator. The intent inference unit classifies the operator's grip posture according to a pre-determined classification based on the shape of the operator's hand, and infers the operator's action intent based on the classification results of the grip posture. The intent inference unit infers the future trajectory of the operator's fingertips as the action intent based on the operator's state information and the robot's state information representing the robot's state. The gripping method determination unit calculates the contact point between the object and the robot's fingers based on the classification result of the gripping posture, the shape of the object, the physical parameters of the object, and the torque that the robot can output. The gripping method determination unit calculates the joint angle of the finger based on the contact point. The gripping method determination unit corrects the movements of the robot's fingers operated by the operator based on the intention of the action and the joint angle.

2. The robot remote operation control device according to claim 1, wherein, The intent inference unit classifies the operator's posture based on the operator's state information, thereby determining the classification of the robot's posture and inferring the operator's action intent.

3. The robot remote operation control device according to claim 1, wherein, The intent inference unit infers, based on the operator's state information, the method of holding the object to be held and at least one of the objects to be held, thereby inferring the operator's intention to act.

4. The robot remote operation control device according to claim 1, wherein, The intent inference unit infers the object to be held based on the operator's state information, and infers the holding method of the object to be held associated with the inferred object, thereby inferring the operator's action intent.

5. The robot remote operation control device according to claim 1, wherein, The intent inference unit infers the method of holding the object to be held based on the operator's state information, and infers the object to be held based on the inferred method of holding the object to be held, thereby inferring the operator's action intent.

6. The robot remote operation control device according to any one of claims 1 to 5, wherein, The operator status information is at least one of the operator's gaze information, the operator's arm movement information, and the operator's head movement information.

7. The robot remote operation control device according to any one of claims 1 to 5, wherein, The information acquisition unit acquires the position information of the object. The gripping method determination unit also uses the obtained position information of the object to infer the object to be gripped and the gripping method of the object.

8. The robot remote operation control device according to any one of claims 1 to 5, wherein, The gripping method determination unit obtains the position information of the gripping part of the robot and corrects the position information of the gripping part based on the operator's state information.

9. The robot remote operation control device according to claim 8, wherein, The robot remote operation and control device also includes a robot status image generation unit. The intent inference unit obtains information related to the object based on the image captured by the imaging device. The robot state image generation unit generates an image to be provided to the operator based on information related to the object, the position information of the gripping part, the operator's state information, and the corrected position information of the gripping part.

10. A remote operation control system for a robot, wherein, The robot remote operation and control system has the following features: A robot having a gripping part for holding the object and a detection part for detecting the position information of the gripping part; The robot remote operation control device according to any one of claims 1 to 9; An environmental sensor detects the position information of the object; as well as Sensors that detect operator state information of the operator performing the operation on the robot.

11. A method for remote operation and control of a robot, wherein, In remote operation and control of robots, where an operator remotely manipulates the robot capable of grasping objects. The information acquisition unit acquires operator status information about the operator performing the operation on the robot. The intent inference unit infers the operator's intention to make the robot perform an action based on the operator's state information. The gripping method determination unit determines the gripping method of the object based on the inferred intention of the operator. The intent inference unit classifies the operator's grip posture according to a pre-determined classification based on the shape of the operator's hand, and infers the operator's action intent based on the classification results of the grip posture. The intent inference unit infers the future trajectory of the operator's fingertips as the action intent based on the operator's state information and the robot's state information representing the robot's state. The gripping method determination unit calculates the contact point between the object and the robot's fingers based on the classification result of the gripping posture, the shape of the object, the physical parameters of the object, and the torque that the robot can output. The gripping method determination unit calculates the joint angle of the finger based on the contact point. The gripping method determination unit corrects the movements of the robot's fingers operated by the operator based on the intention of the action and the joint angle.

12. A storage medium storing a program, wherein, In remote operation and control of robots, where an operator remotely manipulates the robot capable of grasping objects. The program causes the computer to perform the following processes: Obtain operator state information of the operator performing the operation on the robot. Based on the operator's state information, the operator's intention to make the robot perform the action can be inferred. The method of holding the object is determined based on the inferred intention of the operator's actions. The operator's grip posture is classified according to a pre-determined grip posture based on the shape of the operator's hand, and the operator's intention to move is inferred based on the classification results of the grip posture. Based on the operator's state information and the robot's state information representing the robot's state, the future trajectory of the operator's fingertips is inferred as the operator's intention for the action. Based on the classification results of the gripping posture, the shape of the object, the physical parameters of the object, and the torque that the robot can output, the contact point between the object and the robot's fingers is calculated. The joint angle of the finger is calculated based on the contact point. The movements of the robot's fingers operated by the operator are corrected based on the stated intention and the joint angle.

13. A remote operation control device for a robot, wherein, The robot remote operation control device includes an intent inference unit and a control command generation unit. In remote robot operation, the operator's movements are recognized and transmitted to the robot to control the robot. The intent inference unit infers the operator's actions based on robot environment sensor values ​​obtained from environmental sensors located on the robot or its surrounding environment, and operator sensor values ​​obtained from operator sensors as representations of the operator's movement. The control command generation unit generates appropriate control commands based on the inferred actions of the operator, targeting a portion of the degrees of freedom of the operator's actions, thereby reducing the degrees of freedom of the operator's actions and generating control commands.

14. The robot remote operation control device according to claim 13, wherein, The control command generation unit restricts the degrees of freedom that the operator should control and the range that can be controlled, and provides action assistance for the restricted degrees of freedom in the operator's action instructions to the robot.

15. The robot remote operation control device according to claim 13 or 14, wherein, When the distance between the robot's gripping part and the object being manipulated by the operator is outside a predetermined range, the control command generation unit does not reduce the operator's degrees of freedom in their movements. When the distance between the robot's gripping part and the object being manipulated by the operator is within a predetermined range, the control command generation unit reduces the degree of freedom of the operator's movements. The robot's environmental sensor values ​​include captured image information and depth information.

16. The robot remote operation control device according to claim 13 or 14, wherein, The intent inference unit inputs the robot's environmental sensor values ​​and the operator's sensor values ​​into the learned intent inference model to infer the operator's actions.

17. The robot remote operation control device according to claim 13 or 14, wherein, The operator sensor value is at least one of the operator's gaze information and information related to the posture and position of the operator's arm, namely, operator arm information.

18. The robot remote operation control device according to claim 13 or 14, wherein, The robot's environmental sensor values ​​include captured image information and depth information.

19. A remote operation control system for a robot, wherein, The robot remote operation control system comprises the robot remote operation control device, gripping unit, environmental sensor, and operator sensor as described in any one of claims 13 to 18. In remote robot operation, the operator's movements are recognized and transmitted to the robot to control the robot. The gripping part grips the object. The environmental sensors are installed on the robot or its surrounding environment to detect the robot's environmental sensor values. The operator sensor detects the operator's movement and uses it as the operator sensor value.

20. A method for remote operation and control of a robot, wherein, In remote robot operation, the operator's movements are recognized and transmitted to the robot to control the robot. The intent inference unit infers the operator's actions based on robot environment sensor values ​​obtained from environmental sensors located on the robot or its surrounding environment, and operator sensor values ​​obtained from operator sensors representing the operator's movement. The control command generation unit generates appropriate control commands for a portion of the operator's degrees of freedom based on the inferred actions of the operator, thereby reducing the degrees of freedom of the operator's actions and generating control commands.

21. A storage medium storing a program, wherein, In remote robot operation, the operator's movements are recognized and transmitted to the robot to control the robot. The program causes the computer to perform the following processes: The operator's actions are inferred based on robot environmental sensor values ​​obtained from environmental sensors located on or around the robot, and operator sensor values ​​representing the operator's movement obtained from operator sensors. Based on the inferred operator's actions, appropriate control commands are generated for a portion of the operator's degrees of freedom, thereby reducing the operator's degrees of freedom and generating control commands.

22. A control device, wherein, The control device includes: The trajectory prediction unit determines the predicted trajectory of the robot's actuators from the current moment to a predicted moment after a predetermined prediction time, based at least on motion information representing the robot's actions and operation information representing the operation status. as well as The control command generation unit generates control commands based on the predicted trajectory. The control device further includes a motion state inference unit, which infers the robot's motion state based at least on environmental information representing the robot's motion environment and the operation information. The trajectory prediction unit determines the prediction time based on the action status.

23. The control device according to claim 22, wherein, The control device includes a drive control unit that determines the amount of operation on the robot's motion mechanism based on a target value, wherein the target value is a target value that assigns a displacement to the robot's motion mechanism at a target position of the actuator forming the predicted trajectory at each time step. The action state inference unit determines the gain of the target value based on the action state.

24. The control device according to claim 23, wherein, The drive control unit synthesizes a first component and a second component to determine the operating quantity. The first component is obtained based on the deviation between the output value of the displacement assigned to the current position of the actuator and the target value, plus a first gain. The second component is obtained based on the target value and a second gain. The action state inference unit determines the first gain and the second gain based on the action state.

25. The control device according to any one of claims 22 to 24, wherein, The motion status inference unit also infers the motion status based on operator information, which indicates the status of the operator operating the robot.

26. A storage medium storing a program, wherein, The program is used to enable the computer to function as a control device as described in any one of claims 22 to 24.

27. A robot system, wherein, The robot system comprises the control device and the robot as described in any one of claims 22 to 24.

28. A control method, which is a control method in a control device, wherein, The control method comprises: In the first step, the predicted trajectory of the robot's actuators from the current moment to the predicted moment after a predetermined prediction time is determined based at least on motion information representing the robot's actions and operation information representing the operation status. as well as The second step involves generating control commands based on the predicted trajectory. The robot's motion status can be inferred based at least on environmental information representing the robot's motion environment and the operational information. The predicted time is determined based on the action status.

Images