Information processing device, information processing method, program

Abstract

In a technology that remotely controls an object based on the movement of body parts such as hands and feet, the present invention provides an information processing device that improves the operability of the object during remote control. [Solution] When remotely controlling the target device's arm by moving a part of the body that is the target of the movement, a reference position of the body part that is the target of the movement corresponding to the state where the arm tip is in the predetermined position, and a limit position indicating the position where the body part has been moved from the reference position within the range in which the body part that is the target of the movement can be moved are detected. The movement ratio when remotely controlling the target device's arm by moving the body part that is the target of the movement from the reference position to the limit position is determined based on the maximum distance when the target device's arm is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is a technology for remotely operating a remotely located operation target. For example, Patent Document 1 discloses a control device including a slave manipulator installed to access a surgical field and a master manipulator installed within an area where an operator can operate and remotely operating the slave manipulator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when operating a remotely located operation target, a technology is required to move a part such as a hand or foot of the body and remotely operate the operation target based on the movement of the part. And in this remote operation, it is required to improve the operability when remotely operating the operation target.

[0005] An object of this disclosure is to provide an information processing apparatus, an information processing method, and a program for solving the above problems.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present disclosure includes, when performing remote operation to extend the tip of an arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, position detection means for detecting a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position, and a limit position indicating the position in which the part has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved, and movement ratio determination means for determining the movement ratio when remotely operating the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0007] An information processing method according to one aspect of the present disclosure, when performing remote operation to extend the tip of an arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, detects a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position, and a limit position indicating the position in which the part has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved, and determines the movement ratio when remotely operating the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0008] A program according to one aspect of the present disclosure causes the computer of an information processing device to function as a position detection means for detecting a reference position of the body part to be moved, corresponding to the state in which the tip of the arm of the target device is at the predetermined position, and a limit position indicating the position in which the body part has been moved from the reference position within the range in which the body part to be moved can be moved, when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, and a movement ratio determination means for determining the movement ratio when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position. [Effects of the Invention]

[0009] According to one embodiment described above, in a technology that remotely controls an object based on the movement of body parts such as hands and feet, the operability of the object during remote control can be improved. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the remote control system related to this disclosure. [Figure 2] This figure shows the hardware configuration of the HMD related to this disclosure. [Figure 3] This is a functional block diagram of the HMD related to this disclosure. [Figure 4] This diagram illustrates the reference position and limit position related to this disclosure. [Figure 5] This is the first figure showing the relationship between the distance from the reference position and the shift magnification related to this disclosure. [Figure 6] This diagram shows the processing flow of the HMD related to this disclosure. [Figure 7] This figure shows the operation of the robot arm related to this disclosure. [Figure 8] This figure shows the reference positions of the left and right hands related to this disclosure. [Figure 9]A diagram showing an overview of calculating the distance from the reference position to the limit position of the hand according to the present disclosure. [Figure 10] A second diagram showing the relationship between the distance from the reference position and the movement magnification according to the present disclosure. [Figure 11] A third diagram showing the relationship between the distance from the reference position and the movement magnification according to the present disclosure. [Figure 12] A fourth diagram showing the relationship between the distance from the reference position and the movement magnification according to the present disclosure. [Figure 13] A fifth diagram showing the relationship between the distance from the reference position and the movement magnification according to the present disclosure. [Figure 14] A sixth diagram showing the relationship between the distance from the reference position and the movement magnification according to the present disclosure. [Figure 15] A diagram showing the functional blocks of an information processing apparatus according to another configuration of the present disclosure. [Figure 16] A diagram showing the processing flow of an information processing apparatus according to another configuration of the present disclosure.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, the information processing apparatus of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing a remote operation system including the information processing apparatus of the present disclosure. The remote operation system 100 includes a head-mounted display (hereinafter referred to as HMD) 1 and a robot arm 2 that is the object of remote operation. The HMD 1 and the robot arm 2 are communicatively connected via a communication network. In the present disclosure, the HMD 1 is an aspect of the information processing apparatus. In the present disclosure, the robot arm 2 is an aspect of the arm that is the object device of remote operation.

[0012] The HMD1 is worn on the head of an operator who remotely operates the robotic arm 2. The HMD1 performs remote control of the extension and contraction of the robotic arm 2 based on detection based on sensing information such as an image of the position of a hand, which is an example of a part of the operator's body, moving from a reference position close to the body to a limit position in a certain moving direction. Further, the MHD1 performs remote control of attaching or detaching the selection of two actuator fingers that constitute the gripping mechanism of the hand portion provided at the tip of the robotic arm 2 based on detection based on sensing information such as an image of attaching or detaching the index finger and the thumb of the hand, which is an example of a part of the operator's body.

[0013] In the present disclosure, the case where the parts of the body that are the objects of movement when the operator operates the robotic arm 2 are the hands and arms will be described. However, in other disclosures, the parts of the body that are the objects of movement when the operator operates the robotic arm 2 may be the feet and legs.

[0014] Also, in other disclosures, a server device may be communicatively connected during the communication between the HMD1 and the robotic arm 2, and the HMD1 and the robotic arm 2 may be communicatively connected via the server device. In this case, the server device relays the signals between the HMD1 and the robotic arm 2. Also, the server device may perform some of the processing of the functions of the HMD1.

[0015] Figure 2 is a diagram showing the hardware configuration of the HMD. As shown in Figure 2, HMD1 is a computer equipped with hardware such as a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, storage device 104, communication module 105, sensing device 106, input device 107, monitor 108, and speaker 109. The sensing device 106 may include a camera 61, motion sensor 62, and depth sensor 63. The camera 61 generates and outputs an image, which is an example of sensing information. The motion sensor 62 generates and outputs motion presence / absence information, which is an example of sensing information, indicating the presence or absence of human movement. The depth sensor 63 generates and outputs distance information, indicating the distance to each position of the subject. HMD1 may further include other hardware such as a GPU instead of the CPU 101.

[0016] Figure 3 is a functional block diagram of the HMD. The HMD1 performs at least the functions of the control unit 11, acquisition unit 12, position detection unit 13, movement magnification specification unit 14, and remote control unit 15 by having the CPU 101 or GPU execute a program.

[0017] The control unit 11 controls other functional units, etc. The acquisition unit 12 acquires information for processing information from the HMD1. The position detection unit 13 detects, when an operator remotely controls the robot arm 2 by moving their hand to extend the tip of the robot arm 2 from a predetermined position to its maximum position, a reference position of the hand corresponding to the initial predetermined position of the tip of the robot arm 2, and a limit position indicating the position where the hand has been moved from the reference position within the range in which the hand can be moved (extended).

[0018] The movement magnification unit 14 determines the movement magnification when remotely controlling the robot arm 2 by moving the hand from a reference position to a limit position, based on either the maximum distance Ra when the tip of the robot arm 2 is extended from an initial predetermined position to the maximum position, the distance La from the reference position to the limit position of the hand when the arm is extended to the maximum extent by the person, or the distance L2 from the reference position to the limit position of the hand when the arm is not extended to the maximum extent by the person and hits an obstacle. The remote control unit 15 remotely controls the robot arm 2 using the movement magnification ratio used when the operator moves their hand to remotely control the robot arm 2.

[0019] Figure 4 is a diagram illustrating the reference position and the limit position. As shown in Figure 4, we assume a situation where there is an obstacle such as a wall near the operator. The wall is assumed to be closer to the operator's body than the position the operator can reach with their hand extended. The state in which the operator brings their hand closest to their body is defined as the hand being at the reference position p1.

[0020] Within the range a in which the operator can move their hand, the state in which the hand is at its limit position p2 is defined as the state in which the hand is at its limit position p2, where the arm is extended and the hand touches the wall (or the limit position in which the hand can be moved to just before the hand touches an obstacle such as a wall). The distance from the reference position p1 to the limit position p2 is defined as L2. Ra is defined as the maximum distance from the tip position of the robot arm 2 when it is shortened to a predetermined length to the tip position of the arm when it is extended to its maximum length.

[0021] The following explanation assumes that the HMD1 remotely controls the movement of the tip of the robot arm 2 based on the detection of sensing information such as images indicating that the operator's hand position has moved. If there are no obstacles such as walls near the operator, and the length of the operator's arm (the length to the fingertips when the arm is extended) is equal to or greater than the maximum distance Ra when the robot arm 2 is extended, the operator can extend their arm to the maximum distance Ra, thereby extending the robot arm 2, which extends and retracts at a distance equal to the distance to the hand at the tip of the arm, to the maximum distance Ra. However, if there are obstacles such as walls near the operator, the operator cannot extend their arm and move their hand to the position of distance Ra. Also, if the length of the operator's arm is shorter than the length of the robot arm 2, the operator cannot extend their arm and move their hand to the position of distance Ra. The HMD1 according to this disclosure calculates a movement multiplier so that the robot arm 2 extends to the maximum distance Ra when the operator extends their arm and moves their hand to the limit position p2, even when there are obstacles such as walls near the operator or when the length of the operator's arm is shorter than the length of the robot arm 2, and controls the extension and retraction of the robot arm 2 using this movement multiplier. Let L1 be the distance to the position of the hand when the operator extends their arm to a position closer to the operator than the distance L2 to the obstacle. In this case, the HMD1 may control the movement multiplier of the robot arm 2 corresponding to the hand movement until the hand reaches position L1 to different multipliers, and the movement multiplier of the robot arm 2 corresponding to the hand movement from L1 to position La.

[0022] The HMD1 instructs the user to place their hand at reference position p1 through voice guidance output from speaker 109 or text guidance output to the HMD1's monitor 108. At a predetermined timing after the output of the guidance instructing the user to place their hand at reference position p1, the HMD1 can detect the reference position p1 of the hand using image information acquired from camera 61 and distance information to the subject (hand) acquired from depth sensor 63.

[0023] The HMD1 instructs the user to place their hand at the limit position p2 through voice guidance output from speaker 109 and text guidance output to the HMD1's monitor 108. At a predetermined timing after the output of the guidance instructing the user to place their hand at the limit position p2, the HMD1 can detect the limit position p2 of the hand using image information acquired from camera 61 and distance information to the subject (hand) acquired from depth sensor 63.

[0024] In this disclosure, limit position p2 is defined as the position where the operator reaches out and touches the wall. However, limit position p2 could also be defined as the position where a person with a disability who has difficulty reaching out can reach as far as possible. Alternatively, limit position p2 could be defined as the position where a person without disabilities can reach as far as possible in an unobstructed environment. In other words, in other examples of this disclosure, a situation without obstacles may be assumed.

[0025] The HMD1 in this disclosure detects a reference position p1 and a limit position p2, and calculates the movement multiplier of the tip of the robot arm 2 when remotely controlling the extension and retraction of the robot arm 2, which is the target device for remote control, by moving the part of the operator's body that is the target of the operation (hand) from the reference position p1 to the limit position p2. The movement multiplier is determined based on the maximum distance Ra when the robot arm 2 is extended from a predetermined position to the maximum position, and the distance L2 from the reference position p1 of the operator's hand to the limit position p2 (when there is an obstacle) or the distance La from the reference position p1 of the operator's hand to the limit position p2 (when there is no obstacle). As an example when there is an obstacle, the HMD1 calculates the movement multiplier using the formula Movement Multiplier = Ra ÷ L2. As an example when there is no obstacle, the HMD1 calculates the movement multiplier using the formula Movement Multiplier = Ra ÷ La. The value of the distance Ra is stored in advance by the HMD1.

[0026] Figure 5 is the first figure showing the relationship between the distance from the reference position and the magnification factor. For example, if there is an obstacle, HMD1 determines the movement multiplier calculated by the value of Ra÷L2, as shown in Figure 5. If there is no obstacle, HMD1 determines the movement multiplier calculated by the value of Ra÷La, as shown in Figure 5. HMD1 calculates the control distance by multiplying the distance L1 from the reference position of the hand by the movement multiplier and outputs this control distance to robot arm 2. Based on the acquired control distance, robot arm 2 operates so that the distance from a predetermined position before the start of movement to the position of the arm tip (or the distance from the initial position of robot arm 2 to the extended position) becomes the control distance. As a result, when the operator extends their arm and reaches the limit position p2 from the reference position p1 of the hand, and the distance from the reference position p1 to the hand becomes L2 or distance La, the distance from the predetermined position of the arm tip of robot arm 2 becomes Ra, and it can be extended to its maximum extent.

[0027] Figure 6 shows the processing flow of the HMD. The processing flow of the HMD1 will be explained step by step using Figure 6. First, the user inputs the command to start operating the robot arm 2 to the HMD1 using the input device 107 (step S101). The control unit 11 of the HMD1 receives the command to start operation. The control unit 11 outputs a sensing command to the sensing device 106. As a result, the camera 61, motion sensor 62, and depth sensor 63 of the sensing device 106 start operating.

[0028] The control unit 11 starts the calibration process. For example, in this calibration process, the control unit 11 determines whether or not it has received an input to start setting the movement magnification (step S102).

[0029] When the control unit 11 detects that it has received an input to start setting the movement magnification, it outputs first guidance from the monitor 108 or speaker 109 instructing the operator to place the hand at the reference position (step S103). Based on the output of the first guidance, the operator recognizes that they should move the position of their hand to a position close to their body when starting to operate the robot arm 2. The operator moves their hand to the position that will be the starting position for operating the robot arm 2, which is close to their body, and inputs an instruction to the input device 107 that the hand movement is complete. The input device 107 may be a predetermined button provided on the HMD1. The control unit 11 detects that the hand movement is complete and instructs the position detection unit 13 to detect the reference position.

[0030] The position detection unit 13 acquires image information of the subject area from the camera 61 and distance information from the depth sensor 63 to the subject corresponding to each pixel of the image information of the subject area. The position detection unit 13 identifies the area in which the hand is captured from the image information. The position detection unit 13 acquires the distance to each pixel in the area in which the hand is captured. Based on the distance to each pixel in the area in which the hand is captured, the position detection unit 13 detects a reference position p1 in a three-dimensional space with the center of the head as the center of the coordinate system (step S104).

[0031] The coordinate system of the position detected by HMD1 in this disclosure may be centered on the position of HMD1. The image information and distance information may be image information and distance information when the operator wearing HMD1 visually perceives the direction of the position of the hand. HMD1 has a function that can determine the orientation of the operator's face using a gyro sensor and a geomagnetic sensor, and it is assumed that the coordinates of the hand in three-dimensional space can be detected using known technology by using these functions and distance information to the hand.

[0032] The control unit 11 detects that the detection of the reference position p1 is complete. The control unit 11 outputs a second guidance from the monitor 108 or speaker 109 instructing the operator to place their hand at the limit position p2 (step S105). Based on the output of the second guidance, the operator moves their hand to a position where it will touch the obstacle in front of them (or just before it touches the obstacle) and inputs an instruction to the input device 107 that the hand movement is complete. The control unit 11 detects that the hand movement is complete and instructs the position detection unit 13 to detect the limit position p2.

[0033] The position detection unit 13 acquires image information of the subject area from the camera 61 and distance information from the depth sensor 63 to the subject corresponding to each pixel of the image information of the subject area. The position detection unit 13 identifies the area in which the hand is captured from the image information. The position detection unit 13 acquires the distance to each pixel in the area in which the hand is captured. Based on the distance to each pixel in the area in which the hand is captured, the position detection unit 13 detects the limit position p2 in a three-dimensional space with the center of the head as the center of the coordinate system (step S106).

[0034] The position detection unit 13 outputs the reference position p1 and the limit position p2 to the movement magnification unit 14. The movement magnification unit 14 calculates the distance L2 or distance La between the reference position p1 and the limit position p2 (step S107). The movement magnification unit 14 reads the maximum distance Ra from the memory unit or the like when the robot arm 2 is extended from a predetermined position to the maximum position. The position detection unit 13 calculates the movement magnification X = Ra ÷ L2 or movement magnification X = Ra ÷ La and determines the movement magnification X (step S108). Note that the HMD1 does not distinguish between detecting whether the distance between the reference position p1 and the limit position p2 is L2 or La, and the movement magnification X = Ra ÷ L2 (La) may be calculated using the distance between the reference position p1 and the limit position p2 (distance L2 or distance La). The movement magnification unit 14 outputs the determined movement magnification X to the remote control unit 15. The remote control unit 15 stores the movement magnification X.

[0035] The control unit 11 detects the completion of determining the movement magnification X. The control unit 11 instructs the remote control unit 15 to start the remote control process. The operator inputs the start of operation to the HMD1. The operator extends and retracts their arm forward from the reference position p1 to change the position of their hand. The position detection unit 13 acquires image information, distance information, and motion presence / absence information at predetermined intervals, such as every 0.1 seconds. If the motion presence / absence information indicates motion is present, the position detection unit 13 calculates the current position p4 of the hand corresponding to the pixels in the hand range detected based on the image information and outputs it to the remote control unit 15. The remote control unit 15 calculates the distance L1 from the reference position p1 to the current position p4 of the hand in a 3D spatial coordinate system with the center of the HMD1 or the head wearing the HMD1 as the origin (step S109). The remote control unit 15 calculates the control distance = distance L1 × movement magnification X and determines the control distance (step S110). The remote control unit 15 transmits the control distance to the robot arm 2. The robot arm 2 controls its arm so that the length of the arm becomes the control distance. The remote control unit 15 may remotely control the robot arm 2 by performing steps S109 and S110 at predetermined intervals, such as every 0.1 seconds.

[0036] This processing by the HMD1 allows the operator to extend the robot arm 2 to its maximum distance Ra even when there is an obstacle nearby, by moving the hand within the range of the obstacle.

[0037] In the process described above, HMD1 calculates the control distance of robot arm 2. However, HMD1 may also determine the movement magnification X and calculate the distance L1 from the hand's reference position p1 to its current position p4 in 3D spatial coordinates, and transmit the information of movement magnification X and distance L1 to robot arm 2. The control unit of robot arm 2 may then determine the control distance by calculating the control distance = distance L1 × movement magnification X, and control the extension and retraction of the arm based on that control distance.

[0038] Furthermore, in the remote control system 100, if the HMD1 and the robot arm 2 are connected via a server device, the server device may perform some of the above-described processing. For example, the server device may send and receive control signals to and from the HMD1, acquire image information, distance information, and operation status information from the HMD1, and perform all of the above-described processing steps S101 to S110. Alternatively, the server device may acquire a reference position p1 and a limit position p2, perform the processing in steps S107 and S108, and the robot arm 2 may perform the processing in steps S109 and S110.

[0039] Figure 7 shows the movement of the robot arm. As shown in Figure 7(A), when the movement magnification X=1 and the distance L2(La) is less than the distance Ra, the robot arm 2 can only be extended to L2(La), and it is not possible to remotely operate the robot arm 2 to extend it to the maximum distance Ra. On the other hand, as shown in Figure 7(B), when the movement magnification X=Ra÷L2(La) is 1 or greater, the robot arm 2 can be extended to the maximum distance Ra even if the distance L2(La) is less than the distance Ra, and 100% of the range of motion can be secured without any restrictions on the operation of the robot arm 2.

[0040] The above-described process explained an example where there is an obstacle such as a wall in the forward direction. However, if the operator enters a narrow space, and that space is a closed box with all sides (front, back, left, right, up, and down), the HMD1 may calculate a movement multiplier X for each direction of hand movement (forward, backward, left, right, up, and down) and control the movement of the robot arm 2 in each direction.

[0041] Alternatively, the remote control system 100 may be configured such that the HMD1 communicates with two remote robot arms 2, and the operator moves the position of their left and right hands, allowing the HMD1 to remotely control the two remote robot arms 2 corresponding to the left and right hands. The processing of the HMD1 in this case will be described below.

[0042] Figure 8 shows the reference and limit positions of the left and right hands in three-dimensional spatial coordinates. As described above, the center of the head is used as the origin. HMD1 detects the reference position p1H of the left hand and the reference position p1M of the right hand through the process described above. The wall in front of the operator is perpendicular to the wall to the left or right of the operator, and the wall behind the operator is perpendicular to the wall to the left or right of the operator looking forward, indicating that the operator is inside a rectangular box.

[0043] HMD1 performs the processing from step S103 to step S106 for, for example, four left hand movement directions: forward, backward, left, and right, and calculates the reference position p1H and limit position p2H for the left hand in each movement direction. Similarly, HMD1 performs the processing from step S103 to step S106 for, for example, four right hand movement directions: forward, backward, left, and right, and calculates the reference position p1M and limit position p2M for the right hand in each movement direction. In order to calculate the reference position p1 and limit position p2 for each movement direction, the operator should face each direction of hand movement so that HMD1 can acquire image information and distance information of the hand position in that direction.

[0044] Furthermore, HMD1 determines the movement magnification X for each movement direction of the hand (forward, backward, left, and right) through the processing in steps S107 and S108 for each movement direction. HMD1 determines the hand movement direction based on the hand position through the processing in steps S109 and S110, and determines the control distance for each movement direction using the movement magnification X corresponding to the movement direction. HMD1 may output the movement direction and control distance to the robot arm 2. The robot arm 2 controls the extension and retraction of the arm based on the movement direction and control distance. HMD1 may similarly determine the movement magnification X for the upward and downward directions and use that movement magnification X to determine the control distance for the upward and downward directions.

[0045] The HMD1 may calculate the reference position and limit position in movement directions other than forward, backward, left, right, up, and down by interpolation calculation using one or more reference positions or limit positions in the movement directions.

[0046] For example, suppose the direction of movement of the right hand forms an angle θ with the x-axis. Then, let p2Ma be the limit position of the right hand's forward movement, and p2Mb be the limit position of the right hand's right movement. In this case, let L2Ma be the distance to the limit position p2Ma in the forward movement direction, and L2Mb be the distance to the limit position p2Mb in the right movement direction. HMD1 may calculate the limit position p2Mθ (virtual limit position) when the direction of movement of the right hand forms an angle θ with the x-axis. Using this limit position p2Mθ, the movement multiplier X when the direction of movement forms an angle θ with the x-axis may be calculated.

[0047] Figure 9 shows an overview of how to calculate the distance from the reference position to the limit position of the hand in other directions of movement. As shown in Figure 9, for example, if the direction of movement of the right hand forms an angle θ with the x-axis direction, the movement magnification XMθ must be calculated. Currently, the limit position p2Ma in the forward direction of movement of the right hand and the limit position p2Mb in the right direction of movement of the right hand have been detected. The position detection unit 13 calculates the current hand position p4 from the image information and distance information and outputs it to the movement magnification determination unit 14. The movement magnification determination unit 14 calculates the angle θ formed by the straight line L91 connecting the reference position p1M and the limit position p1Mb in the right direction of movement of the right hand, and the straight line L92 connecting the reference position p1M and the current position p4 of the right hand. The movement magnification determination unit 14 calculates the straight line L93 connecting the limit position p2Ma in the forward direction of movement of the right hand and the limit position p2Mb in the right direction of movement of the right hand. The movement magnification unit 14 identifies the intersection point where the lines L93 and L91 intersect as the virtual limit position p2Mθ when the direction of movement of the right hand forms an angle θ with the x-axis. Although the limit position p2Mθ is not actually the location of a wall, it is set so that when the direction of movement of the right hand forms an angle θ with the x-axis direction, the robot arm 2 can be extended to the maximum distance Ra simply by the operator moving their hand to the limit position p2Mθ. Let L2Mθ be the distance from the reference position p1M of the right hand to the virtual limit position p2Mθ when the direction of movement of the right hand forms an angle θ with the x-axis direction. The movement magnification unit 14 calculates the movement magnification XMθ using the formula Movement Magnification XMθ = Ra ÷ L2Mθ. By a similar process, the movement magnification unit 14 can calculate the movement magnification XHθ when the direction of movement of the left hand forms an angle θ with the x-axis direction.

[0048] In the remote control of the robot arm 2 in the moving direction forming an angle θ with the x-axis direction, HMD1 may transmit the value of θ to the robot arm 2. The robot arm 2 may control the expansion and contraction of the angle and distance from the reference direction of the arm using the value of θ and the control distance. The calculation process of the limit position p2Mθ shown in FIG. 9 is an example of a process in which the position detection unit 13 calculates the limit positions in other moving directions where the limit positions have not been calculated by interpolation using the limit positions in a plurality of moving directions where the limit positions have been specified. The process of specifying the movement magnification XHθ is an example of a process in which the movement magnification specifying unit 14 specifies the movement magnification in the case of remotely operating the target device by moving the hand from the reference position to the limit positions in other moving directions where the limit positions have not been specified based on the reference position and the limit positions calculated for other moving directions.

[0049] FIG. 10 is a second diagram showing the relationship between the distance from the reference position and the movement magnification. In the above example, the case where the movement magnification X is a constant Ra÷L2 until the position of the hand touches the obstacle at the distance L2 has been described, but the movement magnification may be set in two steps. For example, when the hand is at a position up to a distance L1 close to the operator's body, the movement magnification may be set to X1, and when the distance to the position of the hand is longer than L1 and up to L2, the movement magnification X2 may be set. In this case, the control unit 11 outputs guidance instructing the hand to move to a position up to the distance L1 for controlling the robot arm 2 at the movement magnification X1. Then, the position detection unit 13 detects the position of the hand corresponding to the distance L1 (L1 < L2) in the same manner as the reference position p1 and the limit position p2. The movement magnification X1 at the position of the hand up to the distance L1 may be determined in advance. It is assumed that the movement magnification X1 is a value within the range indicating X(Ra÷L2)>X1≧0 described above. Also, the movement magnification specifying unit 14 calculates the movement magnification X2 at the position of the hand from the distance L1 to the distance L2 using the calculation formula X2=(Ra - X1×L1)÷(L2 - L1). As a result, the movement magnification X1 at each position of the hand from the reference position to the distance L1 becomes smaller than the movement magnification X2, and the expansion and contraction operation control of the robot arm 2 can be performed such that the position closer to L1 moves relatively slowly and the positions from the distance L1 to the distance L2 move relatively faster.

[0050] When using the movement magnifications X1 and X2 in Figure 10, assume that the distance Ra = 100 cm, distance L1 = 10 cm, distance L2 = 40 cm, and the set movement magnification X1 = 1.5 times. In this case, when L1 < movement distance ≤ L2, the movement magnification X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 85 ÷ 30 = 2.8 times (rounded to the second decimal place).

[0051] Figure 11 is a third figure showing the relationship between the distance from the reference position and the magnification factor. As shown in Figure 11, the movement multiplier X1 to distance L1 may be greater than the movement multiplier X2 from distance L1 to distance L2. In this case, the value of movement multiplier X1 is less than or equal to Ra÷L1 and greater than Ra÷L2. Let's assume that the distance Ra = 100 cm, distance L1 = 10 cm, distance L2 = 40 cm, and the set movement multiplier X1 = 6.7 times. In this case, the movement multiplier X2 for L1 < movement distance ≤ L2 is X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 33 ÷ 30 = 1.1 times.

[0052] Figure 12 is the fourth figure, showing the relationship between the distance from the reference position and the magnification factor. Note that if the movement magnification X1 = X2, that is, if the movement magnification X does not change from the reference position p1 of the hand to the limit position p2, and the distance Ra = 100 cm, distance L1 = 10 cm, distance L2 = 50 cm, and movement magnification X1 = 2, then Ra ÷ L2 = 2 times. Similarly, if calculated using the formula X2 = (Ra - X1 × L1) ÷ (L2 - L1), then X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 80 ÷ 40 = 2 times.

[0053] Figure 13 is the fifth figure showing the relationship between the distance from the reference position and the magnification factor. The movement magnification unit 14 may, depending on the setting of the movement magnification X1 up to distance L1, adjust the robot arm 2 so that its extension reaches the maximum distance Ra by the time the operator extends their arm and reaches distance L1. For example, suppose the operator sets the movement magnification X1 to 10 and inputs it to the HMD1 when distance Ra = 100 cm, distance L1 = 10 cm, and distance L2 = 40 cm. In this case, X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 0 ÷ 30 = 0. Furthermore, once the hand reaches distance L1, the robot arm 2 can be operated so that its extension reaches the maximum distance Ra.

[0054] Figure 14 is the sixth figure showing the relationship between the distance from the reference position and the magnification factor. The operator may set the movement multiplier X1 to 0x in the HMD1 when moving to distance L1. For example, suppose the operator sets the movement multiplier X1 to 0 and inputs it to the HMD1 when the distance Ra=100cm, distance L1=10cm, and distance L2=40cm. In this case, X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 100 ÷ 30 = 3.33x. This processing in the HMD1 allows for control so that the robot arm 2 cannot extend or retract until the hand is moved to distance L1.

[0055] According to the HDM1 processing described above, in a technology that remotely controls a robot arm 2 based on the movement of body parts such as hands and feet, the robot arm 2 can be extended and retracted without restriction even when there are obstacles near the operator, thereby improving the operability of the robot arm 2.

[0056] Figure 15 shows the functional blocks of an information processing device with a different configuration. Figure 16 shows the processing flow of an information processing device with a different configuration.

[0057] As shown in Figure 15, the information processing device may include a position detection unit 13 and a movement magnification specification unit 14. When remotely controlling a body part to be moved to extend the tip of the arm of a target device from a predetermined position to its maximum position, the position detection unit 13 detects a reference position p1 of the body part to be moved that corresponds to the state where the tip of the arm is in the predetermined position, and a limit position p2 that indicates the position where the body part has been moved from the reference position p1 within the range in which the body part to be moved can be moved (step S201). The movement magnification unit 14 determines the movement magnification X when remotely controlling the arm of the target device by moving the body part to be moved from a reference position p1 to a limit position p2, based on the maximum distance Ra when the arm of the target device is extended from a predetermined position to its maximum position and the distance L2 from the reference position p1 to the limit position p2 (step S203).

[0058] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0059] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0060] (Note 1) When remotely controlling a body part to be moved to extend the tip of an arm of a target device from a predetermined position to its maximum position, a position detection means detects a reference position of the body part to be moved that corresponds to the state in which the tip of the arm is at the predetermined position, and a limit position indicating the position of the body part moved from the reference position within the range in which the body part to be moved can be moved. Movement magnification determination means for remotely controlling the arm of a target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position, An information processing device equipped with the following features.

[0061] (Note 2) The part of the body that is subjected to the action is the hand. The information processing device described in Appendix 1.

[0062] (Note 3) The position detection means detects the reference position and the limit position using sensing information acquired from a sensor attached to the head. The information processing device described in Appendix 1 or Appendix 2.

[0063] (Note 4) The aforementioned reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when remotely controlling the target device. An information processing device described in any one of the appendices 1 to 3.

[0064] (Note 5) The position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, The movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement. An information processing device described in any one of the appendices 1 through 4.

[0065] (Note 6) The position detection means uses the sensing information to detect the position where the body part touches an obstacle provided within the range of movement of the body part to be moved, and identifies that position as the limit position. The information processing device described in Appendix 3.

[0066] (Note 7) The position detection means is At a predetermined timing after the output of guidance instructing the part to be operated to move to the aforementioned reference position, the sensing information is used to detect the reference position. The limit position is detected using the sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the limit position. The information processing device described in Appendix 3 or Appendix 6.

[0067] (Note 8) The aforementioned direction of movement is one or more of the following directions of movement for the body: forward, backward, left, right, up, or down. The information processing device described in Appendix 5.

[0068] (Note 9) The position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using the limit positions of multiple movement directions for which limit positions have been identified. The movement magnification determination means determines the movement magnification when remotely controlling the target device by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit position calculated for the other movement direction. The information processing device described in Appendix 5 or Appendix 8.

[0069] (Note 10) Remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification ratio, An information processing device described in any one of the appendices 1 to 9, comprising the features of:

[0070] (Note 11) When remotely controlling a body part to move and extend the tip of the arm of a target device from a predetermined position to its maximum position, the system detects a reference position of the body part to be moved that corresponds to the state where the tip of the arm is in the predetermined position, and a limit position indicating the position of the body part moved from the reference position within the range in which the body part to be moved can be moved. The movement ratio when remotely controlling the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position is determined based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position. Information processing methods.

[0071] (Note 12) The part of the body that is subjected to the action is the hand. The information processing method described in Appendix 11.

[0072] (Note 13) The reference position and the limit position are detected using sensing information acquired from sensors attached to the head. The information processing method described in Appendix 11 or Appendix 12.

[0073] (Note 14) The aforementioned reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when remotely controlling the target device. The information processing method described in any one of the appendices 11 to 13.

[0074] (Note 15) The reference position and the limit position are detected according to the direction of movement of the part to be operated. Based on the reference position and the limit position corresponding to the direction of movement, the movement magnification ratio corresponding to the direction of movement is determined. The information processing method described in any one of the appendices 11 to 14.

[0075] (Note 16) Using the sensing information, the system detects the position where the body part touches an obstacle located within the range of movement of the body part to be operated, and identifies that position as the limit position. The information processing method described in Appendix 13.

[0076] (Note 17) At a predetermined timing after the output of guidance instructing the part to be operated to move to the aforementioned reference position, the sensing information is used to detect the reference position. The limit position is detected using the sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the limit position. The information processing method described in Appendix 13 or Appendix 16.

[0077] (Note 18) The aforementioned direction of movement is one or more of the following directions of movement for the body: forward, backward, left, right, up, or down. The information processing method described in Appendix 15.

[0078] (Note 19) The limit positions in other movement directions for which the aforementioned limit positions have not been calculated are calculated by interpolation calculation using the limit positions of multiple movement directions for which limit positions have been identified. Based on the aforementioned reference position and the limit position calculated for the other movement direction, the movement ratio is determined when the part is moved from the reference position to the limit position for the other movement direction to remotely control the target device. The information processing method described in Appendix 15 or Appendix 18.

[0079] (Note 20) The movement magnification is used to remotely control the target device by moving the part within the range from the reference position to the limit position. The information processing method described in any one of the appendices 11 to 19.

[0080] (Note 21) The computer of the information processing device, When performing remote control to extend the tip of the arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, a position detection means for detecting a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position, and a limit position indicating the position of the part of the body that has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved. Movement magnification determination means for remotely controlling the arm of a target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position. A program that makes it function as such.

[0081] (Note 22) The part of the body that is subjected to the action is the hand. The program described in Appendix 21.

[0082] (Note 23) The position detection means detects the reference position and the limit position using sensing information acquired from a sensor attached to the head. The program described in Appendix 21 or Appendix 22.

[0083] (Note 24) The aforementioned reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when remotely controlling the target device. The program described in any one of the appendices 21 to 23.

[0084] (Note 25) The position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, The movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement. The program described in any one of the appendices 21 through 24.

[0085] (Note 26) The position detection means uses the sensing information to detect the position where the body part touches an obstacle provided within the range of movement of the body part to be moved, and identifies that position as the limit position. The program described in Appendix 23.

[0086] (Note 27) The position detection means is At a predetermined timing after the output of guidance instructing the part to be operated to move to the aforementioned reference position, the sensing information is used to detect the reference position. The limit position is detected using the sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the limit position. The program described in Appendix 23 or Appendix 26.

[0087] (Note 28) The aforementioned direction of movement is one or more of the following directions of movement for the body: forward, backward, left, right, up, or down. The program described in Appendix 25.

[0088] (Note 29) The position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using the limit positions of multiple movement directions for which limit positions have been identified. The movement magnification determination means determines the movement magnification when remotely controlling the target device by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit position calculated for the other movement direction. The program described in Appendix 25 or Appendix 28.

[0089] (Note 30) Remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification ratio, A program described in any one of the appendices 21 to 29, which includes the following: [Explanation of Symbols]

[0090] 1. Head-mounted display (information processing device) 2. Robot arm 11. Control Unit 12... Acquisition part 13. Position detection unit 14...Movement magnification specifying section 15. Remote Control Unit 61...Camera 62. Motion sensor 63. Depth sensor 101···CPU 102···ROM 103...RAM 104...Storage device 105...Communication module 106... Sensing device 107...Input device 108... Monitor 109...Speaker

Claims

1. When performing remote control to move the tip of the arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, a position detection means for detecting a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is the predetermined position, and a limit position indicating the position of the part of the body that has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved, Movement magnification determination means for remotely controlling the arm of a target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position, An information processing device equipped with the following features.

2. The part of the body that is subjected to the action is the hand. The information processing apparatus according to claim 1.

3. The position detection means detects the reference position and the limit position using sensing information acquired from a sensor attached to the head. The information processing apparatus according to claim 2.

4. The aforementioned reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when remotely controlling the target device. The information processing apparatus according to claim 3.

5. The position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, The movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement. The information processing apparatus according to claim 4.

6. The position detection means uses the sensing information to detect the position where the body part touches an obstacle provided within the range of movement of the body part to be moved, and identifies that position as the limit position. The information processing apparatus according to claim 5.

7. The position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using the limit positions of multiple movement directions for which limit positions have been identified. The movement magnification determination means determines the movement magnification when remotely controlling the target device by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit position calculated for the other movement direction. The information processing apparatus according to claim 6.

8. Remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification ratio, An information processing apparatus according to any one of claims 1 to 7, comprising

9. When remotely controlling a body part to move and extend the tip of the arm of a target device from a predetermined position to its maximum position, the system detects a reference position of the body part to be moved that corresponds to the state where the tip of the arm is in the predetermined position, and a limit position indicating the position of the body part moved from the reference position within the range in which the body part to be moved can be moved. The movement ratio when remotely controlling the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position is determined based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position. Information processing methods.

10. The computer of the information processing device, When performing remote control to extend the tip of an arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, a position detection means for detecting a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position, and a limit position indicating the position of the part of the body that has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved. Movement magnification determination means for remotely controlling the arm of a target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position. A program that makes it function as such.

Images