Information processing device, information processing method, and program

Abstract

Provided is an information processing device that improves operability during remote operation of an operation target in a technique for moving a part such as a hand or a foot of a body and remotely operating the operation target on the basis of the movement of the part. When performing a remote operation for extending the distal end of an arm of a target device from a prescribed position to a maximum position by moving the part that serves as a motion target of the body, a reference position of the part that serves as the motion target of the body corresponding to a state in which the distal end of the arm is at the prescribed position, and a limit position indicating a position where the part that serves as the motion target of the body is moved from the reference position within a range in which the part that serves as the motion target can be moved, are detected. Movement magnification in cases in which the arm of the target device is remotely operated by moving the part that serves as the motion target of the body from the reference position to the limit position is specified on the basis of the maximum distance when the arm of the target device is extended from the prescribed position to the maximum position and the distance from the reference position to the limit position.

Description

Information Processing Apparatus, Information Processing Method, Program

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

[0002] There is a technology for remotely operating a remotely located operation target. For example, Patent Document 1 discloses a control device including a slave manipulator installed to access an operative field and a master manipulator installed within an area where an operator can operate and remotely operating the slave manipulator.

[0003] Japanese Patent Application Laid-Open No. 2001-150368

[0004] By the way, when operating a remote operation target, there is a need for a technology to move a part such as a hand or a foot of the body and remotely operate the operation target based on the movement of the part. And in this remote operation, it is required to improve the operability when remotely operating the operation target.

[0005] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and a program that solve the above-described problems.

[0006] An information processing apparatus according to an aspect of the present disclosure includes: position detection means for detecting a reference position of a part of the body that becomes an operation target when performing a remote operation of extending the tip of an arm of a target device from a predetermined position to a maximum position by moving the part of the body that becomes an operation target, and a limit position indicating a position where the part is moved from the reference position within a range in which the part of the body that becomes an operation target can be moved, corresponding to a state where the tip of the arm is at the predetermined position; and movement magnification specifying means for specifying a movement magnification in a case of remotely operating the arm of the target device by moving the part of the body that becomes an operation target from the reference position to the limit position, based on a maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and a distance from the reference position to the limit position.

[0007] An information processing method according to one aspect of the present disclosure, when performing remote operation to extend the tip of an arm of a target device from a predetermined position to its maximum position by moving a part of the body that is the target of movement, detects a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position, and a limit position indicating the position in which the part has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved, and determines the movement ratio when remotely operating the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0008] A program according to one aspect of the present disclosure causes the computer of an information processing device to function as a position detection means for detecting a reference position of the body part to be moved, corresponding to the state in which the tip of the arm of the target device is at the predetermined position, and a limit position indicating the position in which the body part has been moved from the reference position within the range in which the body part to be moved can be moved, when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, and a movement ratio determination means for determining the movement ratio when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0009] According to one embodiment described above, in a technology that remotely controls an object based on the movement of body parts such as hands and feet, the operability of the object during remote control can be improved.

[0010] This diagram shows the remote control system related to this disclosure. This diagram shows the hardware configuration of the HMD related to this disclosure. This is a functional block diagram of the HMD related to this disclosure. This diagram explains the reference position and limit position related to this disclosure. This is the first diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This diagram shows the processing flow of the HMD related to this disclosure. This diagram shows the operation of the robot arm related to this disclosure. This diagram shows the operation of the robot arm related to this disclosure. This diagram shows the reference positions of the left and right hands related to this disclosure. This diagram shows an overview of how the distance from the reference position to the limit position of the hand related to this disclosure is calculated. This is the second diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This is the third diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This is the fourth diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This is the fifth diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This is the sixth diagram showing the relationship between the distance from the reference position and the movement magnification related to this disclosure. This diagram shows the functional block of an information processing device in a different configuration related to this disclosure. This diagram shows the processing flow of an information processing device in a different configuration related to this disclosure.

[0011] The information processing device of this disclosure will be described below with reference to the drawings. Figure 1 is a diagram showing a remote control system including the information processing device of this disclosure. The remote control system 100 comprises a head-mounted display (hereinafter referred to as HMD) 1 and a robot arm 2 that is the target of remote control. The HMD 1 and the robot arm 2 are connected by communication via a communication network. In this disclosure, the HMD 1 is one embodiment of the information processing device. In this disclosure, the robot arm 2 is one embodiment of the arm that is the target of remote control.

[0012] HMD1 is worn on the head of an operator who remotely controls the robot arm 2. Based on sensing information such as images, HMD1 remotely controls the extension and retraction of the robot arm 2 by detecting when the position of the operator's hand, an example of a body part, moves from a reference position close to the body to a limit position in a certain direction of movement. MHD1 also remotely controls the selection of two actuator fingers that constitute the gripping mechanism of the hand part at the tip of the robot arm 2 by detecting when the operator's index finger and thumb are brought together or separated, based on sensing information such as images.

[0013] In this disclosure, the case where the body part that the operator moves when controlling the robot arm 2 is the hand or arm is described. However, in other disclosures, the body part that the operator moves when controlling the robot arm 2 may be the foot or leg.

[0014] In other disclosures, a server device may be connected to the communication between the HMD1 and the robot arm2, so that the HMD1 and the robot arm2 communicate via the server device. In this case, the server device relays the signals between the HMD1 and the robot arm2. The server device may also perform some of the processing for the functions of the HMD1.

[0015] Figure 2 shows the hardware configuration of the HMD. HMD1 is a computer equipped with hardware such as a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, storage device 104, communication module 105, sensing device 106, input device 107, monitor 108, and speaker 109, as shown in Figure 2. The sensing device 106 may include a camera 61, motion sensor 62, and depth sensor 63. The camera 61 generates and outputs an image, which is an example of sensing information. The motion sensor 62 generates and outputs motion presence / absence information, which is an example of sensing information, indicating the presence or absence of human movement. The depth sensor 63 generates and outputs distance information, indicating the distance to each position of the subject. HMD1 may further include other hardware such as a GPU instead of the CPU 101.

[0016] Figure 3 is a functional block diagram of the HMD. The HMD1 performs at least the functions of the control unit 11, acquisition unit 12, position detection unit 13, movement magnification specification unit 14, and remote control unit 15 when the CPU 101 or GPU executes a program.

[0017] The control unit 11 controls other functional units. The acquisition unit 12 acquires information for processing the HMD1. The position detection unit 13 detects a reference position of the hand corresponding to the initial state where the tip of the robot arm 2 is in the predetermined position, and a limit position indicating the position where the hand has been moved from the reference position within the range in which the hand can be moved (extended), when the operator moves their hand to extend the tip of the robot arm 2 from a predetermined position to its maximum position.

[0018] The movement magnification unit 14 determines the movement magnification when remotely controlling the robot arm 2 by moving the hand from a reference position to a limit position, based on either the maximum distance Ra when the tip of the robot arm 2 is extended from an initial predetermined position to the maximum position, the distance La from the reference position of the hand to the limit position when the arm is extended to the maximum extent by the person, or the distance L2 from the reference position of the hand to the limit position when the arm is not extended to the maximum extent by the person and hits an obstacle. The remote control unit 15 remotely controls the robot arm 2 using the movement magnification when the operator remotely controls the robot arm 2 by moving the hand.

[0019] Figure 4 illustrates the reference position and the limit position. As shown in Figure 4, we assume a situation where there is an obstacle such as a wall near the operator. The wall is assumed to be closer to the operator's body than the position the operator can reach with their hand extended as far as possible. Furthermore, the state in which the operator brings their hand as close to their body as possible is defined as the hand being at the reference position p1.

[0020] Within the range a in which the operator can move their hand, the state in which the hand is at its limit position p2 is defined as the state in which the hand is at its limit position p2 (or the state in which the hand can be moved to the point just before it touches an obstacle such as a wall when the arm is extended). The distance from the reference position p1 to the limit position p2 is defined as L2. Ra is defined as the maximum distance from the tip position of the robot arm 2 when it is shortened to a predetermined length to the tip position of the arm when it is extended to its maximum length.

[0021] The following explanation assumes that the HMD1 remotely controls the movement of the tip of the robot arm 2 based on the detection of sensing information such as images indicating that the operator's hand position has moved. If there are no obstacles such as walls near the operator, and the length of the operator's arm (the length to the fingertips when the arm is extended) is equal to or greater than the maximum distance Ra when the robot arm 2 is extended, the operator can extend the robot arm 2, which extends and retracts at a distance equal to the distance to the hand at the tip of the arm, to the maximum distance Ra by extending the arm to the maximum distance Ra. However, if there are obstacles such as walls near the operator, the operator cannot extend their arm and move their hand to the position of distance Ra. Also, if the length of the operator's arm is shorter than the length of the robot arm 2, the operator cannot extend their arm and move their hand to the position of distance Ra. The HMD1 according to this disclosure calculates a movement multiplier so that the robot arm 2 extends to the maximum distance Ra when the operator extends their arm and moves their hand to the limit position p2, even when there is an obstacle such as a wall near the operator or when the length of the operator's arm is shorter than the length of the robot arm 2, and controls the extension and retraction of the robot arm 2 using this movement multiplier. Let L1 be the distance from the operator to the position of their hand when the operator extends their arm to a position closer to the operator than the distance L2 to the obstacle. In this case, the HMD1 may control the movement multiplier of the robot arm 2 corresponding to the movement of the hand until the hand reaches position L1 to be a different multiplier than the movement multiplier of the robot arm 2 corresponding to the movement of the hand from L1 to position Ra.

[0022] The HMD1 instructs the user to place their hand at the reference position p1 through voice guidance output from the speaker 109 or text guidance output to the monitor 108 of the HMD1. At a predetermined timing after the output of the guidance instructing the user to place their hand at the reference position p1, the HMD1 can detect the reference position p1 of the hand using image information acquired from the camera 61 and distance information to the subject (hand) acquired from the depth sensor 63.

[0023] The HMD1 instructs the user to place their hand at the limit position p2 through voice guidance output from the speaker 109 or text guidance output to the monitor 108 of the HMD1. At a predetermined timing after the output of the guidance instructing the user to place their hand at the limit position p2, the HMD1 can detect the limit position p2 of the hand using image information acquired from the camera 61 and distance information to the subject (hand) acquired from the depth sensor 63.

[0024] In this disclosure, limit position p2 is defined as the position where the operator reaches out and touches the wall. However, limit position p2 could also be defined as the position where a person with a disability who has difficulty reaching out can reach as far as possible. Alternatively, limit position p2 could be defined as the position where a person without obstacles can reach as far as possible. In other words, in other examples of this disclosure, a situation without obstacles may be assumed.

[0025] The HMD1 in this disclosure detects a reference position p1 and a limit position p2, and calculates the movement ratio of the tip of the robot arm 2 when remotely controlling the extension and retraction of the robot arm 2, which is the target device for remote control, by moving the part of the operator's body that is the target of the operation (hand) from the reference position p1 to the limit position p2. The movement ratio is determined based on the maximum distance Ra when the robot arm 2 is extended from a predetermined position to the maximum position, and the distance L2 from the reference position p1 of the operator's hand to the limit position p2 (when there is an obstacle) or the distance La from the reference position p1 of the operator's hand to the limit position p2 (when there is no obstacle). As an example when there is an obstacle, the HMD1 calculates the movement ratio using the formula Movement Ratio = Ra ÷ L2. As an example when there is no obstacle, the HMD1 calculates the movement ratio using the formula Movement Ratio = Ra ÷ La. The value of the distance Ra is stored in advance by the HMD1.

[0026] Figure 5 is the first diagram showing the relationship between the distance from the reference position and the movement multiplier. As an example, if there is an obstacle, HMD1 determines the movement multiplier calculated by the value Ra÷L2, as shown in Figure 5. If there is no obstacle, HMD1 determines the movement multiplier calculated by the value Ra÷La, as shown in Figure 5. HMD1 calculates the control distance by multiplying the distance L1 from the reference position of the hand by the movement multiplier and outputs this control distance to the robot arm 2. Based on the acquired control distance, the robot arm 2 operates so that the distance from a predetermined position before the start of operation to the position of the arm tip (or the distance from the initial position of the robot arm 2 to the extended position) becomes the control distance. As a result, when the operator extends their arm and reaches the limit position p2 from the reference position p1 of the hand, and the distance from the reference position p1 to the hand becomes L2 or distance La, the distance from the predetermined position of the arm tip of the robot arm 2 becomes Ra, and it can be extended to its maximum extent.

[0027] Figure 6 is a diagram showing the processing flow of the HMD. The processing flow of HMD1 will be explained step by step using Figure 6. First, the user inputs the start of operation of the robot arm 2 to HMD1 using the input device 107 (step S101). The control unit 11 of HMD1 receives the instruction to start operation. The control unit 11 outputs a sensing instruction to the sensing device 106. As a result, the camera 61, motion sensor 62, and depth sensor 63 of the sensing device 106 start to operate.

[0028] The control unit 11 starts the calibration process. As an example, in this calibration process, the control unit 11 determines whether or not it has received an input to start setting the movement magnification (step S102).

[0029] When the control unit 11 detects that it has received an input to start setting the movement magnification, it outputs first guidance from the monitor 108 or speaker 109 instructing the operator to place the hand at the reference position (step S103). Based on the output of the first guidance, the operator recognizes that the position of the hand when starting to operate the robot arm 2 should be moved to a position close to the body. The operator moves the hand to the position that will be the starting position for operating the robot arm 2 close to the body and inputs an instruction to the input device 107 that the hand movement is complete. The input device 107 may be a predetermined button provided on the HMD1. The control unit 11 detects that the hand movement is complete and instructs the position detection unit 13 to detect the reference position.

[0030] The position detection unit 13 acquires image information of the subject area from the camera 61 and distance information from the depth sensor 63 to the subject corresponding to each pixel of the image information of the subject area. The position detection unit 13 identifies the area in which the hand is captured from the image information. The position detection unit 13 acquires the distance to each pixel in the area in which the hand is captured. Based on the distance to each pixel in the area in which the hand is captured, the position detection unit 13 detects a reference position p1 in a three-dimensional space with the center of the head as the center of the coordinate system (step S104).

[0031] The coordinate system of the position detected by the HMD1 in this disclosure may be centered on the position of the HMD1. The image information and distance information may be the image information and distance information when the operator wearing the HMD1 visually perceives the direction of the position of the hand. The HMD1 has a function that can determine the orientation of the operator's face using a gyro sensor and a geomagnetic sensor, and it is assumed that the coordinates of the hand in three-dimensional space can be detected using known technology by using these functions and distance information to the hand.

[0032] The control unit 11 detects that the detection of the reference position p1 is complete. The control unit 11 outputs a second guidance from the monitor 108 or speaker 109 instructing the operator to place their hand at the limit position p2 (step S105). Based on the output of the second guidance, the operator moves their hand to a position where it will touch the obstacle in front of them (or a position just before the hand touches the obstacle) and inputs an instruction to the input device 107 that the hand movement is complete. The control unit 11 detects that the hand movement is complete and instructs the position detection unit 13 to detect the limit position p2.

[0033] The position detection unit 13 acquires image information of the area to be photographed from the camera 61 and acquires distance information from the depth sensor 63 to the subject corresponding to each pixel of the image information of the area to be photographed. The position detection unit 13 identifies the area in which the hand is photographed from the image information. The position detection unit 13 acquires the distance to each pixel in the area in which the hand is photographed. Based on the distance to each pixel in the area in which the hand is photographed, the position detection unit 13 detects the limit position p2 in a three-dimensional space with the center of the head as the center of the coordinate system (step S106).

[0034] The position detection unit 13 outputs a reference position p1 and a limit position p2 to the movement magnification unit 14. The movement magnification unit 14 calculates the distance L2 or distance La between the reference position p1 and the limit position p2 (step S107). The movement magnification unit 14 reads the maximum distance Ra when the robot arm 2 is extended from a predetermined position to the maximum position from the storage unit or the like. The position detection unit 13 calculates the movement magnification X = Ra ÷ L2 or movement magnification X = Ra ÷ La to determine the movement magnification X (step S108). Note that the HMD1 does not distinguish and detect whether the distance between the reference position p1 and the limit position p2 is L2 or La, and the movement magnification X = Ra ÷ L2 (La) may be calculated using the distance between the reference position p1 and the limit position p2 (distance L2 or distance La). The movement magnification unit 14 outputs the determined movement magnification X to the remote control unit 15. The remote control unit 15 stores the movement magnification X.

[0035] The control unit 11 detects the completion of determining the movement magnification X. The control unit 11 instructs the remote control unit 15 to start the remote control process. The operator inputs the start of operation to the HMD 1. The operator extends and retracts their arm forward from the reference position p1 to change the position of their hand. The position detection unit 13 acquires image information, distance information, and motion presence / absence information at predetermined intervals, such as every 0.1 seconds. If the motion presence / absence information indicates motion is present, the position detection unit 13 calculates the current position p4 of the hand corresponding to the pixels in the hand range detected based on the image information and outputs it to the remote control unit 15. The remote control unit 15 calculates the distance L1 from the reference position p1 to the current position p4 of the hand in a three-dimensional spatial coordinate system with the center of the HMD 1 or the head wearing the HMD 1 as the origin (step S109). The remote control unit 15 calculates the control distance = distance L1 × movement magnification X and determines the control distance (step S110). The remote control unit 15 transmits the control distance to the robot arm 2. The robot arm 2 controls its arm so that the length of the arm becomes the control distance. The remote control unit 15 may remotely control the robot arm 2 by performing steps S109 and S110 at predetermined intervals, such as every 0.1 seconds.

[0036] This processing by the HMD1 allows the robot arm 2 to be extended to its maximum distance Ra even when there is an obstacle near the operator, by moving the hand within the range up to the obstacle.

[0037] In the process described above, the HMD1 calculates the control distance of the robot arm 2. However, the HMD1 may also determine the movement magnification X and calculate the distance L1 from the hand's reference position p1 to its current position p4 in three-dimensional spatial coordinates, and transmit the information of the movement magnification X and distance L1 to the robot arm 2. The control unit of the robot arm 2 may then determine the control distance by calculating the control distance = distance L1 × movement magnification X, and control the extension and retraction of the arm based on that control distance.

[0038] Furthermore, in the remote control system 100, if the HMD 1 and the robot arm 2 are connected via a server device, the server device may perform some of the above-described processing. For example, the server device may send and receive control signals to and from the HMD 1, acquire image information, distance information, and operation status information from the HMD 1, and perform all of the processing in steps S101 to S110. Alternatively, the server device may acquire a reference position p1 and a limit position p2, perform the processing in steps S107 and S108, and the robot arm 2 may perform the processing in steps S109 and S110.

[0039] Figures 7A and 7B show the operation of the robot arm. As shown in Figure 7A, when the movement magnification X = 1 and the distance L2(La) is less than the distance Ra, the robot arm 2 can only be extended to L2(La), and it is not possible to remotely operate the robot arm 2 to extend it to the maximum distance Ra. On the other hand, as shown in Figure 7B, when the movement magnification X = Ra ÷ L2(La) is 1 or greater, the robot arm 2 can be extended to the maximum distance Ra even if the distance L2(La) is less than the distance Ra, and 100% of the range of motion can be secured without any restrictions on the operation of the robot arm 2.

[0040] The above-described process explained an example where there is an obstacle such as a wall in the forward direction. However, if the operator enters a narrow space, and that space is a closed box with the front, back, left, right, up, and down sides enclosed, the HMD1 may calculate the movement multiplier X for each direction of hand movement (forward, backward, left, right, up, and down) and control the movement of the robot arm 2 in each direction.

[0041] Alternatively, the remote control system 100 may be configured such that the HMD1 communicates with two remote robot arms 2, and the operator moves the position of their left and right hands, allowing the HMD1 to remotely control the two remote robot arms 2 corresponding to the left and right hands. The processing of the HMD1 in this case will be described below.

[0042] FIG. 8 is a diagram showing the reference positions and limit positions of the left and right hands in three-dimensional space coordinates. As described above, the center of the head is taken as the origin. The HMD 1 detects the reference position p1H of the left hand and the reference position p1M of the right hand by the above-described process. The front wall near the operator is orthogonal to the left or right wall of the operator, the rear wall is orthogonal to the left or right wall of the operator looking forward, and the operator is shown to be in a rectangular box.

[0043] The HMD 1 performs the processes from step S103 to step S106, for example, for the four moving directions of the left hand, namely, forward, backward, left, and right, and calculates the reference position p1H and the limit position p2H of the left hand for each moving direction. Similarly, the HMD 1 performs the processes from step S103 to step S106, for example, for the four moving directions of the right hand, namely, forward, backward, left, and right, and calculates the reference position p1M and the limit position p2M of the right hand for each moving direction. In calculating the reference position p1 and the limit position p2 for each moving direction, the HMD 1 may acquire the image information and distance information of the position of the hand in that direction while the operator faces each moving direction of the hand.

[0044] Further, the HMD 1 specifies the movement magnification X for each of the forward, backward, left, and right moving directions of the hands by the processes of steps S107 and S108 for each moving direction. The HMD 1 specifies the moving direction of the hand based on the position of the hand by the processes of steps S109 and S110, and specifies the control distance for each moving direction using the movement magnification X corresponding to the moving direction. The HMD 1 may output the moving direction and the control distance to the robot arm 2. The robot arm 2 controls the expansion and contraction of the arm based on the moving direction and the control distance. The HMD 1 may also specify the movement magnification X in the same manner for the upward and downward directions, and specify the control distance for the upward and downward directions using the movement magnification X.

[0045] The HMD 1 may calculate the reference position and the limit position in a moving direction different from the moving directions of forward, backward, left, right, up, and down by interpolation calculation using one or more reference positions or limit positions among the moving directions.

[0046] For example, suppose the direction of movement of the right hand forms an angle θ with the x-axis. Then, let p2Ma be the limit position of the right hand's forward movement, and p2Mb be the limit position of the right hand's right movement. In this case, let L2Ma be the distance to the limit position p2Ma in the forward movement direction, and L2Mb be the distance to the limit position p2Mb in the right movement direction. HMD1 may calculate, for example, the limit position p2Mθ (virtual limit position) when the direction of movement of the right hand forms an angle θ with the x-axis. Using this limit position p2Mθ, the movement magnification X when the direction of movement forms an angle θ with the x-axis may be calculated.

[0047] Figure 9 shows an overview of how to calculate the distance from the reference position to the limit position of the hand in other directions of movement. As shown in Figure 9, for example, if the direction of movement of the right hand is in a direction that forms an angle θ with the x-axis direction, the movement magnification XMθ must be calculated. Currently, the limit position p2Ma in the forward direction of movement of the right hand and the limit position p2Mb in the right direction of movement of the right hand have been detected. The position detection unit 13 calculates the current hand position p4 from the image information and distance information and outputs it to the movement magnification determination unit 14. The movement magnification determination unit 14 calculates the angle θ formed by the straight line L91 connecting the reference position p1M and the limit position p1Mb in the right direction of movement of the right hand, and the straight line L92 connecting the reference position p1M and the current position p4 of the right hand. The movement magnification determination unit 14 calculates the straight line L93 connecting the limit position p2Ma in the forward direction of movement of the right hand and the limit position p2Mb in the right direction of movement of the right hand. The movement magnification unit 14 identifies the intersection point where the lines L93 and L91 intersect as the virtual limit position p2Mθ when the direction of movement of the right hand forms an angle θ with the x-axis. Although the limit position p2Mθ is not actually the location of a wall, it is set so that when the direction of movement of the right hand forms an angle θ with the x-axis direction, the operator can extend the robot arm 2 to the maximum distance Ra simply by moving their hand to the limit position p2Mθ. Let L2Mθ be the distance from the reference position p1M of the right hand to the virtual limit position p2Mθ when the direction of movement of the right hand forms an angle θ with the x-axis direction. The movement magnification unit 14 calculates the movement magnification XMθ using the formula Movement Magnification XMθ = Ra ÷ L2Mθ. By a similar process, the movement magnification unit 14 can calculate the movement magnification XHθ when the direction of movement of the left hand forms an angle θ with the x-axis direction.

[0048] In the remote control of the robot arm 2 in the moving direction forming an angle θ with the x-axis direction, the HMD 1 may transmit the value of θ to the robot arm 2. The robot arm 2 may control the expansion and contraction of the angle and distance from the reference direction of the arm using the value of θ and the control distance. The calculation process of the limit position p2Mθ shown in FIG. 9 is an example of a process in which the position detection unit 13 calculates the limit position in other moving directions where the limit position has not been calculated by interpolation using the limit positions in a plurality of moving directions where the limit position has been specified. The specific process of the movement magnification XHθ is an example of a process in which the movement magnification specifying unit 14 specifies the movement magnification when remotely operating the target device by moving the hand from the reference position to the limit position in other moving directions where the limit position has not been specified based on the reference position and the limit positions calculated for other moving directions.

[0049] FIG. 10 is a second diagram showing the relationship between the distance from the reference position and the movement magnification. In the above example, the case where the movement magnification X is a constant Ra÷L2 until the position of the hand touches the obstacle at the distance L2 has been described, but the movement magnification may be set in two steps. For example, when the hand is at a position up to a distance L1 close to the operator's body, the movement magnification may be set to X1, and when the distance to the position of the hand is longer than L1 and up to L2, the movement magnification X2 may be set. In this case, the control unit 11 outputs guidance instructing the hand to move to a position up to the distance L1 at which the robot arm 2 is controlled with the movement magnification X1. Then, the position detection unit 13 detects the position of the hand corresponding to the distance L1 (L1 < L2) in the same manner as the reference position p1 and the limit position p2. The movement magnification X1 at the position of the hand up to the distance L1 may be determined in advance. It is assumed that the movement magnification X1 is a value within the range satisfying the above-mentioned movement magnification X (Ra÷L2) > X1 ≧ 0. Further, the movement magnification specifying unit 14 calculates the movement magnification X2 at the position of the hand from the distance L1 to the distance L2 by the calculation formula of X2 = (Ra - X1×L1)÷(L2 - L1). Thereby, the movement magnification X1 at each position of the hand from the reference position to the distance L1 becomes smaller than the movement magnification X2, and the expansion and contraction operation control of the robot arm 2 can be performed such that the position up to the nearby L1 moves relatively slowly and the positions from the distance L1 to the distance L2 move relatively fast.

[0050] When using the movement magnifications X1 and X2 in Figure 10, assume that the distance Ra = 100 cm, distance L1 = 10 cm, distance L2 = 40 cm, and the set movement magnification X1 = 1.5 times. In this case, when L1 < movement distance ≤ L2, the movement magnification X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 85 ÷ 30 = 2.8 times (rounded to the second decimal place).

[0051] Figure 11 is a third figure showing the relationship between the distance from the reference position and the movement magnification. As shown in Figure 11, the movement magnification X1 up to distance L1 may be greater than the movement magnification X2 from distance L1 to distance L2. In this case, the value of the movement magnification X1 is less than or equal to Ra ÷ L1 and greater than Ra ÷ L2. Let's assume that the distance Ra = 100 cm, the distance L1 = 10 cm, the distance L2 = 40 cm, and the set movement magnification X1 = 6.7 times. In this case, the movement magnification X2 when L1 < movement distance ≤ L2 is X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 33 ÷ 30 = 1.1 times.

[0052] Figure 12 is the fourth figure showing the relationship between the distance from the reference position and the movement magnification. Note that when the movement magnification X1 = X2, that is, when the movement magnification X does not change from the reference position p1 of the hand to the limit position p2, and the distance Ra = 100 cm, distance L1 = 10 cm, distance L2 = 50 cm, and movement magnification X1 = 2, then Ra ÷ L2 = 2 times. Similarly, when calculated using the formula X2 = (Ra - X1 × L1) ÷ (L2 - L1), X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 80 ÷ 40 = 2 times.

[0053] Figure 13 is the fifth figure showing the relationship between the distance from the reference position and the movement magnification ratio. Depending on the setting of the movement magnification ratio X1 up to distance L1, the movement magnification ratio specification unit 14 may be configured so that the extension of the robot arm 2 reaches the maximum distance Ra by the time the operator extends their arm and reaches distance L1. For example, suppose the operator sets the movement magnification ratio X1 = 10 and inputs it to the HMD1 when the distance Ra = 100 cm, distance L1 = 10 cm, and distance L2 = 40 cm. In this case, X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 0 ÷ 30 = 0. Also, when the hand reaches distance L1, the robot arm 2 can be operated so that its extension reaches the maximum distance Ra.

[0054] Figure 14 is the sixth figure showing the relationship between the distance from the reference position and the movement magnification. The operator may set the movement magnification X1 to 0x in the HMD1 until the distance L1. For example, suppose the operator sets the movement magnification X1 to 0x in a situation where the distance Ra = 100 cm, distance L1 = 10 cm, and distance L2 = 40 cm, and inputs this into the HMD1. In this case, X2 = (Ra - X1 × L1) ÷ (L2 - L1) = 100 ÷ 30 = 3.33x. This processing in the HMD1 allows for control such that the robot arm 2 cannot extend or retract until the hand is moved to distance L1.

[0055] According to the HDM1 processing described above, in a technology that remotely controls a robot arm 2 based on the movement of body parts such as hands and feet, the robot arm 2 can be extended and retracted without restriction even when there are obstacles near the operator, thereby improving the operability of the robot arm 2.

[0056] Figure 15 shows the functional blocks of an information processing device with a different configuration. Figure 16 shows the processing flow of an information processing device with a different configuration.

[0057] As shown in Figure 15, the information processing device may include a position detection unit 13 and a movement magnification determination unit 14. When remote operation is performed to move a part of the body that is the target of movement to extend the tip of the arm of the target device from a predetermined position to its maximum position, the position detection unit 13 detects a reference position p1 of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is at the predetermined position, and a limit position p2 indicating the position to which the part has been moved from the reference position p1 within the range in which the part of the body that is the target of movement can be moved (step S201). The movement magnification determination unit 14 determines the movement magnification X when remote operation is performed to move a part of the body that is the target of movement from the reference position p1 to the limit position p2 to extend the arm of the target device, based on the maximum distance Ra when the arm of the target device is extended from the predetermined position to its maximum position and the distance L2 from the reference position p1 to the limit position p2 (step S203).

[0058] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0059] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0060] (Note 1) When remotely operating a body part to be moved to extend the tip of an arm of a target device from a predetermined position to its maximum position, the information processing device comprises: position detection means for detecting a reference position of the body part to be moved corresponding to the state in which the tip of the arm is in a predetermined position, and a limit position indicating the position in which the body part has been moved from the reference position within the range in which the body part to be moved can be moved; and movement multiplier determination means for determining the movement multiplier when remotely operating the arm of a target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0061] (Note 2) The information processing device described in Note 1, wherein the part of the body that is the target of the movement is the hand.

[0062] (Note 3) The position detection means is an information processing device according to Note 1 or Note 2 that detects the reference position and the limit position using sensing information acquired from a sensor attached to the head.

[0063] (Note 4) The reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when the target device is remotely operated, as described in any one of Notes 1 to 3.

[0064] (Note 5) The information processing apparatus according to any one of Notes 1 to 4, wherein the position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, and the movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement.

[0065] (Note 6) The position detection means is an information processing device according to Note 3, which uses the sensing information to detect the position where the body part to be moved touches an obstacle provided within the range of movement of the body part, and identifies that position as the limit position.

[0066] (Note 7) The information processing device according to Note 3 or Note 6, wherein the position detection means detects the reference position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the reference position, and detects the limit position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the limit position.

[0067] (Note 8) The information processing apparatus according to Note 5, wherein the direction of movement is one or more of the following directions of movement of the body: forward, backward, left, right, up, or down.

[0068] (Note 9) The position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using limit positions in a plurality of movement directions for which limit positions have been identified, and the movement magnification determination means determines the movement magnification when the target device is remotely operated by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit positions calculated for the other movement directions. The information processing apparatus according to any one of Notes 1 to 8.

[0069] (Note 10) An information processing device according to any one of Notes 1 to 9, comprising: a remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification;

[0070] (Note 11) An information processing method for remotely operating a target device by moving a part of the body that is the target of movement to extend the tip of the arm of the target device from a predetermined position to its maximum position, wherein a reference position of the part of the body that is the target of movement corresponding to the state in which the tip of the arm is in the predetermined position and a limit position indicating the position in which the part has been moved from the reference position within the range in which the part of the body that is the target of movement can be moved, and a movement ratio when remotely operating the arm of the target device by moving the part of the body that is the target of movement from the reference position to the limit position, is determined based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0071] (Note 12) The information processing method described in Note 11, wherein the part of the body that is the target of the movement is the hand.

[0072] (Note 13) The information processing method according to Note 11 or Note 12, which detects the reference position and the limit position using sensing information acquired from a sensor attached to the head.

[0073] (Note 14) The reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when remotely operating the target device. The information processing method according to any one of Notes 11 to 13.

[0074] (Note 15) An information processing method according to any one of Notes 11 to 14, which detects the reference position and the limit position according to the direction of movement of the part to be operated, and identifies the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement.

[0075] (Note 16) The information processing method according to Note 13, wherein the sensing information is used to detect the position where the body part touches an obstacle provided within the range in which the body part to be moved is moved, and the position is identified as the limit position.

[0076] (Note 17) The information processing method according to Note 13 or Note 16, wherein the method detects the reference position using sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the reference position, and detects the limit position using sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the limit position.

[0077] (Note 18) The information processing method according to Note 15, wherein the direction of movement is one or more of the following directions of movement of the body: forward, backward, left, right, up, or down.

[0078] (Note 19) An information processing method according to any one of Notes 11 to 18, wherein the limit position in other movement directions for which the limit position has not been calculated is calculated by interpolation calculation using the limit positions of multiple movement directions for which the limit positions have been identified, and the movement ratio when the target device is remotely operated by moving the part from the reference position to the limit position in the other movement direction based on the reference position and the limit positions calculated for the other movement directions.

[0079] (Note 20) The information processing method according to any one of Notes 11 to 19, wherein the remote operation of the target device is performed by moving the part within the range from the reference position to the limit position, and the movement magnification is used for the remote operation.

[0080] (Note 21) A program that causes the computer of an information processing device to function as a position detection means for detecting a reference position of the body part to be moved, corresponding to the state in which the tip of the arm of the target device is at the predetermined position, and a limit position indicating the position in which the body part has been moved from the reference position within the range in which the body part to be moved can be moved, when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, and a movement ratio determination means for determining the movement ratio when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

[0081] (Note 22) The program described in Note 21, wherein the part of the body that is the target of the movement is the hand.

[0082] (Note 23) The position detection means is a program according to Note 21 or Note 22 that detects the reference position and the limit position using sensing information acquired from a sensor attached to the head.

[0083] (Note 24) The program described in any one of Notes 21 to 23, wherein the reference position is the position in which the hand, which is one aspect of the part to be operated, is brought closest to the body when the target device is remotely operated.

[0084] (Note 25) The program according to any one of Notes 21 to 24, wherein the position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, and the movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement.

[0085] (Note 26) The position detection means is the program described in Note 23, which uses the sensing information to detect the position where the body part touches an obstacle provided within the range in which the body part to be moved touches the obstacle, and identifies that position as the limit position.

[0086] (Note 27) The program described in Note 23 or Note 26, wherein the position detection means detects the reference position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the reference position, and detects the limit position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the limit position.

[0087] (Note 28) The program described in Note 25, wherein the direction of movement is one or more of the following directions of movement for the body: forward, backward, left, right, up, or down.

[0088] (Note 29) The position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using limit positions in multiple movement directions for which limit positions have been identified, and the movement magnification determination means determines the movement magnification when the target device is remotely operated by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit positions calculated for the other movement directions, as described in any one of Notes 21 to 28.

[0089] (Note 30) A program according to any one of Notes 21 to 29, comprising: a remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification;

[0090] This application claims priority based on Japanese Patent Application No. 2024-163568, filed on 20 September 2024, and incorporates all of its disclosures herein.

[0091] 1...Head-mounted display (information processing device) 2...Robot arm 11...Control unit 12...Acquisition unit 13...Position detection unit 14...Movement magnification determination unit 15...Remote control unit 61...Camera 62...Motion sensor 63...Depth sensor 101...CPU 102...ROM 103...RAM 104...Storage device 105...Communication module 106...Sensing device 107...Input device 108...Monitor 109...Speaker

Claims

1. An information processing device comprising: position detection means for detecting a reference position of the body part to be moved, corresponding to the state in which the tip of the arm of a target device is at a predetermined position, and a limit position indicating the position of the body part to be moved from the reference position within the range in which the body part to be moved can be moved; and movement ratio determination means for determining the movement ratio when remotely controlling the arm of a target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

2. The information processing device according to claim 1, wherein the part of the body that is the target of the movement is the hand.

3. The information processing apparatus according to claim 1 or 2, wherein the position detection means detects the reference position and the limit position using sensing information acquired from a sensor attached to the head.

4. The information processing device according to any one of claims 1 to 3, wherein the reference position is the position in which a hand, which is one aspect of the part to be operated, is brought closest to the body when remotely operating the target device.

5. The information processing apparatus according to any one of claims 1 to 4, wherein the position detection means detects the reference position and the limit position according to the direction of movement of the part to be operated, and the movement magnification determination means determines the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement.

6. The information processing apparatus according to claim 3, wherein the position detection means uses the sensing information to detect the position where the body part to be moved touches an obstacle provided within the range of movement of the body part, and identifies that position as the limit position.

7. The information processing apparatus according to claim 3 or 6, wherein the position detection means detects the reference position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the reference position, and detects the limit position using the sensing information at a predetermined timing after the output of guidance instructing to move the part to be operated to the limit position.

8. The information processing apparatus according to claim 5, wherein the direction of movement is one or more of the following directions of movement of the body: forward, backward, left, right, up, or down.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the position detection means calculates limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using limit positions in a plurality of movement directions for which limit positions have been identified, and the movement magnification determination means determines the movement magnification when the target device is remotely operated by moving the part from the reference position to the limit position in the other movement direction, based on the reference position and the limit positions calculated for the other movement directions.

10. An information processing apparatus according to any one of claims 1 to 9, comprising: remote control means for remotely controlling the target device by moving the part within the range from the reference position to the limit position, using the movement magnification; 11. An information processing method for remotely operating a target device by moving a part of the body to extend the tip of the arm of the target device from a predetermined position to its maximum position, which involves detecting a reference position of the part of the body to be moved that corresponds to the state in which the tip of the arm is at the predetermined position, and a limit position indicating the position where the part of the body to be moved has been moved from the reference position within the range in which it can be moved, and determining the movement ratio when remotely operating the arm of the target device by moving the part of the body to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

12. The information processing method according to claim 11, wherein the part of the body that is the target of the movement is the hand.

13. The information processing method according to claim 11 or claim 12, which detects the reference position and the limit position using sensing information acquired from a sensor attached to the head.

14. The information processing method according to any one of claims 11 to 13, wherein the reference position is the position in which a hand, which is one aspect of the part to be operated, is brought closest to the body when remotely operating the target device.

15. The information processing method according to any one of claims 11 to 14, which detects the reference position and the limit position according to the direction of movement of the part to be operated, and identifies the movement magnification according to the direction of movement based on the reference position and the limit position according to the direction of movement.

16. The information processing method according to claim 13, wherein the sensing information is used to detect the position where the body part touches an obstacle provided within the range in which the body part to be moved is moved, and the position is identified as the limit position.

17. The information processing method according to claim 13 or claim 16, wherein the method detects the reference position using sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the reference position, and detects the limit position using sensing information at a predetermined timing after the output of guidance instructing the part to be operated to move to the limit position.

18. The information processing method according to claim 15, wherein the direction of movement is one or more of the following directions of movement of the body: forward, backward, left, right, up, or down.

19. An information processing method according to any one of claims 11 to 18, comprising: calculating limit positions in other movement directions for which the limit position has not been calculated by interpolation calculation using limit positions in multiple movement directions for which limit positions have been identified; and determining the movement ratio when remotely operating the target device by moving the part from the reference position to the limit position in the other movement direction based on the reference position and the limit positions calculated for the other movement directions.

20. A program that causes the computer of an information processing device to function as a position detection means for detecting a reference position of the body part to be moved, corresponding to the state in which the tip of the arm of the target device is at the predetermined position, and a limit position indicating the position in which the body part has been moved from the reference position within the range in which the body part to be moved can be moved, when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, and a movement ratio determination means for determining the movement ratio when remotely operating the arm of the target device by moving the body part to be moved from the reference position to the limit position, based on the maximum distance when the arm of the target device is extended from the predetermined position to the maximum position and the distance from the reference position to the limit position.

Images