Information processing apparatus, and information processing system

Abstract

To provide an operation UI that has a high degree of freedom in a direction of movement, while providing constraints for allowing a user to easily perform operations.SOLUTION: An information processing apparatus provides an operator with a user interface for operating a virtual object in a virtual space, and has: a first acquisition unit that acquires a first movement parameter on the basis of input by the movement of the hand; a second acquisition unit that acquires a second movement parameter on the basis of input by a hand gesture; and a position control unit that determines the position of the virtual object after movement on the basis of at least one of the first movement parameter and the second movement parameter. The first movement parameter acquired by the first acquisition unit is a parameter for regulating a movement according to constraints for restricting a direction or a range in which the virtual object is movable. The second movement parameter acquired by the second acquisition unit is a parameter for regulating a movement contradictory to the constraints.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an operation UI in XR experiences, and particularly to the movement of virtual objects.

Background Art

[0002] In experiences using virtual spaces in VR (Virtual Reality) and MR (Mixed Reality), there are scenarios where the user moves virtual objects. At this time, the user operates the virtual object in a three-dimensional space.

[0003] Various methods have been proposed for operating virtual objects in a virtual space. A ray emitted from the user's own hand or a controller is an operation means in which a starting point and a direction are given, and the virtual object can be specified by extending from the starting point and colliding with the virtual object. At the collision point between the virtual object and the ray, a mark called a cursor is displayed. Rays are generally used as a means for selecting and operating virtual objects in a virtual space, and an operation system is generally provided in which objects close to the operator are directly operated by hand and objects far from the operator are operated by rays. Even when operating directly by hand, a cursor is displayed to indicate the position to be operated on.

[0004] In the operation of moving a virtual object, if all three axial directions in three-dimensional coordinates can be operated simultaneously, it becomes very difficult for unaccustomed users to accurately place the virtual object at an arbitrary position. To solve this problem, means for restricting the operable axial directions have been proposed.

[0005] For example, in Patent Document 1, a method is disclosed in which the XYZ axes or rotation axes are displayed on a virtual object, and displacement in a moving direction or a rotation direction can be performed based on the selected axis.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the prior art disclosed in the above-mentioned patent document, the degree of freedom in a single movement operation is restricted.

[0008] An object of the present invention is to provide an operation UI that gives constraints that make it easy for an operator to perform an operation and has a high degree of freedom in the moving direction.

Means for Solving the Problems

[0009] The present disclosure is an information processing apparatus that provides an operator with a user interface for operating a virtual object on a virtual space displayed on a display device, and based on an input by the movement of the operator's hand, a first acquisition unit that acquires a first movement parameter, a second acquisition unit that acquires a second movement parameter based on an input by the operator's hand gesture, a position control unit that determines a position after movement of the virtual object based on at least one of the first movement parameter acquired by the first acquisition unit and the second movement parameter acquired by the second acquisition unit, and an image generation unit that generates an image including the virtual object for display on the display device based on the position determined by the position control unit, wherein the first movement parameter acquired by the first acquisition unit is a parameter that defines movement in accordance with a constraint that restricts a direction or range in which the virtual object can move, and the second movement parameter acquired by the second acquisition unit is a parameter that defines movement contrary to the constraint, including the information processing apparatus.

Effects of the Invention

[0010]

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings.

[0013] An information processing system according to an embodiment of the present invention (hereinafter also referred to as "this system") is a system for providing a user with an XR experience. This system provides a user (operator) with a user interface (operation UI) for operating virtual objects on a virtual space, and the user can perform operations such as selection and movement of virtual objects using this operation UI. There are two methods for operating virtual objects: a method of directly operating a virtual object with the operator's hand (hereinafter also referred to as "direct operation", "first operation"), and a method of operating a virtual object with a virtual ray emitted from the operator's hand (hereinafter also referred to as "ray operation", "second operation"). Either operation method may be adopted. The direct operation has the advantage that it is easy to accurately (as intended) operate a virtual object because the operator's own hand is used as an operation means. The ray operation has the convenience that virtual objects at a distance that the hand cannot reach can also be operated.

[0014] In the embodiment described below, an example of a process for automatically switching between direct operation and ray operation based on the distance between the operator's hand and the virtual object during the operation of the virtual object will be described. For example, if it is automatically switched to direct operation when the virtual object is within a distance that can be touched by the hand, and to ray operation when it is out of reach of the hand, both the advantages of direct operation and the convenience of ray operation can be enjoyed.

[0015] FIG. 1 is a block diagram showing a configuration example of an information processing system according to the present embodiment of the present invention. The information processing system according to the present embodiment has a configuration in which a display device 1000 and an information processing device 1100 are connected. The connection may be either wired or wireless. Furthermore, it may be realized by combining wired and wireless. Alternatively, the display device 1000 and the information processing device 1100 may be configured as a single device.

[0016] The display device 1000 can output a composite reality image in which an image of the real space and an image of the virtual space are combined. Note that the present invention relates to an information processing apparatus that displays an image of a virtual space, and is not limited to a system that displays a composite reality image. It may be a VR system that only displays an image of a virtual space, or an AR (Augmented Reality) system that displays an image of a virtual space on a transmitted real image. It may also be.

[0017] The display device 1000 includes an imaging unit 1010 and a display unit 1020. The imaging unit 1010 captures images of the real space continuously in time. The captured images of the real space are output to the information processing apparatus 1100. The imaging unit 1010 may be composed of a plurality of cameras so as to be able to simultaneously capture the real space from a plurality of different directions. The display unit 1020 displays the composite reality image output from the information processing apparatus 1100.

[0018] The display device 1000 may have a structure that can be worn by an operator. For example, the display device 1000 may be a non-transmissive or video see-through type HMD (Head Mounted Display). At this time, the display unit 1020 may be composed of two displays arranged at positions corresponding to the right eye and the left eye of the operator wearing the HMD, respectively. A video for the right eye may be displayed on the display for the right eye, and a video for the left eye may be displayed on the display for the left eye. Also, the display unit 1020 may be a handheld display.

[0019] Next, the information processing apparatus 1100 will be described. FIG. 2 is a hardware configuration diagram of the information processing apparatus 1100 in the present embodiment. The information processing apparatus 1100 includes a CPU (Central Processing Unit) 201, a storage device 202, a ROM (Read only memory) 203, and a RAM (Random Access Memory) 204. The information processing apparatus 1100 also includes an input I / F (Interface) 205 and an output I / F 206.

[0020] The CPU 201 comprehensively controls each device connected via the bus 200. The CPU 201 is a hardware processor that reads and executes programs stored in the storage device 202, ROM 203, RAM 204, etc. Note that the information processing apparatus 1100 may have a processor such as a GPU (Graphics Processing Unit) for assisting the operations of the CPU 201, and peripherals.

[0021] The storage device 202 is a device in which programs and data used in the information processing apparatus 1100 are non-temporarily stored. For example, the storage device 202 may be an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a flash memory.

[0022] The ROM 203 is a non-volatile memory in which an operating system (OS), device drivers, and boot programs are non-temporarily stored. The RAM 204 is a volatile memory for temporarily storing programs and data loaded from the storage device 202 and the ROM 203. The RAM 204 is used as a work area for the CPU 201.

[0023] The input I / F 205 is an interface to which an input device for inputting signals to the information processing apparatus 1100 is connected. For example, the input device may include a controller (not shown) used by an operator for operations, an imaging unit 1010 of the display device 1000, etc. The output I / F 206 is an interface to which an output device for processing signals output from the information processing apparatus 1100 is connected. For example, the output device may include a speaker (not shown), a display unit 1020 of the display device 1000, etc.

[0024] Returning to FIG. 1, the functional configuration of the information processing apparatus 1100 will be described. The information processing apparatus 1100 includes a position and orientation acquisition unit 1110, a ray cursor position and orientation acquisition unit 1120, an operation means determination unit 1130, a hand gesture detection unit 1140, and an operation target selection unit 1150. Information The processing apparatus 1100 further includes a movement constraint setting unit 1160, a gesture movement parameter acquisition unit 1170, a virtual object position control unit 1180, an image generation unit 1190, an image composition unit 1200, a data storage unit 1210, and a movement parameter acquisition unit 1220. These functions are realized by the CPU 201 reading and executing a program stored in the storage device 202 or the ROM 203. However, some or all of these functions may also be realized by a dedicated circuit such as an ASIC (application specific integrated circuit).

[0025] The position and orientation acquisition unit 1110 calculates the position and orientation of the imaging unit 1010 in the world coordinate system and the position and orientation of the joint points of the operator's hand. Specifically, the position and orientation acquisition unit 1110 extracts characteristic information such as points and lines in the image from the image of the real space captured by the imaging unit 1010, and calculates the position and orientation of the imaging unit 1010 from that information. The calculated position and orientation information of the imaging unit 1010 is output to the data storage unit 1210. Also, the position and orientation acquisition unit 1110 calculates the position and orientation information of the joint points of the operator's hand from the image of the real space captured by the imaging unit 1010. The calculated position and orientation information of the joint points of the hand is output to the data storage unit 1210. The joint points of the hand may include the wrist, the joints of each finger, and the fingertips. The position and orientation information calculated by the position and orientation acquisition unit 1110 may be the position information of some joint points of the hand. In this embodiment, the position and orientation measurement of the imaging unit 1010 and the operator's hand is described by position and orientation measurement by image processing, but the present invention is not limited to this. The position and orientation information may be measured using infrared light, or may be measured using an ultrasonic sensor, a magnetic sensor, or a depth sensor.

[0026] The ray cursor position and orientation acquisition unit 1120 calculates the position and orientation of the ray cursor from the position and orientation of the imaging unit 1010 calculated by the position and orientation acquisition unit 1110 and the position and orientation of the joint points of the operator's hand.

[0027] A method for the ray cursor position and orientation acquisition unit 1120 to calculate the position and orientation of the ray cursor will be described using a specific example. The ray cursor position and orientation acquisition unit 1120 estimates the position of the operator's shoulder from the position and orientation information of the imaging unit 1010 calculated by the position and orientation acquisition unit 1110. Using the position of the wrist among the estimated position of the operator's shoulder and the position and orientation information of the joint points of the hand, the direction from the shoulder to the position of the wrist is taken as the direction of the ray and the direction of the X-axis in the orientation of the ray. The direction parallel to the horizontal plane is taken as the Y-axis direction, and the Z-axis direction is determined according to the world coordinate system of the virtual space to obtain the orientation of the ray. The starting point of the ray is set to the middle position between the positions of the thumb and the index finger. In this example, the positions of the operator's shoulder and wrist are used for calculating the orientation of the ray, but the present invention is not limited to this. Other body parts such as the chest and hip positions may be estimated and used, or positions displaced by an arbitrary amount from the position and orientation information of the imaging unit 1010 may be used. Also, among the position and orientation information of the joint points of the operator's hand, any two points may be used, or points calculated from the position and orientation information of the joint points of the hand may be used. Also, the determination of the starting point position is not limited to using the middle position between the positions of the thumb and the index finger. Any one of the joint points of the hand may be used, or points calculated from the position and orientation information of the joint points of the hand may be used.

[0028] The ray cursor position and orientation acquisition unit 1120 determines whether the calculated ray collides with a virtual object in the virtual space, and sets the three-dimensional coordinates of the collision point as the cursor position. Further, the ray cursor position and orientation acquisition unit 1120 calculates the orientation of the cursor from the shape of the surface of the virtual object at the collision point. When the ray does not collide with the virtual object, the reach point of the preset maximum length of the ray is set as the cursor position. The calculated position and orientation of the ray cursor are output to the data storage unit 1210. In this embodiment, the cursor position in the case of ray operation and the cursor position in the case of direct operation are set to the same position, but the present invention is not limited to this. In direct operation, it may be obtained independently based on the position and orientation of the joint points of the operator's hand calculated by the position and orientation acquisition unit 1110. For example, the cursor may be displayed at the tip of the operator's index finger during direct operation.

[0029] The operation means determination unit 1130 determines which of "the operator's hand" and "the ray" is used as the operation means, that is, which operation method of direct operation and ray operation is used. Specifically, the operation means determination unit 1130 determines whether it is a direct operation or a ray operation by comparing the distance between the operator's hand (for example, the starting point position of the ray) and the virtual object (for example, the cursor position) with a preset threshold value.

[0030] The hand gesture detection unit 1140 detects hand gestures made by the operator. Specifically, the hand gesture detection unit 1140 detects gestures by image recognition from the image of the real space captured by the imaging unit 1010. The detection result is output to the data storage unit 1210. The detection result may be a binary value of true or false indicating whether a gesture has been made, or may be a continuous value between 0 and 1 based on the state where the operation is completely performed. It may also be configured to change the value output according to the detected gesture. The detection result is output to the data storage unit 1210. In this embodiment, gesture detection using image recognition is described, but the present invention is not limited to this. Gestures may be detected by setting specific conditions for the relative positional relationship and distance between the joint points of the operator's hand. Also, gestures may be detected by measuring the movement of muscles and bones using sensors. It is preferable that the hand gesture detection unit 1140 can detect multiple types of gestures, and it is preferable that commands (meanings) are assigned in advance for each type of gesture.

[0031] When the operation target selection unit 1150 detects a gesture assigned to the "selection" command by the hand gesture detection unit 1140, it determines the operation target from the position information of the cursor calculated by the ray cursor position and orientation acquisition unit 1120. For example, a gesture of pinching (clamping) an object with a finger, a gesture of grasping an object with a hand, etc. may be assigned to the "selection" command, or other gestures may be used. When the cursor position is on the surface of a virtual object in the virtual space, the operation target selection unit 1150 may determine that virtual object as the operation target.

[0032] The movement constraint setting unit 1160 gives a constraint to the operation target selected by the operation target selection unit 1150, restricting the direction or range in which the operation target can be moved. Information on the given constraint is output to the data storage unit 1210. The constraint may be read from a previously set constraint, or may be selected from several constraints when the operation target is determined. For example, when a constraint is set to restrict movement in one axis direction of three axes in a three-dimensional coordinate system and allow movement in the remaining two axis directions, the operation target moves in a two-dimensional plane following the operator's movement of the hand in the three-dimensional space. By restricting the degree of freedom of the movement direction in this way, it becomes easier to perform the operation of moving the operation target to an intended position. Note that in this embodiment, an example of restricting movement in one axis direction will be described, but the method of setting the constraint is not limited to this. Movement in two axis directions may be restricted and only movement in one axis direction may be allowed, or a constraint may be set based on the position and orientation of the imaging unit 1010 or the position and orientation of the joint point of the hand calculated by the position and orientation acquisition unit 1110. A constraint that restricts the movement range of a virtual object may also be set.

[0033] The operator can move a selected virtual object in a virtual space by using an operation UI provided by this system. This system provides two input methods for parameters (hereinafter referred to as "movement parameters") that define the movement of a virtual object: a first input method in which the operator moves his / her hand in space to follow the virtual object, and a second input method in which input is made by hand gestures. The first input method is used to instruct a movement that complies with the above-mentioned constraints, and the second input method is used to instruct a movement that violates the constraints. Here, the movement parameter acquisition unit 1220 is a first acquisition unit that acquires the movement parameters by the first input method, and the gesture movement parameter acquisition unit 1170 is a second acquisition unit that acquires the movement parameters by the second input method. The movement parameters include the amount of movement, the direction of movement, the speed of movement, and the like. In this embodiment, an example in which the amount of movement and the direction of movement are used will be described.

[0034] The movement parameter acquisition unit 1220 calculates the movement parameters of the virtual object based on the input by the movement of the operator's hand. Specifically, the movement parameter acquisition unit 1220 acquires the position of the cursor calculated by the ray cursor position and orientation acquisition unit 1120 from the data storage unit 1210, and calculates the movement amount and movement direction of the virtual object so as to follow the change in the position of the cursor. At this time, the movement parameter acquisition unit 1220 calculates the movement amount and movement direction so that the virtual object moves according to the constraints set for the virtual object. For example, if the movement in the direction parallel to the ray is restricted and only the movement in the direction perpendicular to the ray is allowed, the movement amount and movement direction in the plane perpendicular to the ray can be calculated by projecting the movement vector of the position (three-dimensional coordinates) of the cursor onto the plane perpendicular to the ray. The calculated movement amount and movement direction are output to the data storage unit 1210.

[0035] The gesture movement parameter acquisition unit 1170 calculates the movement parameters of the virtual object based on the input by the operator's hand gesture. Specifically, the gesture movement parameter acquisition unit 1170 calculates the movement amount and movement direction input by the operator when the gesture assigned to the "move" command is detected by the hand gesture detection unit 1140. For example, it is preferable that the movement amount and movement direction are calculated from the position and orientation of the hand joint points calculated by the position and orientation acquisition unit 1110 and the position and orientation of the ray calculated by the ray cursor position and orientation acquisition unit 1120. While the movement parameter acquisition unit 1220 calculates movement parameters according to the constraints, the gesture movement parameter acquisition unit 1170 calculates movement parameters contrary to the constraints. For example, if the movement in the direction parallel to the ray is restricted and only the movement in the direction perpendicular to the ray is allowed, the gesture movement parameter acquisition unit 1170 calculates movement parameters that define the movement in the direction parallel to the ray. The calculated movement amount and movement direction are output to the data storage unit 1210.

[0036] In this embodiment, an example of calculating the amount and direction of movement from a hand gesture and moving a virtual object in a ray operation is shown in FIG. 3. In this example, it is assumed that the virtual object 301 is restricted from moving in a direction parallel to the ray 303, and two-degree-of-freedom movement within a two-dimensional plane orthogonal to the ray 303 is allowed. The gesture movement parameter acquisition unit 1170 calculates the amount of movement from the "gesture of rotating the hand 302" by the operator. Specifically, the gesture movement parameter acquisition unit 1170 uses the ray 303 as the rotation axis, and takes the posture of the hand 302 at the time when the virtual object 301 is selected as the reference posture (i.e., 0 degrees), and calculates the amount of movement based on the degree of change (rotation degree) of the posture of the hand 302 with respect to the reference posture. The movement direction is determined based on the movement restriction and the change direction (rotation direction) of the posture of the hand 302 with respect to the reference posture. For example, when a gesture of rotating the hand 302 counterclockwise with the ray 303 as the rotation axis is performed, the movement direction may be the forward direction, and when a gesture of rotating the hand 302 clockwise is performed, the movement direction may be the depth direction.

[0037] Next, in this embodiment, an example of calculating the amount and direction of movement from a hand gesture and moving a virtual object in a direct operation is shown in FIG. 4. In this example, the movement range of the virtual object 401 is restricted to the range where the operator's hand 402 can move. The gesture movement parameter acquisition unit 1170 calculates the amount of movement from the "gesture of rotating the hand 402" by the operator. Specifically, using the straight line connecting the wrist and the cursor position 404 on the virtual object 401 as the rotation axis, and taking the posture of the hand 402 at the time when the virtual object 401 is selected as the reference posture (i.e., 0 degrees), the amount of movement is calculated based on the degree of change (rotation degree) of the posture of the hand 402 with respect to the reference posture. The movement direction is determined based on the change direction (rotation direction) of the posture of the hand 402 with respect to the reference posture. For example, when a gesture of rotating the hand 402 clockwise with the straight line connecting the wrist and the cursor position 404 as the rotation axis is performed, the movement direction is the depth direction. This enables the input of movement parameters that define movement in a direction away from the operator's hand 402 by hand gestures. In direct operation, gestures that rotate counterclockwise may be ignored, and the virtual object 401 held by the hand 402 may not be moved forward.

[0038] In this embodiment, the method for calculating the amount and direction of movement by gestures is the same for both direct operation and ray operation, but the present invention is not limited to this. The joint points and rotation axes of the hand used in the calculation may be switched by the operation means. Also, the gesture for performing the movement may be switched by the operation means.

[0039] In this embodiment, an example of inputting the amount of movement of the operation target by the gesture of rotating the hand and the degree of rotation has been described, but the method of inputting the amount of movement is not limited to this. For example, the gesture of opening and closing the hand and the degree of opening, the gesture of bending and extending a specific finger and the degree of expansion and contraction, the gesture of changing the relative positions of a specific plurality of fingers and the degree of change, the gesture of bending and extending the wrist and the angle thereof, etc. may be used. For other gestures, the amount of movement may be calculated based on the value from 0 to 1 output by the hand gesture detection unit 1140. Also, the hand performing the gesture may be the hand opposite to the hand during operation. By instructing movement in accordance with the constraints (for example, movement within a plane perpendicular to the ray) with one hand and movement contrary to the constraints (for example, movement in a direction parallel to the ray) with the other hand, high operability and convenience can be realized.

[0040] The virtual object position control unit 1180 determines the position of the virtual object after movement based on at least one of the movement parameters calculated by the movement parameter acquisition unit 1220 and the movement parameters calculated by the gesture movement parameter acquisition unit 1170. In a state where the hand gesture corresponding to the "move" command is not performed, position control in accordance with the constraints is performed using only the former movement parameters. On the other hand, when the hand gesture corresponding to the "move" command is being performed, position control that integrates movement in accordance with the constraints and movement contrary to the constraints is performed using both movement parameters.

[0041] The image generation unit 1190 constructs a virtual space based on the data stored in the data storage unit 1210. The data includes data of virtual objects and virtual light sources. At this time, the virtual objects are arranged at positions determined by the virtual object position control unit 1180. Then, the image generation unit 1190 generates an image of the virtual space as seen from the operator's viewpoint based on the position and orientation calculated by the position and orientation acquisition unit 1110.

[0042] The image composition unit 1200 composes the image of the virtual space generated by the image generation unit 1190 and the image of the real space captured by the imaging unit 1010 to generate a composite reality image. The generated composite reality image is output to the display unit 1020.

[0043] As described above, the data storage unit 1210 stores various information. In addition to the information output to the data storage unit 1210 in the above description, it is assumed that the data storage unit 1210 also stores information described as known information in this embodiment.

[0044] FIG. 5 is an example of a flowchart showing a process of switching between two operation means, a direct operation and a ray operation.

[0045] In step S500, the position and orientation acquisition unit 1110 calculates the position and orientation of the imaging unit 1010. In step S501, the display unit 1020 displays the composite reality image composed by the image composition unit 1200. In step S502, the position and orientation acquisition unit 1110 acquires the position and orientation of the joint points of the operator's hand. In step S503, the ray cursor position and orientation acquisition unit 1120 calculates the position and orientation of the ray cursor. In step S504, the operation means determination unit 1130 determines whether the distance between the start point of the ray and the cursor is less than or equal to a threshold value. If it is less than or equal to the threshold value, the process proceeds to step S505, and if it is not less than or equal to the threshold value, the process proceeds to step S506.

[0046] In step S505, direct operation processing for operating a virtual object by direct hand operation is executed. In step S506, ray operation processing for operating a virtual object by a ray emitted from the hand is executed. Details of each operation processing will be described in the flowchart below. In step S507, when an end instruction of the application is input by the operator or the condition for ending the application is satisfied, this processing ends. On the other hand, when an end instruction of the application has not been input and the end condition is not satisfied, the processing returns to step S500.

[0047] Figure 6 is an example of a flowchart of the direct operation processing in step S505.

[0048] In step S600, based on the position and orientation of the cursor calculated by the ray cursor position and orientation acquisition unit 1120, the display unit 1020 displays a composite reality image including the cursor. In step S601, when the hand gesture detection unit 1140 detects a gesture corresponding to the "selection" command, it is determined that the operator (user) has performed an operation of selecting a virtual object to be operated, and the process proceeds to step S602. In step S602, the operation target selection unit 1150 determines an operation target from among the virtual objects in the virtual space. Information on the cursor position calculated in step S503 of FIG. 5 is used to determine the virtual object to be the operation target. Note that when the gesture corresponding to the "selection" command cannot be detected in step S601, the direct operation processing ends.

[0049] In step S603, the position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 and the joint points of the operator's hand. In step S604, the ray cursor position and orientation acquisition unit 1120 calculates the position and orientation of the cursor from the position and orientation of the hand joint points calculated in step S603.

[0050] In step S605, the movement parameter acquisition unit 1220 calculates the amount and direction of movement based on the input by the movement of the operator's hand, and the gesture movement parameter acquisition unit 1170 calculates the amount and direction of movement based on the input by the hand gesture. In step S606, the virtual object position control unit 1180 integrates the amount and direction of movement calculated by the movement parameter acquisition unit 1220 and the amount and direction of movement calculated by the gesture movement parameter acquisition unit 1170, and determines the position of the virtual object after movement during selection.

[0051] In step S607, the image generation unit 1190 generates an image of the virtual space, and the image synthesis unit 1200 synthesizes the image of the virtual space and the image of the real space to generate a composite reality image including the cursor and the virtual object being selected. Then, the display unit 1020 displays the composite reality image.

[0052] In step S608, the hand gesture detection unit 1140 determines whether it has detected the cancellation of the object selection operation. If the cancellation of the object selection operation is not detected, the process proceeds to step S609.

[0053] In step S609, the ray - cursor position and orientation acquisition unit 1120 calculates the position and orientation of the ray - cursor. In step S610, the operation means determination unit 1130 compares the distance between the starting point of the ray calculated in step S609 and the cursor with a threshold value. If the distance is not less than the threshold value, the process proceeds to step S611. If the distance is less than or equal to the threshold value, the process proceeds to step S604 to continue the direct operation process.

[0054] In step S611, it is determined whether the position of the cursor is on the virtual object being selected based on the position of the cursor calculated by the ray - cursor position and orientation acquisition unit 1120 and the position and orientation of the virtual object determined by the virtual object position control unit 1180. If the cursor is on the virtual object being selected, the process proceeds to step S704 in the flowchart of FIG. 7 to switch to the ray operation process. If the cursor is not on the virtual object being selected, the direct operation process ends.

[0055] Figure 7 is an example of a flowchart of the ray operation process in step S506.

[0056] In step S700, based on the position and orientation of the ray cursor calculated by the ray cursor position and orientation acquisition unit 1120, the display unit 1020 displays a composite reality image including the ray cursor. In step S701, when the hand gesture detection unit 1140 detects a gesture corresponding to the "selection" command, it is determined that the operator (user) has performed an operation to select a virtual object to be operated on, and the process proceeds to step S702. In step S702, it is determined whether the position of the cursor calculated by the ray cursor position and orientation acquisition unit 1120 is on the virtual object. If the cursor is on the virtual object, the process proceeds to step S703. The information on the cursor position calculated in step S503 of FIG. 5 is used for the processes of steps S701 and S702. Note that if the gesture corresponding to the "selection" command cannot be detected in step S701, and if it is determined in step S702 that the position of the cursor is not on the virtual object, the ray operation process ends.

[0057] In step S703, the operation target selection unit 1150 determines an operation target from the virtual objects in the virtual space. In step S704, the movement constraint setting unit 1160 sets or updates movement constraints for the virtual object to be operated on. For example, based on the start point and direction of the ray calculated in step S503 of FIG. 5, the axial direction in which the movement of the virtual object is restricted and the axial direction in which the movement of the virtual object is allowed are determined. In this embodiment, it is assumed that the movement in the axial direction parallel to the ray is restricted.

[0058] In step S705, the position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 and the joint points of the operator's hand. In step S706, the ray cursor position and orientation acquisition unit 1120 calculates the position and orientation of the ray cursor from the position and orientation of the hand joint points calculated in step S705.

[0059] In step S707, the movement parameter acquisition unit 1220 calculates the amount and direction of movement based on the input by the movement of the operator's hand, and the gesture movement parameter acquisition unit 1170 calculates the amount and direction of movement based on the input by the hand gesture. In step S708, the virtual object position control unit 1180 integrates the amount and direction of movement calculated by the movement parameter acquisition unit 1220 and the amount and direction of movement calculated by the gesture movement parameter acquisition unit 1170, and determines the position of the virtual object after movement during selection.

[0060] In step S709, the image generation unit 1190 generates an image of the virtual space, and the image composition unit 1200 composes the image of the virtual space and the image of the real space to generate a composite reality image including the ray, the cursor, and the virtual object being selected. Then, the display unit 1020 displays the composite reality image.

[0061] In step S710, the hand gesture detection unit 1140 determines whether it has detected the cancellation of the object selection operation. If the cancellation of the object selection operation is not detected, the process proceeds to step S711.

[0062] In step S711, the ray - cursor position and orientation acquisition unit 1120 calculates the position and orientation of the ray - cursor In step S712, the operation means determination unit 1130 compares the distance between the starting point of the ray calculated in step S711 and the cursor with a threshold value. If the distance is less than or equal to the threshold value, the process proceeds to step S604 in FIG. 6 and switches to the direct operation process. Otherwise, the process proceeds to step S704 and continues the ray operation process.

[0063] According to this embodiment, in a scene where a virtual object is moved using a hand or a ray according to constraints, by performing a hand gesture, the virtual object can be moved outside the constraints. As a result, while ensuring the ease of operation of the virtual object by restricting the movement direction or movement range by constraints, the degree of freedom of operation can be increased by enabling movement against the constraints by additional hand gestures.

[0064] (Modification Example 1) In the above embodiment, no restrictions are imposed on the calculation of the movement amount and direction of movement by hand gestures and the movement. Since the gesture of the operator is directly reflected in the position and orientation of the virtual object by this method, responsive operation is possible. However, on the other hand, depending on the operator, even if the hand moves slightly, the virtual object may move sensitively, and it may be difficult to stabilize the position and orientation of the virtual object.

[0065] In order to solve such problems, a predetermined margin (dead zone) may be set for the input by hand gestures. Specifically, when the degree of change in the hand gesture with respect to the reference gesture is equal to or less than a threshold value (for example, when the degree of rotation is equal to or less than ±10 degrees), the input by hand gestures may be ignored and the movement amount may be set to 0 by the gesture movement parameter acquisition unit 1170.

[0066] (Modification Example 2) The movement direction by hand gestures may be temporarily fixed in one direction. For example, once the movement direction is set based on the change direction of the hand gesture with respect to the reference gesture, the movement direction may be fixed (backward movement is prohibited) until the hand gesture returns to the reference gesture.

[0067] Regarding such a method of restricting direction input, taking the case of using hand gestures by hand rotation as an example, it will be specifically described with reference to FIGS. 8(A) to 8(C). FIG. 8(A) shows a scene immediately after selecting a virtual object to be operated in a ray operation. At this time, the hand rotation is 0 degrees. This state is recorded as a reference posture. Hereinafter, clockwise rotation is represented by a positive angle (0 degrees to +180 degrees), and counterclockwise rotation is represented by a negative angle (0 degrees to -180 degrees). In this example, the movement of the virtual object by hand movement is restricted to movement within a two-dimensional plane perpendicular to the ray, and the movement in the direction parallel to the ray is operated by hand gestures. FIG. 8(B) shows a scene immediately after rotating the hand clockwise by +90 degrees to move the virtual object. At this time, in addition to the virtual object moving in the direction in which the ray extends (the depth direction) by hand gestures, the virtual object also moves in a direction perpendicular to the ray due to the movement of the hand accompanying the rotation.

[0068] As shown in FIG. 8(B), when an input in a certain movement direction is performed by hand gestures, the movement direction of the virtual object is temporarily fixed. In this example, until the hand state is reset (returned to the reference posture), only the input of movement in the direction in which the ray extends (the depth direction) is accepted. Therefore, even if the hand is rotated counterclockwise, the movement of the virtual object by gestures (movement toward the front side) does not occur, and the depth position of the virtual object remains maintained.

[0069] FIG. 8(C) shows a state where the hand is returned to 0 degrees again. With the depth position of the virtual object maintained, the virtual object moves in a direction perpendicular to the ray due to the movement of the hand accompanying the rotation. Since the hand has returned to 0 degrees, the fixation of the movement direction is released, and the movement by hand gestures can be performed in either direction. returned, the fixation of the movement direction is released, and the movement by hand gestures can be performed in either direction.

[0070] As can be seen from FIGS. 8(A) to 8(C), by a series of gestures of twisting and returning the hand, an operation of changing only the depth position of the virtual object can be easily performed. Further, if the gesture of twisting and returning the hand is repeated between 0 degrees and +180 degrees, the virtual object can be continuously moved in the depth direction. Conversely, if the gesture of twisting and returning the hand is repeated between 0 degrees and -180 degrees, an operation of pulling the virtual object to the hand can be easily realized.

[0071] FIG. 9 shows a flowchart for performing this direction input restriction. This flowchart starts after the movement is input once by a hand gesture. Although the description is omitted in the flowchart of FIG. 9, it is assumed that the movement of the virtual object by the movement of the hand continues to be performed constantly during the operation of the object.

[0072] In step S901, the data storage unit 1210 records the movement direction of the movement by the input hand gesture. In step S902, the movement parameter acquisition unit 1220 calculates the movement amount and movement direction based on the input by the movement of the operator's hand, and the gesture movement parameter acquisition unit 1170 calculates the movement amount and movement direction based on the input by the hand gesture. In step S903, the gesture movement parameter acquisition unit 1170 determines whether the movement direction recorded in the data storage unit 1210 in step S901 is the same as the movement direction by the gesture calculated in step S902. If they are the same, the process proceeds to step S904. If they are not the same, the process proceeds to step S906.

[0073] In step S904, the virtual object position control unit 1180 integrates the movement of the virtual object that occurs according to the movement amount and movement direction calculated by the gesture movement parameter acquisition unit 1170 and the movement of the virtual object generated by the movement of the hand in the space, and determines the position of the selected virtual object. In step S905, the image generation unit 1190 generates an image of the virtual space, the image composition unit 1200 composes the image of the virtual space and the image of the real space to generate a composite reality image, and the display unit 1020 displays the composite reality image.

[0074] In step S906, the hand gesture detection unit 1140 determines whether the reference posture (hand state at 0 degrees) stored in the data storage unit 1210 is the same as the current hand state. If they are the same, the process proceeds to step S907. If they are not the same, the process proceeds to step S902. In step S907, the moving direction recorded in the data storage unit 1210 is deleted, and the input restriction on the moving direction is released.

[0075] (Modification Example 3) While the input of movement by hand gesture is being performed, the movement of the virtual object by the movement of the hand may be invalidated. The state of the operation in that case is shown in FIGS. 10(A) to 10(B). FIG. 10(A) is a scene immediately after selecting a virtual object to be operated on in the operation by ray. In this example, the movement of the virtual object by the movement of the hand is restricted to a plane perpendicular to the ray, and the direction of movement by hand gesture is a direction parallel to the ray. FIG. 10(B) is a scene after rotating the hand by 90 degrees and performing movement by hand gesture. When the hand gesture detection unit 1140 detects a gesture (in this example, a gesture of rotating the hand) assigned to the "move" command, it temporarily invalidates the movement of the virtual object by the movement of the hand. Specifically, the virtual object position control unit 1180 ignores the movement parameters calculated based on the movement of the hand and calculates the position of the virtual object after movement using only the movement parameters calculated based on the hand gesture. As a result, as shown in FIG. 10(B), even if the hand moves with the rotation, the virtual object does not move in a direction perpendicular to the ray accordingly , and the virtual object moves only in a direction parallel to the ray according to the input by hand gesture.

[0076] By the methods of the above-described specific modification examples 1 to 3, it becomes easier to avoid unintentional hand movement and position changes of virtual objects due to gestures. Therefore, the operability in the virtual space can be further improved. Note that the methods of modification examples 1 to 3 may be implemented alone, a plurality of methods may be combined, or modification examples 1 to 3 may be adaptively switched depending on the operation intention and the operation target.

[0077] (Modification Example 4) In the above embodiment, no display is performed when the movement restriction or the operation means is switched, but feedback that allows the operator to recognize that the movement restriction and the operation means have changed may be provided. Specifically, when the movement restriction range is updated during operation by the ray, the movement restriction range may be displayed. Further, when switching from the ray operation to the direct operation, the movable directions may be displayed and the hand part may be highlighted. When switching from the direct operation to the ray operation, the movement restriction range may be displayed and the ray part may be highlighted. Note that the present invention is not limited to these display methods. A message may be displayed on the operator's field of view, or notification may be made by vibration or sound. By these means, the operations that can be performed by the operator can be grasped, and the operations can be performed smoothly.

[0078] (Modification Example 5) In Modification 5, when the operation method is switched, reception of input by hand gestures is temporarily stopped. For example, assume that the operator pulls a virtual object toward the front by hand gesture during ray operation and intentionally switches to direct operation. At this time, it is highly likely that the hand is in a state of rotating greatly to pull the virtual object. Therefore, although it is a natural movement to return the hand orientation to the original state, if this movement is recognized as a hand gesture, the virtual object will move in the direction away from the hand and return to ray operation again. Also, the same problem can occur when the operator moves a virtual object backward by hand gesture during direct operation and intentionally switches to ray operation. Therefore, by temporarily stopping reception of input by hand gestures when ray operation and direct operation are switched, it is possible to prevent the virtual object from moving due to the operation of returning the hand orientation to the original state. Thereby, the operability can be further improved.

[0079] (Others) As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0080] For example, in the above-described embodiment, an example in which the present invention is applied to a non-transmissive or video see-through type HMD has been described. However, the present invention may be applied to an optical see-through type HMD. Further, the present invention may be applied to a display device that is not head-mounted, for example, a handheld display device, a stationary display device, a display device of a computer or a smartphone, a projector screen, a retinal projection type display, and the like.

[0081] In the above-described embodiment, the movement parameter acquisition unit 1220 acquired the movement parameters by calculating based on the input (temporal change in the position of the cursor) due to the movement of the operator's hand. However, data such as a correspondence table may be stored in advance, and instead of calculating each time, the movement parameter acquisition unit 1220 may acquire the movement parameters by selecting a value corresponding to the calculation result from the correspondence table or the like. The same applies to the gesture movement parameter acquisition unit 1170. Instead of calculating each time, the movement parameters for gesture input may be acquired by selecting a value corresponding to the calculation result from the correspondence table or the like. The same also applies to the position and orientation acquisition unit 1110 and the ray cursor position and orientation acquisition unit 1120. Instead of calculating each time, the position and orientation may be acquired by selecting a value corresponding to the calculation result from the correspondence table or the like.

[0082] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0083] The disclosure of this specification includes the following configurations, methods, and programs.

[0084] [Configuration 1] An information processing apparatus that provides an operator with a user interface for operating a virtual object on a virtual space displayed on a display device, a first acquisition unit that acquires a first movement parameter based on an input due to the movement of the operator's hand; a second acquisition unit that acquires a second movement parameter based on an input due to the operator's hand gesture; a position control unit that determines the position of the virtual object after movement based on at least one of the first movement parameter acquired by the first acquisition unit and the second movement parameter acquired by the second acquisition unit; An image generation unit that generates an image including the virtual object for display on the display device based on the position determined by the position control unit, The first movement parameter acquired by the first acquisition unit is a parameter that defines movement in accordance with a constraint that restricts the direction or range in which the virtual object can move. The second movement parameter acquired by the second acquisition unit is a parameter that defines movement contrary to the constraint. An information processing apparatus.

[0085] [Configuration 2] The first movement parameter is a parameter that defines movement within a predetermined plane. The second movement parameter is a parameter that defines movement in a direction orthogonal to the predetermined plane. The information processing apparatus according to Configuration 1.

[0086] [Configuration 3] The first movement parameter is a parameter that defines movement within the range in which the operator's hand can move. The second movement parameter is a parameter that defines movement in a direction away from the operator's hand. The information processing apparatus according to Configuration 1 or Configuration 2.

[0087] [Configuration 4] As a method of operating the virtual object, it includes at least one of a first operation of directly operating the virtual object with the operator's hand and a second operation of operating the virtual object with a virtual ray emitted from the operator's hand. The information processing apparatus according to any one of Configurations 1 to 3.

[0088] [Configuration 5] The first operation and the second operation can be switched during the operation of the virtual object. The information processing apparatus according to Configuration 4.

[0089] [Configuration 6] A determination unit that determines the switching between the first operation and the second operation based on the distance between the operator's hand and the virtual object. The information processing apparatus according to Configuration 5.

[0090] [Configuration 7] The image generation unit generates an image in which the ray part is highlighted when switching from the first operation to the second operation, and generates an image in which the hand part is highlighted when switching from the second operation to the first operation. The information processing apparatus according to Configuration 5 or Configuration 6.

[0091] [Configuration 8] The hand gesture is an operation that changes the hand posture with respect to the reference posture. The information processing apparatus according to any one of Configurations 1 to 7.

[0092] [Configuration 9] The second movement parameter includes a movement amount. The second acquisition unit acquires the movement amount based on the degree of change in the posture with respect to the reference posture. The information processing apparatus according to Configuration 8.

[0093] [Configuration 10] When the degree of change in the posture with respect to the reference posture is equal to or less than a threshold value, the second acquisition unit sets the movement amount to 0. The information processing apparatus according to Configuration 9.

[0094] [Configuration 11] The second movement parameter includes a movement direction. The second acquisition unit determines the movement direction based on the direction of change in the posture with respect to the reference posture. The information processing apparatus according to any one of Configurations 8 to 10.

[0095] [Configuration 12] After determining the movement direction based on the direction of change in the posture with respect to the reference posture, the second acquisition unit temporarily fixes the movement direction. The information processing apparatus according to Configuration 11.

[0096] [Configuration 13] After the second acquisition unit determines the movement direction based on the direction of change in the posture with respect to the reference posture, the movement direction is fixed until the hand posture returns to the reference posture. The information processing apparatus according to Configuration 11 or Configuration 12.

[0097] [Configuration 14] While an input by a hand gesture is being performed, the position control unit determines the position after the movement of the virtual object using only the second movement parameter. The information processing apparatus according to any one of Configurations 1 to 13.

[0098] [Configuration 15] An information processing system that provides an operator with a user interface for operating a virtual object in a virtual space, A display unit that displays an image including the virtual object, An imaging unit that images a real space including the operator's hand, A first acquisition unit that acquires a first movement parameter based on an input by the movement of the operator's hand detected from the image captured by the imaging unit, A second acquisition unit that acquires a second movement parameter based on an input by the hand gesture of the operator detected from the image captured by the imaging unit, A position control unit that determines the position after the movement of the virtual object based on at least one of the first movement parameter acquired by the first acquisition unit and the second movement parameter acquired by the second acquisition unit, An image generation unit that generates an image including the virtual object for display on the display unit based on the position determined by the position control unit, The first movement parameter acquired by the first acquisition unit is a parameter that defines movement in accordance with constraints that limit the direction or range in which the virtual object can move. The second movement parameter acquired by the second acquisition unit is a parameter that defines a movement that violates the constraint. Information processing system.

[0099] [Method 16] A method for providing an operator with a user interface for operating a virtual object on a virtual space displayed on a display device, a step in which an information processing apparatus acquires a first movement parameter based on an input by the movement of the operator's hand; a step in which an information processing apparatus acquires a second movement parameter based on an input by the operator's hand gesture; a step in which an information processing apparatus determines a position after movement of the virtual object based on at least one of the first movement parameter and the second movement parameter; a step in which an information processing apparatus generates an image including the virtual object for display on the display device based on the determined position, and the first movement parameter is a parameter that defines a movement in accordance with a constraint that restricts a direction or range in which the virtual object can move, the second movement parameter is a parameter that defines a movement that violates the constraint. Method.

[0100] [Program 17] A program for causing an information processing apparatus to execute each step described in Method 16.

Explanation of Signs

[0101] 1000: Display device 1100: Information processing apparatus 1220: Movement parameter acquisition unit (first acquisition unit) 1170: Gesture movement parameter acquisition unit (second acquisition unit) 1180: Virtual object position control unit (position control unit) 1190: Image generation unit

Claims

1. An information processing apparatus that provides an operator with a user interface for operating a virtual object on a virtual space displayed on a display device, a first acquisition unit that acquires a first movement parameter based on an input by the movement of the operator's hand, a second acquisition unit that acquires a second movement parameter based on an input by the operator's hand gesture, a position control unit that determines the position of the virtual object after movement based on at least one of the first movement parameter acquired by the first acquisition unit and the second movement parameter acquired by the second acquisition unit, and an image generation unit that generates an image including the virtual object for display on the display device based on the position determined by the position control unit, wherein the first movement parameter acquired by the first acquisition unit is a parameter that defines movement in accordance with a constraint that restricts the direction or range in which the virtual object can move, and the second movement parameter acquired by the second acquisition unit is a parameter that defines movement that violates the constraint, An information processing apparatus.

2. The first movement parameter is a parameter that defines movement within a predetermined plane, and the second movement parameter is a parameter that defines movement in a direction orthogonal to the predetermined plane, The information processing apparatus according to claim 1.

3. The first movement parameter is a parameter that defines movement within the range in which the operator's hand can move, and the second movement parameter is a parameter that defines movement in a direction away from the operator's hand, The information processing apparatus according to claim 1.

4. As a method of operating the virtual object, it includes at least one of a first operation of directly operating the virtual object with the operator's hand and a second operation of operating the virtual object with a virtual ray emitted from the operator's hand, The information processing apparatus according to claim 1.

5. The first operation and the second operation can be switched during the operation of the virtual object, The information processing apparatus according to claim 4.

6. It has a determination unit that determines the switching between the first operation and the second operation based on the distance between the operator's hand and the virtual object, The information processing apparatus according to claim 5.

7. The image generation unit generates an image in which a portion of the ray is highlighted when switching from the first operation to the second operation, and generates an image in which a portion of the hand is highlighted when switching from the second operation to the first operation. The information processing apparatus according to claim 5.

8. The hand gesture is an operation that changes the posture of the hand with respect to a reference posture. The information processing apparatus according to claim 1.

9. The second movement parameter includes a movement amount. The second acquisition unit acquires the movement amount based on the degree of change in the posture with respect to the reference posture. The information processing apparatus according to claim 8.

10. When the degree of change in the posture with respect to the reference posture is equal to or less than a threshold value, the second acquisition unit sets the movement amount to 0. The information processing apparatus according to claim 9.

11. The second movement parameter includes a movement direction. The second acquisition unit determines the movement direction based on the direction of change in the posture with respect to the reference posture. The information processing apparatus according to claim 8.

12. After determining the movement direction based on the direction of change in the posture with respect to the reference posture, the second acquisition unit temporarily fixes the movement direction. The information processing apparatus according to claim 11.

13. After determining the movement direction based on the direction of change in the posture with respect to the reference posture, the second acquisition unit fixes the movement direction until the hand posture returns to the reference posture. The information processing apparatus according to claim 11.

14. While an input by a hand gesture is being performed, the position control unit determines the position of the virtual object after movement using only the second movement parameter. The information processing apparatus according to claim 1.

15. An information processing system that provides an operator with a user interface for operating a virtual object in a virtual space, A display unit that displays an image including the virtual object, An imaging unit that images a real space including the operator's hand, A first acquisition unit that acquires a first movement parameter based on an input by the movement of the operator's hand detected from the image captured by the imaging unit, A second acquisition unit that acquires a second movement parameter based on an input by the hand gesture of the operator detected from the image captured by the imaging unit. A position control unit that determines the position of the virtual object after movement based on at least one of the first movement parameter acquired by the first acquisition unit and the second movement parameter acquired by the second acquisition unit; An image generation unit that generates an image including the virtual object for display on the display unit based on the position determined by the position control unit; and The first movement parameter acquired by the first acquisition unit is a parameter that defines movement in accordance with constraints that limit the direction or range in which the virtual object can move; The second movement parameter acquired by the second acquisition unit is a parameter that defines movement that violates the constraints. An information processing system.

16. A method for providing an operator with a user interface for operating a virtual object on a virtual space displayed on a display device, the method comprising: The information processing apparatus acquiring a first movement parameter based on an input by the movement of the operator's hand; The information processing apparatus acquiring a second movement Parameter based on an input by the operator's hand gesture; The information processing apparatus determining the position of the virtual object after movement based on at least one of the first movement parameter and the second movement parameter; The information processing apparatus generating an image including the virtual object for display on the display device based on the determined position; and The first movement parameter is a parameter that defines movement in accordance with constraints that limit the direction or range in which the virtual object can move; The second movement parameter is a parameter that defines movement that violates the constraints. A method.

17. A program for causing an information processing apparatus to execute each step of the method according to Claim 16.

Images